You are here

Home » Forums » Support Forums » Other Apps Support (Deprecated)

GnuCash Portable takes a LONG time to save files

18 posts / 0 new
Log in or register to post comments
Last post
May 30, 2014 - 2:31pm
#1
lalligood
lalligood's picture
Offline
Last seen: 8 years 11 months ago
Joined: 2007-07-02 11:01
GnuCash Portable takes a LONG time to save files

I've been running GnuCash Portable for quite some time, however, ever since at least the 2.6.x version it now takes a Really. Long. Time. To. Save. Like over 2 minutes to saves my file. My main account file is only 590KB, so I'm really perplexed as to what could take so long for it to save.

A description of my entire situation:

  1. The drive is a 64GB Transcend USB 3.0 flash drive. In a USB 3.0 port, I can regularly read files at 80+MB/sec & writes are easily 40MB/sec. It flies. This USB drive goes back & forth daily between the 2 machines described below.
  2. My main machine is a laptop running Ubuntu (14.04 64-bit), which is using a 2.4.x version of GnuCash & has USB 3.0 ports. The laptop/GnuCash has no problems opening and/or saving my account file. Saves are FAST--easily under 3 seconds--as I would expect.
  3. Then I run GnuCash Portable 2.6.3 on a Windows 7 PC 64-bit with 8GB RAM & "only" USB 2.0 ports. If I copy a large file, like a CD .iso image on this PC, it takes longer than my laptop, but reads are still in a very acceptable ~20MB/sec range. Writes are pretty much the same speed since the USB 2.0 port is the bottleneck.
  4. I've had the drive long enough that I was using at least the last couple v2.4.x releases of GnuCash Portable. Somewhere along the way, these long writes (saves) started. I do not distinctly recall if it was with one of the last 2.4.x releases or if it started with 2.6.x. I am certain that it started no later than when I installed the first 2.6.x release of GnuCash Portable.

It's only when I run GnuCash that is takes so darn long to save. If anyone has any ideas, I'm all ears. And I would be willing to provide any additional information that I can. Thank you for looking into this!

Top
May 30, 2014 - 2:37pm
John T. Haller
John T. Haller's picture
Offline
Last seen: 1 hour 7 min ago
AdminDeveloperModeratorTranslator
Joined: 2005-11-28 22:21
Sequential vs Random Writes, Copy

First off, a note. Write speed of a large file like an ISO is very different from the the write speed of portable software saving small bits of information. To see what I mean, grab a copy of CrystalDiskMark Portable and run the benchmark on your drive. Lots of small, non-sequential writes can be very very slow, even on a 'fast' USB 3.0 drive. What matters is the memory and the controller. Additionally, I'd wager that GnuCash saves data inefficiently, which means it isn't just writing 600KB all at once but in small little pieces as it updates individual transactions.

Second, try a fresh copy of GnuCash Portable (you can install it alongside the original) and see if a blank ledger is saving properly and without issue. Then we can at least rule out something odd going on.

Sometimes, the impossible can become possible, if you're awesome!

Top
May 31, 2014 - 5:42pm
(Reply to #2)
Wm ...
Offline
Last seen: 6 years 5 months ago
Joined: 2010-07-17 12:37
Ahem. More details needed.

L: how are you storing your GnuCash file/data? XML or SQL? If SQL which DB? If XML are you compressing or not?

John: GnuCash storage behaves according to the data format the user chooses: XML or a choice of SQL backends. The backends behave differently and the user gets a good combination of choices wrt saving options.

So, if you are using the XML format (I'd recommend that for a USB drive and most similar PA situations) you'll only get a write to the drive as per your Edit > Preferences > General settings. i.e. you can tell it to, effectively, never save and only do so when you manually say save, which you get reminded about on exit, or you can ask it to save periodically. You can also set it to ask you if you want to save so if your drive is busy you can tell it to wait.

GnuCash, is, if you think about it, portable drive friendly by co-incidence, depending on your storage choice. I wouldn't run postgres or mysql on a pen drive. sqlite is workable because it can work in memory like GnuCash's xml.

L: How long does Accounts Tab / Menu / Actions / Check & Repair / Check & Repair All take ? My uncompressed XML file is around 3.9MB at the moment and the ck+ takes around a minute on over a decade old kit.

L: I'm hoping you're going to come back and say, "I made a bad backend choice for my device" Smile

P.S. I can be defensive of GNC in general, I like it, a lot.

Wm

Top
June 2, 2014 - 5:10pm
(Reply to #3)
lalligood
lalligood's picture
Offline
Last seen: 8 years 11 months ago
Joined: 2007-07-02 11:01
More information...

I deleted the GnuCashPortable directory from my drive & installed 2.6.3 anew. Adding single entry to the ledger & forcing it to save manually took a full 2.5 minutes to complete. Sad

Files are stored with all of the defaults: Compress files is checked, Show auto-save confirmation question is checked, Auto-save time interval of 5 minutes, & Retain log/backup files for 30 days. (Enable timeout... is left unchecked.) I do not have any SQL databases installed on my flash drive--and I'm quite certain that my laptop writes to flash drive too. So if I made a bad backend choice, that's because I took the defaults...

The Check & Repair ran for a good 10-12 minutes. The drive activity LED would blink a couple times every few seconds, and the title bar eventually had '(Not Responding)' appended onto it. Frankly, I'd all but given up on it & looked away to do something else. Then I happened to notice it was done.

Wm, I appreciate your defensiveness of GNC. I've really enjoyed using it for several years & would like to continue using it for many more!

--Lance

Top
June 3, 2014 - 8:16pm
(Reply to #4)
Wm ...
Offline
Last seen: 6 years 5 months ago
Joined: 2010-07-17 12:37
I'm still here but puzzled, bear with me, please

Your settings look good (gnc doesn't do weird defaults).

gnc / Actions / Check & repair should be in memory unless (these are just thoughts) your available memory is already consumed and your system is swapping for some reason in which case it is back to you.

To clarify, if Ck&Rep is taking 10-12min rather than a few seconds on a simple file something else is wrong.

I've just created a new file using A Simple Checkbook as a template, added one transaction to get my lottery winnings into my account (which I think replicates your scenario above) and if I run C&R it is less than my watch can measure, i.e. I need a computer to time it; certainly not minutes or multiples of seconds.

Your kit is almost certainly better than mine, I have an XP era laptop, yes, old stuff that still works.

Two tests, please.

1. Create a new gnc file and save it to a local hard drive (not usb) it will only be about 7kb in uncompressed XML and won't contain anything personal, add a transaction, run C&R. Tell us how long that takes. It should be done inside a second. Now repeat with the USB stick.

2. Presuming you have installed your gnc to your portable device reinstall it on a hard drive and run the experiment again.

This isn't trick stuff. I believe you, I'm trying to narrow down where the problem is. Your privacy won't be compromised and we'll have a better idea of a way forward.

If it turns out you were running some super demanding game at the same time I'll be miffed Smile

Wm

Top
June 4, 2014 - 9:40am
(Reply to #5)
lalligood
lalligood's picture
Offline
Last seen: 8 years 11 months ago
Joined: 2007-07-02 11:01
Narrowing it down...

Wm,

Thank you for the suggestions as I feel like I've got some VERY interesting news to report back...

  1. The new file creation on both HDD & USB were as you (& I both) expected: time to save was nearly instantaneous.
  2. I copied my ~590KB GNC file from USB => my HDD. Opening (I did mention that it takes as long to open the file as it does to save it, didn't I?) or saving STILL TOOK THE SAME PAINFUL LENGTH OF TIME as when it was on the USB drive!

So it looks like my problem is local to my FILE, not GNC itself. (Still not a good thing, but IMHO it absolves GNC of being the source of the problem...) I guess that now leaves me with an entirely new question:

What are my options for exporting some/all of my existing accounts to a totally new file?

I know this has now crossed over the boundaries of GNC in the portable sense, so I won't take offense if you tell me to go seek assistance with the GNC project itself. Thanks again!

--Lance

Top
June 4, 2014 - 9:41am
(Reply to #6)
John T. Haller
John T. Haller's picture
Offline
Last seen: 1 hour 7 min ago
AdminDeveloperModeratorTranslator
Joined: 2005-11-28 22:21
Disable Antivirus

Have you tried disabling your antivirus/firewall temporarily to see if they are interfering? Several of the free ones can cause really strange issues in cross platform software.

Sometimes, the impossible can become possible, if you're awesome!

Top
June 4, 2014 - 2:18pm
(Reply to #7)
lalligood
lalligood's picture
Offline
Last seen: 8 years 11 months ago
Joined: 2007-07-02 11:01
A/V makes no difference

I tried saving my GNC file several times--even with exiting & restarting GNC in between attempts--but I did not observe any difference in save times.

--Lance

Top
June 6, 2014 - 12:07pm
(Reply to #8)
Wm ...
Offline
Last seen: 6 years 5 months ago
Joined: 2010-07-17 12:37
Next check, L

Open your usual gnc file and choose

File / SaveAs / DataFormat sqlite3 from the drop down at the top / Name: test0_sql3 / SaveAs button bottom right.

While you're at it do a SaveAs / DataFormat XML / Name: test0_XML / ditto

Notes:
1. sqlite3 is built in, no external db required
2. your existing data will not be affected, the test0 files are completely separate to your existing data

Close gnc ; pause if necessary for stuff to flush.

Ck your gnc data dir you should have your live data file + two test0 files to play with, one XML the other SQLite3 (the files with yyyymmddhhmmss in their name are backups).

a. Start gnc ; the last saved file should open automatically coz it was the last file saved.
b. go File and choose the one you want from the numbered list.
c. note time to load, any diff?

Wm

Top
June 13, 2014 - 3:51pm
(Reply to #9)
lalligood
lalligood's picture
Offline
Last seen: 8 years 11 months ago
Joined: 2007-07-02 11:01
No difference, but...

Sorry for taking so long to get you another update.

My GNC file contains nothing but account registers, 2 current & a (hidden) closed checking/savings account or two.

In going through all of your latest tests, there was no significant difference between XML vs SQLite3 formats.

However, in just playing around with different things, I did see a 40-50% improvement after UNCHECKING the Compress files option. Obviously this made my XML-based .gnucash file was considerably larger (~10x), but it seems that the overhead of compression causes it to take longer to save... Not an unacceptable option, but it's still not where it ought to be time-wise. I spent some time looking through the release notes & bug reports on gnucash.org to see if there was anything about changing the compression algorithm or anything like that--there wasn't.

--Lance

Top
June 14, 2014 - 5:47am
(Reply to #10)
Wm ...
Offline
Last seen: 6 years 5 months ago
Joined: 2010-07-17 12:37
possible last diagnostic

instructions known to work on WinXP-SP3, hopefully you'll get the gist

exit gnc if already running

find your gnc portable binary, you're looking for
PortableApps\GnuCashPortable\GnuCashPortable.exe

right click / send to desktop / create shortcut (or whatever your Win calls it)

right click / properties to edit the shortcut and add
--debug --extra
after GnuCashPortable\GnuCashPortable.exe

start gnc, pause for a minute and then exit gnc

look in
PortableApps\GnuCashPortable\Data\Temp
you should find a recently modified file named something like
gnucash.trace.BLAH999.log
expect it to be at last 2MB

Open it in a decent text editor (it is a plain text file) and you'll see what gnc has been doing while you put the kettle on Smile

Edit: I've just had a play with mine in a spreadsheet (it imports easily as a CSV with spaces as the delimiters, you don't, or shouldn't, care about the conversational stuff on the RHS).

Time passes: I'm finding it quite interesting looking at my log file in detail. Let me know if anything jumps out at you as I may write a script to analyse the file and see what the folk in the gnc lists think.

Wm

Top
June 20, 2014 - 9:12am
(Reply to #11)
lalligood
lalligood's picture
Offline
Last seen: 8 years 11 months ago
Joined: 2007-07-02 11:01
Pretty boring really...

Wm,

I did as you asked. (BTW, no kidding about it taking longer to launch GNC!) There are essentially 3 "sections" of the logfile.

The first is roughly 500 lines of this:
* 12:19:55 INFO [gnc_commodity_table_insert] insert 07BF5FE0 AFA into nsp=07159B20 CURRENCY

Followed by *several* thousand blocks like this:
* 12:20:03 INFO [xaccTransSetDateInternal] addr=07D98158 set date to 1322103081.000000000 Wed Nov 23 21:51:21 2011
* 12:20:03 INFO [xaccTransWriteLog] Attempt to write disabled transaction log
* 12:20:03 INFO [trans_cleanup_commit] get rid of rollback trans=07D98028
* 12:20:03 INFO [xaccOpenLog] Attempt to open disabled transaction log
* 12:20:03 INFO [xaccTransWriteLog] Attempt to write disabled transaction log
* 12:20:03 INFO [xaccTransWriteLog] Attempt to write disabled transaction log
* 12:20:03 INFO [trans_cleanup_commit] get rid of rollback trans=07D98028
* 12:20:03 INFO [xaccOpenLog] Attempt to open disabled transaction log
* 12:20:03 INFO [xaccTransWriteLog] Attempt to write disabled transaction log
* 12:20:03 INFO [xaccTransSetDateInternal] addr=07D98028 set date to 1322110800.000000000 Thu Nov 24 00:00:00 2011

And then finally ~600 of these:
* 12:36:45 INFO [xaccAccountRecomputeBalance] acct= starting baln=0/1
* 12:36:45 INFO [xaccAccountSetGUID] acct=07D96240

And as an extra measure, I assumed all of the messages that started with "* INFO ..." were purely informational. So I filtered them all out, which left me with these 13 lines (out of 56,000!):
* 12:19:54 MESSG Could not locate optional module gnucash/gtkmm interface v.0
* 12:19:54 MESSG Could not locate optional module gnucash/python interface v.0
* 12:19:54 MESSG loading system configuration
* 12:19:54 MESSG loading user configuration
* 12:19:54 MESSG loading auto configuration
* 12:19:54 MESSG loading saved reports
* 12:19:54 MESSG loading stylesheets
* 12:19:55 WARN Could not spawn perl: Failed to execute child process (No such file or directory)
* 12:37:03 MESSG num_instances: 0
* 12:37:03 MESSG num_to_create: 0
* 12:37:03 MESSG num_auto_create_instances: 0
* 12:37:03 MESSG num_auto_create_no_notify_instances: 0
* 12:37:03 MESSG need dialog? false

To me this was interesting but also falls under the rare circumstance of "no news is bad news". Thank you again.
--L

--Lance

Top
June 20, 2014 - 5:28pm
(Reply to #12)
Wm ...
Offline
Last seen: 6 years 5 months ago
Joined: 2010-07-17 12:37
Well, I find news interesting and don't own a TV

Dunno if you want to play Watson or Holmes but you've got the facts right before you, I think, L

Ignore gtkmm, python and perl, if you had the first two working you'd know about it and most people can get the last to function if they need to.

I think the 12:20:nn stuff is "not right" but the
===
* 12:19:55 WARN Could not spawn perl: Failed to execute child process (No such file or directory)
* 12:37:03 MESSG num_instances: 0
===
gap is obvious.

the test for perl (gnc is trying to work out if you can get share prices from external sources) is fine.

the 18 minutes in between is a mystery of sorts.

I'm guessing a broken file of sorts, certainly not a PA issue.

I think it is the main gnc lists for you Sad

Sorry, I'm stuck, L, I know your file is fucked but I don't know in which exact way it is fucked or became fucked.

Wm

Top
June 20, 2014 - 6:38pm
(Reply to #13)
Gord Caswell
Gord Caswell's picture
Offline
Last seen: 5 days 17 hours ago
DeveloperModerator
Joined: 2008-07-24 18:46
gap

The gap is there because he filtered out the info messages, which obviously filled that time.

Top
June 21, 2014 - 9:49am
(Reply to #14)
Wm ...
Offline
Last seen: 6 years 5 months ago
Joined: 2010-07-17 12:37
Not with you

I'm not with you. Whatever the 18 minutes were spent doing it was something to do with the data. He doesn't have massive files.

Wm

Top
June 24, 2014 - 9:15pm
(Reply to #15)
lalligood
lalligood's picture
Offline
Last seen: 8 years 11 months ago
Joined: 2007-07-02 11:01
Perhaps I wasn't clear...?

Gord hit the nail on the head. Sorry for the confusion. I filtered out anything that was not an INFO message, hoping that would be easier/more helpful/more insightful if I showed just the bad stuff... However, if you think I should head on over to the GNC site for further assistance, I'm still OK with that.

Thanks again for all the help!

--Lance

Top
June 25, 2014 - 2:26pm
(Reply to #16)
Wm ...
Offline
Last seen: 6 years 5 months ago
Joined: 2010-07-17 12:37
It is going to be your data

Try this logically and see what you conclude, L.

It isn't specific to the PA version of gnc, I know quite a lot about gnc but others know a lot more, you'll get more help about a general gnc issue in a more specialised forum.

If it was happening to everyone it would be a known issue by now (it isn't); maybe you should share what it is your gnc is doing when it wanders off by itself with people that might share similar data (my data is different).

Wherever you choose to take it someone is going to ask what gnc spent those ~18 minutes doing. Unless you are happy to share your file with someone else (people tend not to do that with finances hence why I haven't asked) *you* are going to have to distill what is going on.

Is it (for e.g. and at random) a lot of reversed transactions? a scheduled transaction that writes on a daily basis to revalue an investment? unless you are doing a lot by hand it will probably be something you set up and noticed over time. When did you notice your file put on weight?

I'll look at your log file if you want but you seem quite capable of looking at it yourself to me. Maybe it is a wood for trees situation?

Either way those ~18 mins take around 20 seconds here if we're looking at the same thing.

Wm

Top
June 29, 2014 - 1:56pm
(Reply to #17)
Wm ...
Offline
Last seen: 6 years 5 months ago
Joined: 2010-07-17 12:37
Keeping an eye out in gnucash lists

I'm keeping an eye out for a posting from you, L, but someone DoS'd the gnc lists these last few days so if you tried to subscribe you may not even have got a response.

No reason why you should use the same name there as here so I've been looking out for "slow" or similar in the Subject: as a possible posting from you.

Update: 29 Jun

Conversation now in gnc lists. L has provided extra info on-list that makes it clear that this wasn't a PA issue though he couldn't have known that to begin with.

Wm

Top
Log in or register to post comments