Re: DBD::SQLite tuning

It appears, from cursory examination, that commits on SQLite are implied to be slow.

From the DBD::SQLite documentation:

SQLite is fast, very fast. I recently processed my 72MB log file with it, inserting the data (400,000+ rows) by using transactions and only committing every 1000 rows (otherwise the insertion is quite slow), and then performing queries on the data. (Emphasis added)

From http://www.sqlite.org/cvstrac/wiki?p=PerformanceConsiderations, it says:

When doing lots of updates/inserts on a table it is a good idea to contain them within a transaction, . . . This will make SQLite write all the data to the disk in one go, vastly increasing performance. (Emphasis added)

Apparently, what's happening is there is a disk flush of the actual changes to the table upon commit. This apparently is to allow for preservation of committed data if the database crashes. Oracle and MySQL (using InnoDB) have REDO logs which capture the commands that made the change, then actually apply the changes when it's convenient or necessary. (MySQL, using MyISAM tables, doesn't have this guarantee. This is why you can have corruption of MyISAM tables, but not InnoDB tables.)

This constant flushing of the tables (and the attendant indices) looks to be why you have the performance issues.

Note: I don't have proof this is what's happening, but it seems like a reasonable guess.

Being right, does not endow the right to be rude; politeness costs nothing.
Being unknowing, is not the same as being stupid.
Expressing a contrary opinion, whether to the individual or the group, is more often a sign of deeper thought than of cantankerous belligerence.
Do not mistake your goals as the only goals; your opinion as the only opinion; your confidence as correctness. Saying you know better is not the same as explaining you know better.

Comment on Re: DBD::SQLite tuning

Replies are listed 'Best First'.
Re^2: DBD::SQLite tuning by perrin (Chancellor) on Mar 16, 2005 at 21:20 UTC
As I said in the intro, leaving out all commits makes it go about three times as fast, but then it wouldn't work for sharing the data between multiple processes. At least I'm assuming that other processes can't see uncomitted data.	[reply]
Re^3: DBD::SQLite tuning by dragonchild (Archbishop) on Mar 17, 2005 at 14:37 UTC
I understand all of that. You were asking "Why is this slow?" (which is the necessary precursor to "How do I make this faster?"). I replied "Commits are implied to be slow in SQLite." I suspect that you are using SQLite correctly, but not for its intended purpose. It's like using Oracle for an online forum or BDB for a data warehouse. You could do that, but it wouldn't be recommended as best. The person who claimed Foo is Nx faster than Bar is correct, for his/her needs. And, for your needs, Bar is Nx faster than Foo. We've all seen situations where a semi-tuned MySQL 4.1 database on a single CPU blew away an tuned Oracle 9.2 database running on a quad-Xeon. Yet, we're not going to claim that MySQL 3.x is faster than Oracle in all situations. For your usage, BDB is clearly better. SQLite is probably better than BDB if the following is true: Reads * LargeNumber >= Writes Reads involve complex queries Different reads may use different indices Otherwise, it's probably best to use BDB. Of course, I'm still wondering why SQLite is good when embedded MySQL exists. (Though the weird license is probably part of it ...) Being right, does not endow the right to be rude; politeness costs nothing. Being unknowing, is not the same as being stupid. Expressing a contrary opinion, whether to the individual or the group, is more often a sign of deeper thought than of cantankerous belligerence. Do not mistake your goals as the only goals; your opinion as the only opinion; your confidence as correctness. Saying you know better is not the same as explaining you know better.	[reply]


Don't ask to ask, just ask
	PerlMonks