Re: Hash lookups, Database lookups, and Scalability

First, DBD::SQLite is not a sound database any way. Also there is a big performance boost when you wrap your queries in DBD::SQLite's transaction. I didn't see you use it, and it would be interesting to get another benchmark with that enabled.

In general, database search is less efficient than in-memory ones. First you have more freedom to tailor your algorithm and data structure for in-memory search; secondly for database search, the results are transferred through network communication, two-way communication for each query, which obviously costs extra time. How good is the database's cache ability will also impact the performance.

However this does not mean that database search is not a good viable solution. Database gives you lots of important benefits than just speed, which is OT here.

Comment on Re: Hash lookups, Database lookups, and Scalability

Replies are listed 'Best First'.
Re^2: Hash lookups, Database lookups, and Scalability by davido (Cardinal) on Oct 31, 2004 at 17:53 UTC
I'm curious what you mean by "DBD::SQLite is not a sound database". I know it's a single file, lightweight, minimal-SQL set database, but to me something that isn't sound is buggy. Is that what you're saying? Just curious. (By the way, I just discovered a GPF-producing bug if I switch my code to use selectrow_array() with a pre-prepared statement on Win32, so maybe you've got a point. ;) And yes, I fully expected the database solution to be much slower than the in-memory hash solution. I mentioned that in my post. I was attempting to quantify how much slower, and how sensitive the database version is to dataset-size. Wrapping in transactions definately helps. Take the example of my building the database initially. With transactions, it takes the blink of an eye, but without, it takes about 45 seconds to build the database on my system. But on the query side, I'm specifically trying to engage in single queries, and didn't want to see transactions influencing the speed. However, I agree it would be interesting to rewrite the test in such a way that it could take advantage of transactions, and I may do that just to see what comes of it. Anyway, it's kind of fun to tinker with this stuff. ...hope you agree, even it the database discussion is diverging a little from core-Perl topics. Dave	[reply]
Re^3: Hash lookups, Database lookups, and Scalability by pg (Canon) on Oct 31, 2004 at 18:16 UTC
"Anyway, it's kind of fun to tinker with this stuff. ...hope you agree, even it the database discussion is diverging a little from core-Perl topics. " I agree 100 percent. I did notice that recently we are having more and more database related discussions here, which is very possitive, at least to my standard. This indicates to me that more and more people are using Perl for an increased range of application areas, and most likely the average size of the applications is going upwards. This is definitely a positive news.	[reply]
A reply falls below the community's threshold of quality. You may see it by logging in.


good chemistry is complicated, and a little bit messy -LW
	PerlMonks