http://qs321.pair.com?node_id=612776


in reply to Perl solution for storage of large number of small files

It sounds like you didn't read the BerkeleyDB documentation. If your record is larger than your page size then the record is stored as an overflow record. I don't know how overflow records are accessed. In your case, perhaps you should try using a 64K page size and trying your benchmark again. The default size is 8K which far below your typical record size.

First, the page size implicitly sets the size of an overflow record. Overflow records are key or data items that are too large to fit on a normal database page because of their size, and are therefore stored in overflow pages. Overflow pages are pages that exist outside of the normal database structure. For this reason, there is often a significant performance penalty associated with retrieving or modifying overflow records. Selecting a page size that is too small, and which forces the creation of large numbers of overflow pages, can seriously impact the performance of an application.

Berkeley DB Reference Guide: Selecting a page size

⠤⠤ ⠙⠊⠕⠞⠁⠇⠑⠧⠊