Beefy Boxes and Bandwidth Generously Provided by pair Networks
Syntactic Confectionery Delight
 
PerlMonks  

Re: Perl solution for storage of large number of small files

by diotalevi (Canon)
on Apr 30, 2007 at 14:08 UTC ( [id://612776]=note: print w/replies, xml ) Need Help??


in reply to Perl solution for storage of large number of small files

It sounds like you didn't read the BerkeleyDB documentation. If your record is larger than your page size then the record is stored as an overflow record. I don't know how overflow records are accessed. In your case, perhaps you should try using a 64K page size and trying your benchmark again. The default size is 8K which far below your typical record size.

First, the page size implicitly sets the size of an overflow record. Overflow records are key or data items that are too large to fit on a normal database page because of their size, and are therefore stored in overflow pages. Overflow pages are pages that exist outside of the normal database structure. For this reason, there is often a significant performance penalty associated with retrieving or modifying overflow records. Selecting a page size that is too small, and which forces the creation of large numbers of overflow pages, can seriously impact the performance of an application.

Berkeley DB Reference Guide: Selecting a page size

⠤⠤ ⠙⠊⠕⠞⠁⠇⠑⠧⠊

  • Comment on Re: Perl solution for storage of large number of small files

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://612776]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others about the Monastery: (6)
As of 2024-04-18 11:59 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found