DB_File size limitations?

skazat has asked for the wisdom of the Perl Monks concerning the following question:

Hello All,

I'm currently inspecting a problem with new entries in a Berkeley DB File (file tells me: Berkeley DB (Hash, version 7, native byte-order)) not being saved,

I'm currently using the DB_File module to access this database. I'm not getting any errors in the error log, but trying to save a key/value pair and then doing a quick exists() in my code comes up with nothin'

Is there an arbitrary limit on the size of a Berkeley DB file that can be added to using the DB_File module?

Currently, the Berkeley DB file is hovering at about 10 megs (10633216 bytes). I was under the impression that the Berkeley DB format supports files that are very Large indeed. Cheers,

-justin simoni
!skazat!

Comment on DB_File size limitations?

Replies are listed 'Best First'.
Re: DB_File size limitations? by tilly (Archbishop) on Sep 21, 2004 at 06:09 UTC
When the file gets to 2 GB, it is possible that you'll have an OS problem. After that, you're good until a couple of terabytes, or running out of disk. I'd guess that your problem is due to a simple oversight on your part. A common suspect is forgetting to chomp data consistently - you save a key with a newline in it and then look for the same key without that, or vice versa. Or (more insidiously) you're reading a Windows file on Linux and can't see the \r that snuck into your key. There are plenty of other possible suspects. But size limitations on BerkeleyDB is not one of them.	[reply]
Re: DB_File size limitations? by archen (Pilgrim) on Sep 21, 2004 at 14:01 UTC
If your system can't handle a file over 2Gb then what tilly says is probably going to be your only problem. Berkely DB should be able to handle a database into the terabytes, with key/values limited to 1-2 gigabytes. So realistically you will hit size restrictions on the hard drive before the database. I created a utility to edit berkely db files and I was experiencing a similar issue. I'd add keys, then try to get a list of keys from 'keys %db' but I wasn't getting new keys, or sometimes I'd get ones that were deleted. Eventually I resolved this by doing a sync. `my $bdb = tie my %db .... $db{'new'} = 'stuff'; $bdb->sync();` [download] Give that a try and see if it helps.	[reply] [d/l]
Re^2: DB_File size limitations? by skazat (Chaplain) on Sep 22, 2004 at 01:23 UTC
That sounds like a great idea; and may relieve my problem. Hopefully I can get it to work on this program, since it's indirectly using the DB_File module via AnyDBM_File. I'm guessing I can wrap this call into a eval(), so if you try to run this code using a different DB file type (other than Berkeley DB) it won't produce an error. Still very weird how this is topping out at 10 megs instead of say, 2 gigs. I checked to see if this was an account quota issue (it wasn't) -justin simoni !skazat!	[reply]
Re: DB_File size limitations? by jZed (Prior) on Sep 21, 2004 at 11:58 UTC
While not related to your file size question, you may want to use the perl module BerkeleyDB rather than DB_File since it can take advantage of much more of Berkeley DB features. Also, BTW the anonymonk's comment that Berkeley DB is not a server is wrong - of course Berkeley DB is a server. Perhaps what he/she meant is that there is no SQL front end to that server, but there again he/she is also now wrong - the DBI distribution comes with DBD::DBM which provides a DBI/SQL front end to Berkeley DB using either of the perl modules (BerkeleyDB or DB_File). DBD::DBM's SQL is limited, but it can do INNER and OUTER joins on two tables so minimally qualifies as an RDBMS.	[reply]
Re^2: DB_File size limitations? by Anonymous Monk on Sep 21, 2004 at 12:06 UTC
how is it a server?	[reply]
Re^3: DB_File size limitations? by jZed (Prior) on Sep 21, 2004 at 13:23 UTC
From the Sleepycat home page: Berkeley DB provides fast, reliable, recoverable data storage and retrieval services to developers building applications.	[reply]
Re^4: DB_File size limitations? by perrin (Chancellor) on Sep 21, 2004 at 14:01 UTC
Re^5: DB_File size limitations? by jZed (Prior) on Sep 21, 2004 at 15:16 UTC
Re: DB_File size limitations? by TedPride (Priest) on Sep 21, 2004 at 08:15 UTC
I don't know if this is true of your DB, but some DBs have a small lag time between when you submit your data and when it actually arrives at the database, and querying for your record just afterwards usually returns nothing. Check if your information is actually making it to the database ok, and if it is, get around the problem by printing submitted information rather than queried information. I've run up against this exact same problem with the mySQL database on my current hoster.	[reply]
Re^2: DB_File size limitations? by Anonymous Monk on Sep 21, 2004 at 09:23 UTC
Berkeley DB is not an RDBMS. There is no server.	[reply]


No such thing as a small change
	PerlMonks