Cache::BerkeleyDB Problems

glwtta has asked for the wisdom of the Perl Monks concerning the following question:

Hi everyone,

I've introduced a Cache::BerkeleyDB based caching mechanism into a web application I'm working on, and it seems to work fine at first, but breaks down when I give it a bit more data.

I have an expensive computation that pre-loads the cache with about 20,000 values. Running it reproduceably wedges the bdb database:

Can't create BerkeleyDB::Env (home=/data/webdata/cache): DB_RUNRECOVERY: Fatal error, run database recovery at /apps/perl-5.8.6/lib/site_perl/5.8.6/Cache/BerkeleyDB_Backend.pm line 29, <DATA> line 225.

Running the exact same query with a smaller input works fine. This is under mod_perl, but I have been testing it with a single httpd process, so I don't think it's a concurrency issue.

Any idea where I should look to solve this?

In the meantime I've been using Cache::File instead, but both the performance and space-efficiency are unacceptable (I think it would be ok on something like reiserfs, but I'm stuck on ext3). Because of the size of the data memory based caches won't work for me, so BDB was really the all-around best solution; if I can't get this working I'll have to go to a relational database, and I'd rather avoid the hassle.

Thanks in advance.

Comment on Cache::BerkeleyDB Problems Download Code

Replies are listed 'Best First'.

Re: Cache::BerkeleyDB Problems
by kvale (Monsignor) on May 10, 2006 at 03:55 UTC

my $env = new BerkeleyDB::Env
-Cachesize => 131072,  # or larger
...
[download]

-Mark

[reply]
[d/l]

Re^2: Cache::BerkeleyDB Problems

by glwtta (Hermit) on May 10, 2006 at 22:59 UTC

Hmm, ironic. I'll take a look at this, though I'm not quite sure how to make Cache::BerkeleyDB do this.

[reply]

Re^2: Cache::BerkeleyDB Problems

by glwtta (Hermit) on May 10, 2006 at 23:34 UTC

What do I have to keep an eye on that concerns the cache usage? Is it the number of keys? Largest entry? Total size?

Thanks for the help.

[reply]

Re: Cache::BerkeleyDB Problems
by PodMaster (Abbot) on May 10, 2006 at 18:08 UTC

BerkeleyDB DB_RUNRECOVERY

Berkeley DB Reference Guide: Error returns to applications

DB_RUNRECOVERY

There exists a class of errors that Berkeley DB considers fatal to an entire Berkeley DB environment. An example of this type of error is a corrupted database page. The only way to recover from these failures is to have all threads of control exit the Berkeley DB environment, run recovery of the environment, and re-enter Berkeley DB. (It is not strictly necessary that the processes exit, although that is the only way to recover system resources, such as file descriptors and memory, allocated by Berkeley DB.)

Recovery procedures

MJD says "you can't just make shit up and expect the computer to know what you mean, retardo!"
I run a Win32 PPM repository for perl 5.6.x and 5.8.x -- I take requests (README).
** The third rule of perl club is a statement of fact: pod is sexy.

[reply]

Re^2: Cache::BerkeleyDB Problems

by glwtta (Hermit) on May 10, 2006 at 22:57 UTC

Logging doesn't seem to be turned on, any idea how I can do that with Cache::BerkeleyDB?

Thanks!

[reply]

Re: Cache::BerkeleyDB Problems
by perrin (Chancellor) on May 10, 2006 at 20:31 UTC

Cache::FastMmap

[reply]


Your skill will accomplish what the force of many cannot
	PerlMonks