It sounds like there are multiple collections in use at the same time, so it should be worth storing all in files for one collection in it's own directory. That will make examining the files quicker. But if there are thousands of files then sub directories could be help too.

Hello RichardK, friends,

Sorry for answering via a shotgun message. First, again thanks for your time and help. After I posted my first reply I began playing around with http://search.cpan.org/~cleishman/Cache-2.04/ and fell in love with it. It was simple and handled the metadata caching format nicely. Collection entries could be pushed/popped as hash key=>value pairs. It also handled file locking and provided many methods to do all of the things I needed to do. Unfortunately I found out later from my boss that not only are Dbases not allowed, but any Perl Module that is not a Perl5 core module cannot be used either. Mulligan!

Regarding the heap vs files debate; I learned that the required level of persistence is actually quite high, certainly high enough to warrant the use of a Dbase if that was an option. Essentially collections will be kept indefinitely. That is the reason I chose to use files. I also found out for certain that I could not modify file names. As of now I plan on creating a pseudo-namespace for each collection by throwing collection metadata and files unique directories.

Cheers,
Hok

P.S. I used a lot of buzzwords and somehow left out "Cloud" so there I said it.