in reply to Re: Perl solution for storage of large number of small files
in thread Perl solution for storage of large number of small files
I do *almost* the same exact thing, for a similarly large (and growing) archive
I do:
agreed on the notion that "The distributive properties of MD5 garantuee a good spread among buckets.". You should never do this with numbers, but if you must, you can reverse the numbers to get a decent distribution ( ie - 12345 -> 54321 ) another difference that i do is the following-- i name the file the md5, and store the original file name in postgres. that gives me a speedup on fileserving. its relatively simple to create rewrite rules under apache & nginx to directly serve these files too./123/456/789/123456789...32 or /123/456/789/entire-hash
In Section
Seekers of Perl Wisdom