Beefy Boxes and Bandwidth Generously Provided by pair Networks
Pathologically Eclectic Rubbish Lister
 
PerlMonks  

Re^2: Perl solution for storage of large number of small files

by 2xlp (Sexton)
on May 03, 2007 at 02:37 UTC ( [id://613306]=note: print w/replies, xml ) Need Help??


in reply to Re: Perl solution for storage of large number of small files
in thread Perl solution for storage of large number of small files

I do *almost* the same exact thing, for a similarly large (and growing) archive I do:
/123/456/789/123456789...32 or /123/456/789/entire-hash
agreed on the notion that "The distributive properties of MD5 garantuee a good spread among buckets.". You should never do this with numbers, but if you must, you can reverse the numbers to get a decent distribution ( ie - 12345 -> 54321 ) another difference that i do is the following-- i name the file the md5, and store the original file name in postgres. that gives me a speedup on fileserving. its relatively simple to create rewrite rules under apache & nginx to directly serve these files too.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://613306]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others lurking in the Monastery: (3)
As of 2024-04-16 21:56 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found