Re^4: Store a huge amount of data on disk

Thanks, I used that solution for a number of issues in the past, but I'm curious if our filesystem will deal with this number of files an still stay maintainable.

I learned that any overall-operation (like movie, backup) may become arbitrary long and filesystems suffer from heavy file numbers over time.

I thought about merging multiple items into one file (maybe replacing the last directory of your suggestion), but the very different item sizes and text blocks arriving unsorted but still must be finally delivered in their correct order are hard to handle.

Comment on Re^4: Store a huge amount of data on disk

Replies are listed 'Best First'.
Re^5: Store a huge amount of data on disk by BrowserUk (Patriarch) on Oct 19, 2011 at 14:51 UTC
but I'm curious if our filesystem will deal with this number of files an still stay maintainable. I guess the obvious question is: which file system? And can't you use a different one? You should also seriously investigate the various NoSQL DBs around. I guess you were hoping for more feedback on those with your question. but none were forthcoming. With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday' Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice.	[reply]
Re^5: Store a huge amount of data on disk by zentara (Archbishop) on Oct 19, 2011 at 16:34 UTC
but I'm curious if our filesystem will deal with this number of files an still stay maintainable. See maximum number of files in an ext4 linux filesystem. The limit is around 4 billion, 2^32 - 3, if the posts are correct, and maintaining usablility is a matter of handling sub-directory structure properly, setting ionode size, and other filesystem details. It seems doable for you. listing directories with 8 million files may interest you. I'm not really a human, but I play one on earth. Old Perl Programmer Haiku ................... flash japh	[reply]


There's more than one way to do things
	PerlMonks