My solution would be:
- Keep a list of file handles, but limit the number of file handles you could open at the same time, say 100.
- For each open file, keep a timestamp on it. Update the timestamp, everytime when you write to it.
- If you got a line, and its related file is open, fine, just write out whatever you want.
- If the related file is not open, go thru the list, close the one has the longest unused time. open the one you wanted.
Well, you can come up other ways, other than the last access time, to rank your files. For example, number of lines wrote. You have to try them out, and find the best way of ranking.