in reply to Reading huge file content
I think you'd better rewrite your function so it can read directly from the filehandle. Reading Gigs of data is no fun, it can easily take many, many minutes. And then it hasn't done anything yet.
|
Re^2: Reading huge file content
by downer (Monk) on Dec 06, 2007 at 14:02 UTC
|
in my experience, perl has some problems reading vary large amounts of data into a single variable. it may be better if you can load each line into an entry of an array, (if you've got plenty of memory) or better yet, try to just process a line at a time. For very large jobs like this, I find it preferrable to take one large job and make it in to several small jobs which usually just scan through the file. its probably work while to spend some time thinking of your algorithm to see if this is possible. | [reply] |
|
|
or better yet, try to just process a line at a time.
Indeed. Especially since even reading this much data (never mind processing it) will take considerable time. Reading one line, processing it, then reading the next would also allow you to store a counter on disk somewhere or displayed on the screen with the number of bytes successfully read and processed (obtained using tell) so that, if the process dies partway through, it can pick up from where it left off when restarted instead of having to go back to the beginning of the file and repeat what's already been done.
| [reply]
[d/l] |