Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl-Sensitive Sunglasses
 
PerlMonks  

Re: Reading huge file content

by bart (Canon)
on Dec 06, 2007 at 13:44 UTC ( #655399=note: print w/replies, xml ) Need Help??


in reply to Reading huge file content

I think you'd better rewrite your function so it can read directly from the filehandle. Reading Gigs of data is no fun, it can easily take many, many minutes. And then it hasn't done anything yet.

Replies are listed 'Best First'.
Re^2: Reading huge file content
by downer (Monk) on Dec 06, 2007 at 14:02 UTC
    in my experience, perl has some problems reading vary large amounts of data into a single variable. it may be better if you can load each line into an entry of an array, (if you've got plenty of memory) or better yet, try to just process a line at a time. For very large jobs like this, I find it preferrable to take one large job and make it in to several small jobs which usually just scan through the file. its probably work while to spend some time thinking of your algorithm to see if this is possible.
      or better yet, try to just process a line at a time.

      Indeed. Especially since even reading this much data (never mind processing it) will take considerable time. Reading one line, processing it, then reading the next would also allow you to store a counter on disk somewhere or displayed on the screen with the number of bytes successfully read and processed (obtained using tell) so that, if the process dies partway through, it can pick up from where it left off when restarted instead of having to go back to the beginning of the file and repeat what's already been done.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://655399]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others scrutinizing the Monastery: (3)
As of 2022-01-27 06:15 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    In 2022, my preferred method to securely store passwords is:












    Results (70 votes). Check out past polls.

    Notices?