Beefy Boxes and Bandwidth Generously Provided by pair Networks
Think about Loose Coupling
 
PerlMonks  

Re: How to work on a large Data file line by line

by Anonymous Monk
on Jul 08, 2008 at 12:47 UTC ( [id://696200]=note: print w/replies, xml ) Need Help??


in reply to How to work on a large Data file line by line

You can open the file without having all (however many) GB's read into memory. Opening the file just gives you access to it; what you do from that point is up to you. You can read in a file line by line this way:
open FOO, "<my_hug_file.dat"; while (<FOO>) { # reads one line of the file into $_ # do something } close FOO;

Replies are listed 'Best First'.
Re^2: How to work on a large Data file line by line
by tubaandy (Deacon) on Jul 08, 2008 at 16:03 UTC
    I agree with AM, I've found that opening up and reading the file line by line (and operating on a line, then reading in the next line) works well in this case. However, this typically is only useful if all the data you are working with is in that line. If the data is spread across more lines, that's still not as big of a problem. However, I'd go with MidLifeXis' suggestion below if you need to act on the whole file at once. FWIW.

    tubaandy

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://696200]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others making s'mores by the fire in the courtyard of the Monastery: (8)
As of 2024-03-28 11:29 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found