Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl: the Markov chain saw

Re: Reading huge file content

by graff (Chancellor)
on Dec 06, 2007 at 13:08 UTC ( #655389=note: print w/replies, xml ) Need Help??

in reply to Reading huge file content

Looks like you need to add more RAM to that machine. Holding all of a huge file in memory when you don't have enough RAM for it is kind of a non-starter.

I'm confused -- you say the file is 4 GB, but you show a listing for a 690 MB file. In any case, if/when you have more memory on the machine, the first option is likely to work best, adding the smallest amount of storage overhead (assuming that internal storage handling in Perl is able to manage a single scalar variable whose length is close to (possibly greater than?) 2**32. (I don't know.)

So perhaps a different/better question to ask is why pass such a huge amount of data as a parameter in a subroutine call? What is the sub supposed to do with that? (If, heaven forbid, it involves making a copy of the data, you may still have a problem.) -- update: As indicated by moritz's reply below, just passing a scalar string as a subroutine arg will create a copy of that string in memory, so you will need a lot more RAM to do that; alternatively, the sub would at least need to accept a reference to a scalar string (but if you're going to change the sub, change it to accept a file handle or name instead...) </update>

Is the sub a piece of code that you wrote? If so, you should consider altering it so that it can use a file handle or file name as its input parameter, and have it handle the file reading in a reasonable way (so the whole file is not stored in memory at all). Or you need to reconsider your algorithm for achieving whatever it is you are trying to achieve. There is generally a way to break it down to work on portions of a large data set, and work around hardware limitations.

Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://655389]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others meditating upon the Monastery: (1)
As of 2022-01-28 06:55 GMT
Find Nodes?
    Voting Booth?
    In 2022, my preferred method to securely store passwords is:

    Results (73 votes). Check out past polls.