Beefy Boxes and Bandwidth Generously Provided by pair Networks
P is for Practical
 
PerlMonks  

comment on

( [id://3333]=superdoc: print w/replies, xml ) Need Help??
Looks like you need to add more RAM to that machine. Holding all of a huge file in memory when you don't have enough RAM for it is kind of a non-starter.

I'm confused -- you say the file is 4 GB, but you show a listing for a 690 MB file. In any case, if/when you have more memory on the machine, the first option is likely to work best, adding the smallest amount of storage overhead (assuming that internal storage handling in Perl is able to manage a single scalar variable whose length is close to (possibly greater than?) 2**32. (I don't know.)

So perhaps a different/better question to ask is why pass such a huge amount of data as a parameter in a subroutine call? What is the sub supposed to do with that? (If, heaven forbid, it involves making a copy of the data, you may still have a problem.) -- update: As indicated by moritz's reply below, just passing a scalar string as a subroutine arg will create a copy of that string in memory, so you will need a lot more RAM to do that; alternatively, the sub would at least need to accept a reference to a scalar string (but if you're going to change the sub, change it to accept a file handle or name instead...) </update>

Is the sub a piece of code that you wrote? If so, you should consider altering it so that it can use a file handle or file name as its input parameter, and have it handle the file reading in a reasonable way (so the whole file is not stored in memory at all). Or you need to reconsider your algorithm for achieving whatever it is you are trying to achieve. There is generally a way to break it down to work on portions of a large data set, and work around hardware limitations.


In reply to Re: Reading huge file content by graff
in thread Reading huge file content by siva kumar

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":



  • Are you posting in the right place? Check out Where do I post X? to know for sure.
  • Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
    <code> <a> <b> <big> <blockquote> <br /> <dd> <dl> <dt> <em> <font> <h1> <h2> <h3> <h4> <h5> <h6> <hr /> <i> <li> <nbsp> <ol> <p> <small> <strike> <strong> <sub> <sup> <table> <td> <th> <tr> <tt> <u> <ul>
  • Snippets of code should be wrapped in <code> tags not <pre> tags. In fact, <pre> tags should generally be avoided. If they must be used, extreme care should be taken to ensure that their contents do not have long lines (<70 chars), in order to prevent horizontal scrolling (and possible janitor intervention).
  • Want more info? How to link or How to display code and escape characters are good places to start.
Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others meditating upon the Monastery: (4)
As of 2024-04-19 14:09 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found