Beefy Boxes and Bandwidth Generously Provided by pair Networks
XP is just a number
 
PerlMonks  

comment on

( [id://3333] : superdoc . print w/replies, xml ) Need Help??

If you are comparing two files for common/different keys, and if both files are about the same (huge) size, I guess you will have to get smarter than keeping all the information in memory (because you don't have enough memory).

If you can make an educated guess as to where in a file a key is likely to be found, you could use seek to look for the key in the file. This is horribly slow, but likely still faster than swapping memory. If you want to be fancy, you can cache parts of the file in memory.

If you cannot make an educated guess, I guess it will pay off to convert at least one file into a file with all your keys in fixed width, sorted by the keys. Then you can easily make an educated guess to find a given key. If you convert both files to that structure, you can easily find the keys missing in one of the two files by reading through the sorted key files line by line. This approach will roughly double your disk requirements.


In reply to Re: Reduce CPU utilization time in reading file using perl by Corion
in thread Reduce CPU utilization time in reading file using perl by madtoperl

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":



  • Are you posting in the right place? Check out Where do I post X? to know for sure.
  • Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
    <code> <a> <b> <big> <blockquote> <br /> <dd> <dl> <dt> <em> <font> <h1> <h2> <h3> <h4> <h5> <h6> <hr /> <i> <li> <nbsp> <ol> <p> <small> <strike> <strong> <sub> <sup> <table> <td> <th> <tr> <tt> <u> <ul>
  • Snippets of code should be wrapped in <code> tags not <pre> tags. In fact, <pre> tags should generally be avoided. If they must be used, extreme care should be taken to ensure that their contents do not have long lines (<70 chars), in order to prevent horizontal scrolling (and possible janitor intervention).
  • Want more info? How to link or How to display code and escape characters are good places to start.