Beefy Boxes and Bandwidth Generously Provided by pair Networks
Just another Perl shrine
 
PerlMonks  

Re: sorting very large text files

by talexb (Chancellor)
on Dec 21, 2009 at 18:58 UTC ( [id://813746]=note: print w/replies, xml ) Need Help??


in reply to sorting very large text files

How about splitting the large file into smaller pieces, sorting the individual pieces, then merging those pieces into a single sorted file?

The disadvantage is that you'd need at least double the original file size in free space, but you might mind that the total CPU time would be less, since merging would be fairly fast.

There are lots of cool ways to do it:

  • Split the original file into 2/4/8 pieces based on file size;
  • Split the original file into 2/4/8 pieces by doing a round-robin choice on each line;
  • Sort into individual files based on the first 1/2/3 characters in each line;
.. and there are lots more possibilities .. that would be a fun project.

Alex / talexb / Toronto

Team website: Forex Chart Monkey, Forex Technical Analysis and Pickpocket Prevention

"Groklaw is the open-source mentality applied to legal research" ~ Linus Torvalds

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://813746]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others perusing the Monastery: (4)
As of 2024-03-29 07:57 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found