Beefy Boxes and Bandwidth Generously Provided by pair Networks
The stupid question is the question not asked
 
PerlMonks  

Re^4: Looking for ways to speed up the parsing of a file...

by sgifford (Prior)
on May 18, 2008 at 18:59 UTC ( [id://687239]=note: print w/replies, xml ) Need Help??


in reply to Re^3: Looking for ways to speed up the parsing of a file...
in thread Looking for ways to speed up the parsing of a file...

Instead of actually splitting the file into several additional files, you could just determine the positions as you describe, then work on the different parts by seeking to the right position before starting your processing loop. For example, you could determine the start and end position and then fork() off a new process to work on that chunk. Reading from multiple files (or different places in the same file) in parallel might end up being less efficient from an I/O perspective, though, as it could require the drive to seek a lot more. So you'd need to experiment a bit to find the right way to parallelize this.
  • Comment on Re^4: Looking for ways to speed up the parsing of a file...

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://687239]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others about the Monastery: (9)
As of 2024-04-23 11:05 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found