Pathologically Eclectic Rubbish Lister | |
PerlMonks |
Re: About text file parsing -- MCEby Discipulus (Canon) |
on Aug 29, 2018 at 07:27 UTC ( [id://1221293]=note: print w/replies, xml ) | Need Help?? |
Hello dideod.yang, if your file is huge a line by line processsing will result slow with any variation of the algorithm. But you can throw more CPUs at this with, hopefully, better results. While parallel programming is not so easy to implement correctly in Perl, a gentle monk, marioroy, spent a lot of time and energy to help us, producing MCE and it seems that the second example of the documentation can be easely modified to suit your needs. The example uses MCE::Loop to work on a file in chunks: pay attention to OS dependant implementation inside the mce_loop_f call below and choose the appropriate one for your OS
L* UPDATE you can also be interested in some other tecniques you can find in my library
There are no rules, there are no thumbs.. Reinvent the wheel, then learn The Wheel; may be one day you reinvent one of THE WHEELS.
In Section
Seekers of Perl Wisdom
|
|