Beefy Boxes and Bandwidth Generously Provided by pair Networks
XP is just a number
 
PerlMonks  

Re: Need to speed up many regex substitutions and somehow make them a here-doc list

by xnous (Sexton)
on Nov 14, 2022 at 22:04 UTC ( [id://11148178] : note . print w/replies, xml ) Need Help??


in reply to Need to speed up many regex substitutions and somehow make them a here-doc list

Hello, again. I apologize for the long absence, real life and such.

I extensively tested both Marshall's fork implementation and marioroy's MCE "entire-chunk" implementation and, depending on the test case, the results are very variable and very interesting.

Test case 1, a few thousands of <1MB files: fork.pl->2.2", mce.pl->9.4". The MCE one couldn't saturate my 4C/8T CPU, load was at around 60%.

Test case 2, the above files concatenated into a 30MB one: fork.pl->3.2", mce.pl->1.8". Here, the fork one left the CPU mostly idle.

Test case 3, 200 files sized 1.4MB each: fork.pl->15.2", mce.pl->25.7".

In general, MCE can't keep the CPU 100% busy, even with max_workers set at 2X the thread count. Also, chunk_size can make a big difference depending on the size of the files processed. For large single-file input, MCE simply dominates; for lots of small files, the fork implementation is the undisputed winner.

Most importantly, you people proved that Perl, in the right hands of course, can outperform almost everything and I'd like to thank every monk who shared his wisdom with me in this thread. My OP question has been fully answered.

  • Comment on Re: Need to speed up many regex substitutions and somehow make them a here-doc list