Beefy Boxes and Bandwidth Generously Provided by pair Networks
Just another Perl shrine
 
PerlMonks  

Re^3: Perl Program to efficiently process 500000 small files in a Directory (AIX)

by rminner (Chaplain)
on Mar 17, 2018 at 08:02 UTC ( [id://1211112]=note: print w/replies, xml ) Need Help??


in reply to Re^2: Perl Program to efficiently process 500000 small files in a Directory (AIX)
in thread Perl Program to efficiently process 500000 small files in a Directory (AIX)

Actually it did exactly give this performance gain. I changed the reading mechanism after benchmarking the programm. The benchmarks showed me that more than 90% of the real runtime of my programm was spent on I/O. The files were however larger (a few thousand xml files), and were located on a normal HDD not an SSD. He could simply try and see, whether it changes anything for him.
  • Comment on Re^3: Perl Program to efficiently process 500000 small files in a Directory (AIX)

Replies are listed 'Best First'.
Re^4: Perl Program to efficiently process 500000 small files in a Directory (AIX)
by Anonymous Monk on Mar 17, 2018 at 08:41 UTC
    :) there aint no way the OPs way of slurping is 15 times slower (45/3) than any of the File::Slurp... modules

    yes i believe you saw time reduction ... Whatever part slurp module played was by accident. Its not gonna provide 15x speedup

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://1211112]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others taking refuge in the Monastery: (6)
As of 2024-04-25 15:28 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found