Re^2: Perl Program to efficiently process 500000 small files in a Directory (AIX)


The stupid question is the question not asked
	PerlMonks

by Anonymous Monk

on Mar 17, 2018 at 07:46 UTC ( [id://1211106]=note: print w/replies, xml )

Need Help??

File slurp cannot make that kind of speed improvement

Comment on Re^2: Perl Program to efficiently process 500000 small files in a Directory (AIX)

Replies are listed 'Best First'.
Re^3: Perl Program to efficiently process 500000 small files in a Directory (AIX) by rminner (Chaplain) on Mar 17, 2018 at 08:02 UTC
Actually it did exactly give this performance gain. I changed the reading mechanism after benchmarking the programm. The benchmarks showed me that more than 90% of the real runtime of my programm was spent on I/O. The files were however larger (a few thousand xml files), and were located on a normal HDD not an SSD. He could simply try and see, whether it changes anything for him.	[reply]
Re^4: Perl Program to efficiently process 500000 small files in a Directory (AIX) by Anonymous Monk on Mar 17, 2018 at 08:41 UTC
:) there aint no way the OPs way of slurping is 15 times slower (45/3) than any of the File::Slurp... modules yes i believe you saw time reduction ... Whatever part slurp module played was by accident. Its not gonna provide 15x speedup	[reply]

Domain Nodelet^?

Node Status^?

node history
Node Type: note [id://1211106]
help

Chatterbox^?

How do I use this? • Last hour • Other CB clients

Other Users^?

Others browsing the Monastery: (3)

As of 2024-04-19 21:37 GMT

Sections^?

Information^?

Find Nodes^?

Leftovers^?

Voting Booth^?

No recent polls found