Beefy Boxes and Bandwidth Generously Provided by pair Networks
No such thing as a small change
 
PerlMonks  

Re^3: Perl Program to efficiently process 500000 small files in a Directory (AIX)

by Marshall (Canon)
on Mar 17, 2018 at 19:00 UTC ( #1211161=note: print w/replies, xml ) Need Help??


in reply to Re^2: Perl Program to efficiently process 500000 small files in a Directory (AIX)
in thread Perl Program to efficiently process 500000 small files in a Directory (AIX)

move is part of File::Copy and can work across file systems. rename is a built in Perl function and cannot work across file systems. Often more restrictions means faster. This is easy enough to try, that I'd just try it and see the benchmark results. I would imagine that just processing the first 50,000 files would give you enough of a performance idea when comparing various alternatives. The if statement to stop after 50,000 files will make no speed difference.

Update..The rename should be faster because the actual data bits don't have to be moved - just a directory modification. An actual copy would move the data bits to a new location on the disk - that is way slower. I am also not sure that in this case slurping the file in is best? It sounds like although there are few lines, they are long lines. You could also benchmark letting the filesystem do the line division for you, throw away the first 5 lines and only run the regex on the 6th line. My thinking here is that the line division probably uses the C index function which is faster than the regex engine. Also there apparently is no need to process the rest of the lines. The overall effect might (or not) be a speed increase. I again suggest using say 10% of the data for testing so that you can test 4-5 scenarios within a couple of hours.

  • Comment on Re^3: Perl Program to efficiently process 500000 small files in a Directory (AIX)

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1211161]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others rifling through the Monastery: (9)
As of 2021-04-16 12:22 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found

    Notices?