Beefy Boxes and Bandwidth Generously Provided by pair Networks
There's more than one way to do things
 
PerlMonks  

Re^2: Perl Program to efficiently process 500000 small files in a Directory (AIX)

by afoken (Canon)
on Mar 17, 2018 at 18:42 UTC ( #1211160=note: print w/replies, xml ) Need Help??


in reply to Re: Perl Program to efficiently process 500000 small files in a Directory (AIX)
in thread Perl Program to efficiently process 500000 small files in a Directory (AIX)

Would it be faster if you do reading and moving using a shell script?

Something like:

for afile in $dir/*; do achannel=$(awk -F'|' '{print $20}' "${afile}") mv "${afile}" "${outdir}/${achannel}" done

Unlikely. Creating an awk and a mv process for each of half a million files sums up to spawning a million processes. I doubt that this idea will be faster than running a single process (perl script), even on an insanely fast machine.

Alexander

--
Today I will gladly share my knowledge and experience, for there are no sweeter words than "I told you so". ;-)

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1211160]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others browsing the Monastery: (3)
As of 2021-04-23 02:24 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found

    Notices?