Beefy Boxes and Bandwidth Generously Provided by pair Networks
more useful options
 
PerlMonks  

Re: Perl Program to efficiently process 500000 small files in a Directory (AIX)

by bliako (Prior)
on Mar 17, 2018 at 17:57 UTC ( #1211156=note: print w/replies, xml ) Need Help??


in reply to Perl Program to efficiently process 500000 small files in a Directory (AIX)

Would it be faster if you do reading and moving using a shell script?

Something like:

for afile in $dir/*; do achannel=$(awk -F'|' '{print $20}' "${afile}") mv "${afile}" "${outdir}/${achannel}" done
  • Comment on Re: Perl Program to efficiently process 500000 small files in a Directory (AIX)
  • Download Code

Replies are listed 'Best First'.
Re^2: Perl Program to efficiently process 500000 small files in a Directory (AIX)
by afoken (Canon) on Mar 17, 2018 at 18:42 UTC

    Would it be faster if you do reading and moving using a shell script?

    Something like:

    for afile in $dir/*; do achannel=$(awk -F'|' '{print $20}' "${afile}") mv "${afile}" "${outdir}/${achannel}" done

    Unlikely. Creating an awk and a mv process for each of half a million files sums up to spawning a million processes. I doubt that this idea will be faster than running a single process (perl script), even on an insanely fast machine.

    Alexander

    --
    Today I will gladly share my knowledge and experience, for there are no sweeter words than "I told you so". ;-)

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1211156]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others about the Monastery: (3)
As of 2021-04-19 22:03 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found

    Notices?