http://qs321.pair.com?node_id=1211156


in reply to Perl Program to efficiently process 500000 small files in a Directory (AIX)

Would it be faster if you do reading and moving using a shell script?

Something like:

for afile in $dir/*; do
  achannel=$(awk -F'|' '{print $20}' "${afile}")
  mv "${afile}" "${outdir}/${achannel}"
done

[download]
  • Comment on Re: Perl Program to efficiently process 500000 small files in a Directory (AIX)
  • Download Code

Replies are listed 'Best First'.
Re^2: Perl Program to efficiently process 500000 small files in a Directory (AIX)
by afoken (Chancellor) on Mar 17, 2018 at 18:42 UTC

    Would it be faster if you do reading and moving using a shell script?

    Something like:

    for afile in $dir/*; do
  achannel=$(awk -F'|' '{print $20}' "${afile}")
  mv "${afile}" "${outdir}/${achannel}"
done

    [download]

    Unlikely. Creating an awk and a mv process for each of half a million files sums up to spawning a million processes. I doubt that this idea will be faster than running a single process (perl script), even on an insanely fast machine.

    Alexander

    --
    Today I will gladly share my knowledge and experience, for there are no sweeter words than "I told you so". ;-)
[reply]
[d/l]
[select]

In Section Seekers of Perl Wisdom