Re: Perl Program to efficiently process 500000 small files in a Directory (AIX)


Perl Monk, Perl Meditation
	PerlMonks

Re: Perl Program to efficiently process 500000 small files in a Directory (AIX)

by bliako (Monsignor)

on Mar 17, 2018 at 17:57 UTC ( [id://1211156]=note: print w/replies, xml )

Need Help??

in reply to Perl Program to efficiently process 500000 small files in a Directory (AIX)

Would it be faster if you do reading and moving using a shell script?

Something like:

for afile in $dir/*; do
  achannel=$(awk -F'|' '{print $20}' "${afile}")
  mv "${afile}" "${outdir}/${achannel}"
done
[download]

Comment on Re: Perl Program to efficiently process 500000 small files in a Directory (AIX) Download Code

Replies are listed 'Best First'.

Re^2: Perl Program to efficiently process 500000 small files in a Directory (AIX)
by afoken (Chancellor) on Mar 17, 2018 at 18:42 UTC

Would it be faster if you do reading and moving using a shell script?

Something like:
for afile in $dir/*; do
  achannel=$(awk -F'|' '{print $20}' "${afile}")
  mv "${afile}" "${outdir}/${achannel}"
done
[download]

Unlikely. Creating an awk and a mv process for each of half a million files sums up to spawning a million processes. I doubt that this idea will be faster than running a single process (perl script), even on an insanely fast machine.

Alexander

--
Today I will gladly share my knowledge and experience, for there are no sweeter words than "I told you so". ;-)

[reply]
[d/l]
[select]

In Section Seekers of Perl Wisdom