in reply to Re^2: Perl Program to efficiently process 500000 small files in a Directory (AIX)
in thread Perl Program to efficiently process 500000 small files in a Directory (AIX)
Update..The rename should be faster because the actual data bits don't have to be moved - just a directory modification. An actual copy would move the data bits to a new location on the disk - that is way slower. I am also not sure that in this case slurping the file in is best? It sounds like although there are few lines, they are long lines. You could also benchmark letting the filesystem do the line division for you, throw away the first 5 lines and only run the regex on the 6th line. My thinking here is that the line division probably uses the C index function which is faster than the regex engine. Also there apparently is no need to process the rest of the lines. The overall effect might (or not) be a speed increase. I again suggest using say 10% of the data for testing so that you can test 4-5 scenarios within a couple of hours.
|
---|