Perl file operations

SkullOne has asked for the wisdom of the Perl Monks concerning the following question:

I wrote a perl script that does some basic file operations, in this case moving files around based on their creation date.
I have a large folder with 60,000+ files inside, and when I run this particular Perl script, it exits without any output and no errors.
Is there a memory limit or something I could be running into?
Heres a snippet:

for  my $file (<*.txt>)
  { 
      $count = $count + 1;
      
      my ($name,$path,$suffix) = fileparse($file,"\.txt") ;
    my $info = stat($file);
    my $datestamp = strftime(".%Y%m%d", localtime($info->mtime));
    my $datestamp2 = strftime("%Y%m%d", localtime($info->mtime));
    #print "\nMaking Directory if it doesnt exist: $dst2\\$datestamp2"
+;
    mkdir "$dst2\\$datestamp2" or "Error making Directory\n";
    print "\n Moving \"$file\" to >> $dst2\\$datestamp2\n";
    move $file,"$dst2\\$datestamp2\\$name$suffix" or warn "Cannot copy
+ $file $!\n" ;
    if($count > $limit)
    {exit;}
  }
[download]

Comment on Perl file operations Download Code

Replies are listed 'Best First'.
Re: Perl file operations by kyle (Abbot) on Feb 23, 2008 at 03:22 UTC
I did `perl -le 'print for <>' \| wc -l`, and it listed out over 100000 files (after a while). I don't think the size of your directory is the problem. It still could* be the problem if your memory is very limited, but I'd expect you to get a message about that explicitly if that were the case. (The file globbing you do builds up a list of all those 60000+ files in memory at once.) The problem may be that after mkdir, you `or` into a string. The string doesn't do anything. You may want: `mkdir "$dst2\\$datestamp2" or die "Can't mkdir '$dst2\\$datestamp2': $!\n";` [download] After that, I expect the `move` to fail too, but the warn there should be telling you why. If it's not, I wonder if STDERR is being redirected, or there's a `$SIG{__WARN__}` handler or something. I'd make your end-on-limit more explosive: `if ( $count > $limit ) { die "limit exceeded: $count files processed"; }` [download] Generally I recommend setting `$limit` to about 10 and sprinkle a lot of print statements in the loop to make sure all those variables have the values you think they do.	[reply] [d/l] [select]
Re: Perl file operations by jwkrahn (Abbot) on Feb 23, 2008 at 03:23 UTC
While I don't have a directory with 60,000 files in it to test your code I would guess that `<*.txt>` is creating such a large list that it runs out of memory. Try using opendir and then readdir in a while loop to cut down on memory usage: `opendir my $DH, '.' or die "Cannot open the current directory: $!"; while ( my $file = readdir $DH ) { next if '.txt' ne substr $file, -4;` [download]	[reply] [d/l] [select]
Re^2: Perl file operations by chromatic (Archbishop) on Feb 23, 2008 at 04:42 UTC
While I don't have a directory with 60,000 files in it to test your code I would guess that `<.txt>` is creating such a large list that it runs out of memory.* At an estimate of 64 bytes of overhead per SV, 60,000 strings produced by that list will take up 3.6 megabytes of memory, not counting the lengths of the filenames.	[reply] [d/l]
Re^3: Perl file operations by SkullOne (Acolyte) on Feb 23, 2008 at 05:54 UTC
Monks, I am stupid. The file extensions in the directory were different that I was expecting. I fixed the file extension, and it slowly but surely processed the 60,000+ files into their respective directories. I watched memory usage on the successfull run, and it hovered between 60 and 70 megabytes, using Activeperl. Please dont harm me :(	[reply]
Re^4: Perl file operations by zshzn (Hermit) on Feb 23, 2008 at 21:41 UTC


Problems? Is your data what you think it is?
	PerlMonks