http://qs321.pair.com?node_id=1221316


in reply to About text file parsing

XY problem question here...
What are you going to be using those arrays for? Are they huge, or is the sample/good a small subset of the input data? There may be better ways to approach the entire task.

Perhaps consider something more like this:

use strict; use warnings; open my $ifh, '<', 'test.pl' or die; open my $ofh_samples, '>', 'samples.txt' or die; open my $ofh_good, '>', 'goodlines.txt' or die; $|=1; my $total = -s 'test.pl'; my $progress = 0; my $linecount = 0; while (my $line = <$ifh>) { $linecount++; $progress += length ($line); print $ofh_samples $1 if $line =~ /^sample\s+(\S+)/; print $ofh_good $1 if $line =~ /^good\s+(\S+)/; printf "Processing... %3.1f%% completed... \r", 100*$progress/$tot +al unless $linecount %100; }
That will keep little more than one line in memory at a time, and you can then deal with the pieces separately. Uncomment the comments to give a progress display.