Beefy Boxes and Bandwidth Generously Provided by pair Networks
go ahead... be a heretic

Re: About text file parsing

by SuicideJunkie (Vicar)
on Aug 29, 2018 at 17:42 UTC ( #1221316=note: print w/replies, xml ) Need Help??

in reply to About text file parsing

XY problem question here...
What are you going to be using those arrays for? Are they huge, or is the sample/good a small subset of the input data? There may be better ways to approach the entire task.

Perhaps consider something more like this:

use strict; use warnings; open my $ifh, '<', '' or die; open my $ofh_samples, '>', 'samples.txt' or die; open my $ofh_good, '>', 'goodlines.txt' or die; $|=1; my $total = -s ''; my $progress = 0; my $linecount = 0; while (my $line = <$ifh>) { $linecount++; $progress += length ($line); print $ofh_samples $1 if $line =~ /^sample\s+(\S+)/; print $ofh_good $1 if $line =~ /^good\s+(\S+)/; printf "Processing... %3.1f%% completed... \r", 100*$progress/$tot +al unless $linecount %100; }
That will keep little more than one line in memory at a time, and you can then deal with the pieces separately. Uncomment the comments to give a progress display.

Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://1221316]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others musing on the Monastery: (2)
As of 2021-10-24 23:08 GMT
Find Nodes?
    Voting Booth?
    My first memorable Perl project was:

    Results (89 votes). Check out past polls.