Reporting "percent done" while processing a large file

mbond has asked for the wisdom of the Perl Monks concerning the following question:

Replies are listed 'Best First'.
Re: status by MZSanford (Curate) on Sep 18, 2001 at 16:50 UTC
I am a huge proponent of sttaus messages, but with text files, to do a percent you would have to first get the number of lines (which on large data is a waste). I tend to do the following for text data : `while (my $line = <INFILE>) { if ( ($. % 5000) == 0) { print "Processed $. records\n"; # or, just to let them know # print "."; } ## process away }` [download] Now, being Mr. Binary-data-can-be-done-with-perl, i tend to do alot of binary op's on files, and for that you can do (this is untested, but mostly just to show the idea): `my $TOTAL = (-s $FILE); while (read(INFILE,$buf,$bufsize)) { ## do work $BYTES_READ+=$bufsize; if ( ($BYTES_READ % 5000) == 0) { printf("%.3f %% (%d of %d)\n",($BYTES_READ/$TOTAL)*100,$BYTES_ +READ,$TOTAL); } }` [download] _{my own worst enemy} -- MZSanford	[reply] [d/l] [select]
Re: Re: status by Masem (Monsignor) on Sep 18, 2001 at 17:00 UTC
Well, with text, you can still use the latter solution as to not determine the number of lines beforehand if the file is not a fixed record file: `my $total = ( -s $FILE ); my $cummulative = 0; while ( my $line = <FILE> ) { $cummulative += length( $line ); do_processing( $line ); printf( "%2d%% Done\n", $cummulative/$total100.0 ); }` [download] ----------------------------------------------------- Dr. Michael K. Neylon - mneylon-pm@masemware.com \|\| "You've left the lens cap of your mind on again, Pinky" - The Brain It's not what you know, but knowing how to find it if you don't know that's important*	[reply] [d/l]
Re: Re: status by blakem (Monsignor) on Sep 18, 2001 at 21:31 UTC
Perhaps its just me, but I always find the construct: `if ( ( whatever ) == 0 ) { }` [download] easier to read, when written like so: `unless ( whatever ) { }` [download] So, I would have written the above as: `unless ($. % 5000) { print "Processed $. records\n"; # or, just to let them know # print "."; }` [download] -Blake off to try that cool -s trick for status bars....	[reply] [d/l] [select]
Re: Reporting "percent done" while processing a large file by clintp (Curate) on Sep 18, 2001 at 20:06 UTC
Whether you're processing text or binary, the point is to show some sort of progress towards the end goal. Now for text this kind of assumes that the lines are somewhat balanced (the standard deviation of their lengths isn't outrageous). You don't need to be too exact though, and tell() can be your friend: `open(F, "/usr/dict/words") \|\| warn; # File to process $s=-s F; $\|=1; while(<F>) { # Binary read() or readline()...whatever. # Do something with the data I presume... printf "\rComplete %.2f", 100tell(F)/$s; }` [download] For large files this slows things down considerably. You may want to try: `printf("\rComplete: %.2f", 100tell(F)/$s) unless $i++ % 10;` [download] Or something like it so there's not quite so much output.	[reply] [d/l] [select]
Re: Reporting "percent done" while processing a large file by archen (Pilgrim) on Sep 19, 2001 at 00:00 UTC
Well my Perl skills are pretty weak, but here's what I usually do: I make a cheesy text status bar. More or less print a line that sort of marks where the progress is, then print an '*' for something like every 5% done. Usually I just use the `stat` function to find the file size, and divide by the number of astriks you want in the bar, etc (this similar to a computer science problem I once had in college). There is some computational overhead with comparisons with an accumulator, but as far as I can tell, not a whole lot. (I'd post some code, but truthfully my code looks pretty sad).	[reply]