You are Suffering from Buffering.

The brief answer is -- make your standard output (STDOUT) flush immediately. To do so, add this line near the beginning of your script (before doing any print statments):

    $| = 1;
[download]

The corresponding section from the article Corion linked to is the one entitled "Disabling Inappropriate Buffering", where it talks about making the "filehandle hot".

If you have use for a really fancy ascii-oriented progress bar, here is something I wrote that I've found useful in a lot of different applications:

#
#  progress_bar
#
#  Input:  $1 - A message to display with the progress bar.  It can co
+ntain
#               any of the following special 3-character strings, each
+ of which
#               will be converted to the following value:
#
#                   <b> - the progress bar itself
#                   <c> - the current count
#                   <p> - the percentage finished
#                   <r> - estimated remaining time
#                   <t> - the total count
#
#          $2 - A total count
#          $3 - (optional) The 'done' symbol (default is '@')
#          $4 - (optional) The 'todo' symbol (default is '-')
#
#  Output:  A closure that takes a count, and updates the progress bar
+ based
#           on its value.  When the progress bar is done, it should be
+ called
#           with no arguments, to show that the progress is complete -
+and- to
#           print a final newline.
##
sub progress_bar {
    my ($msg, $total, $done_sym, $todo_sym) = @_;

    ($total || 0) or fatal("total can NOT be zero/undefined!");
    $done_sym ||= '@';
    $todo_sym ||= '-';
    $| = 1;

    my $start  = time();
    my $last   = $start;
    my $remain = "???";
    my $b_left = ($msg =~ /<r>/)? 1: 0;

    my $c_prog = sub {
        my ($count) = @_;
        my $b_done = defined($count)? 0: 1;
        $b_done and $count = $total;
        ($count > $total) and $count = $total;
        my $pcnt    = sprintf "%6.2f", 100.0 * $count / $total;
        my $new_msg = $msg;

        # Calculate estimated remaining time
        my $time  = time();
        my $dtime = $time - $start;
        if ($b_done) {
            $remain = "Finished";
        } elsif ($b_left and $count and $dtime and $time - $last > 3) 
+{
            $last    = $time;
            my $rate = ($count / $dtime);
            my $nsec = int(($total - $count) / $rate);
            my $hr   = my $min = 0;
            if ($nsec > 3600) { $hr  = int($nsec / 3600); $nsec -= 360
+0 * $hr }
            if ($nsec > 60)   { $min = int($nsec / 60);   $nsec -= 60 
+* $min  }
            if ($hr) {
                $remain = sprintf "%02d:%02d:%02d", $hr, $min, $nsec;
            } elsif ($min) {
                $remain = sprintf "%02d:%02d", $min, $nsec;
            } else {
                my $s   = (1 == $nsec)? "": "s";
                $remain = "$nsec second$s";
            }
        }

        $new_msg =~ s/<c>/$count/g;
        $new_msg =~ s/<p>/$pcnt/g;
        $new_msg =~ s/<t>/$total/g;
        $new_msg =~ s/<r>/$remain/g;
        my $len   = 79 + 3 - length($new_msg);
        my $ndone = int($len * $count / $total);
        my $ntodo = $len - $ndone;
        my $bar   = ($done_sym x $ndone) . ($todo_sym x $ntodo);
        $new_msg =~ s/<b>/$bar/;
        print "$new_msg\r";
        $b_done and print "\n";
    };

    return $c_prog;
}
[download]

You can call it like this ... let's say you were processing a bunch of files, and wanted progress indication for each file processed:

use strict;
use warnings;

my @files    = ( 'some', 'list', 'of', 'files' );
my $nfiles   = @files;
my $nhandled = 0;
my $c_prog = progress_bar("File #<c> of <t> [<p>%] <b>", $nfiles);
foreach my $file (@files) {
    do_something_with_this_file($file);
    $c_prog->(++$nhandled);
}

$c_prog->();
[download]

Note that you don't want to print anything while the progress bar is displaying, or it will disrupt the progress bar's output. :)

Subroutine do_something_with_this_file left as an exercise to the reader.

use Term::ProgressBar;
my $file = ...;
my ( $line_count ) = qx{ wc -l $file } =~ /^(\d+)/;
my $pbar = Term::ProgressBar->new({count => $line_count });
open my $fh, '<', $file or die "$file: $!";
my $i = 0;
while (<$fh>) {
    chomp;
    $pbar->update($i++);

    ...
}
[download]

$|=1; or $|++; will turn off buffering globally and that might not be what you want because it will slow down disk writes. I guess this could be called a "cheater method", but if you 'print STDERR "progress\n"', that output shows up immediately on the console because stderr is not buffered (stdout is). Alternatively, you can call flush() on a filehandle to flush the buffers.

But for console commands that take awhile, consider just printing the status to STDERR instead of the default print of STDOUT. Of course there are some "yeah, but's" with that which are application specific.

I do congratulate you on the idea of presenting an indication of progress that actually means something! I hate this Windows graphic like when you copy files...it is a separate doo-dad that has nothing at all do with any "real" progress - the real program can hang and that thing merrily goes along with its graphical fake out.

If you wind up calling some progress bar graphical thing, be aware that graphical output is "expensive" CPU wise. Call it only when necessary (ie you expect that the progress bar will actually move on the screen).

As another point, the first time I saw this in Windows, I was really surprised because I was used to O/S'es that printed to the console so fast that you could hardly tell that it happened! But Windows is slow when it prints to the console. I don't know why that is. But that appears to be the case. And sometimes the program speeds up quite a bit when you print less.

I also seriously question the wisdom, and the necessity, of buffering “from 1K to 1 million lines” in memory.

“Why is {the virtual} memory {paging file} superior to the file that you are already reading?”

Also:   is this logfile a file of a well-known format, such as an Apache server log, for which a suitable CPAN module already exists?   Might you, quite without intending to do so, be “doing a thing already done?”   Aye, ’tis dreadfully easy to do...

Thanks for the replies. I tried replying when last week, but the website just times out.



I also seriously question the wisdom, and the necessity, of buffering 
+“from 1K to 1 million lines” in memory. 


“Why is {the virtual} memory {paging file} superior to the file that y
+ou are already reading?” 

Also:   is this logfile a file of a well-known format, such as an Apac
+he server log, for which a suitable CPAN module already exists?   Mig
+ht you, quite without intending to do so, be “doing a thing already d
+one?”   Aye, ’tis dreadfully easy to do...
[download]



It is a log generated by a machine every time it does an action. It is a standard english text file, but unique in the sense of the text sequences it creates. If there is a better way then doing an OPEN on the file and reading the contents to a Array then I am ears. I need to be able to read each line of the file to look for certain sequences in each of those lines. In a file that has say 100,000 lines, it processes the array in like 8 seconds....so I'm not longing for performance just yet.