Hi oryan, welcome to the monastery. It's not a soup kitchen.

Your problem would be trivial to solve in Perl, and you could implement a solution that split the job into chunks that could be worked on simultaneously by several processes. However, you are unlikely to be handed a working solution for free.

The first question that springs to my mind is whether the sets of lines can overlap, or whether it is guaranteed that one set completes before the next matching set begins. If it is the former, the job will be more complicated.

In any case, using very standard Perl techniques for opening files, looping through them, matching patterns and writing out the results, you could learn how to do this in short order.

Here are some links to the documentation to get you started:
Introduction to Perl
open a file
read ines from a file
split a line into chunks
pattern matching quick start
write to a filehandle (and more on opening and closing files)
Data::Dumper, an indispensable tool for development
The Basic Debugging Checklist by toolic
How to post an effective question on PerlMonks after you have some code written

Good luck, hope this helps!

a number of people will tell you that this site is not a free code-writing service and that you should show some efforts at doing things yourself, and I agree.

Having said that, even if you did not show any Perl code, I appreciate that you have shown code in another scripting language. I consider this as a real effort. And, although I do not have any benchmark, I am fairly sure that a Perl program will be much much faster than a DOS batch script (well, you don't say it, but I guess your script is DOS batch).

So, you should really look at the links that were provided to you, but I am willing to help and I can give you the skeleton of a Perl program for what you want (quick draft, untested):

use strict;
use warnings;

my $input = "input.txt";
open my $IN, "<", $input or die "cannot open $input $!";
my $output = "output.txt";
open my $OUT, ">", $output or die "cannot open $output $!";
while (my $line = <$IN>) {
    chomp $line; # remove newline character from end of line
    if ($line =~ /INTERPOLATED HYDROGRAPH AT (\w+)$/){
        print $OUT $1;
        next for 1..5; # skip 5 lines
        my $val2 = (split / /, $line)[1]; get the second column
        print $OUT $val2; # a separator may be needed here
        next; # skip one line
        my $val3 = (split / /, $line)[-1]; # get the last column
        print $OUT "$val3\n"; # a separator may be needed here
    }
}
[download]

Update: There are two errors in the code above (thanks to poj for pointing them out. Please see below (Re^3: Perl solution for current batch file to extract specific column text) a corrected version.

This prints line 1 twice


#!perl
use strict;
while (my $line = <DATA>){
  if ($line =~ /line 1/){
    print $line;
    next for 1..5; # skip 5 lines
    print $line;
  }
}
__DATA__
line 1
line 2
line 3
line 4
line 5
line 6
line 7
[download]

Perhaps you meant

#!perl
use strict;
while (my $line = <DATA>){
  if ($line =~ /line 1/){
    print $line;
    <DATA> for 1..5;
    $line = <DATA>;
    print $line;
  }
}
[download]

poj

Yes, poj, you're right, there were two mistakes.

This is the amended (and tested) version:

use strict;
use warnings;

my $input = "input.txt";
open my $IN, "<", $input or die "cannot open $input $!";
my $output = "output.txt";
open my $OUT, ">", $output or die "cannot open $output $!";
while (my $line = <$IN>) {
    chomp $line; # remove newline character from end of line
    if ($line =~ /INTERPOLATED HYDROGRAPH AT (\w+)$/){
        print $OUT $1;
        $line = <$IN> for 1..6; # skip 5 lines
        my $val2 = (split / /, $line)[1]; # get the second column
        print $OUT " $val2";
        $line = <$IN>for 1..2; # skip one line
        chomp $line;
        my $val3 = (split / /, $line)[-1]; # get the last column
        print $OUT " $val3\n"; 
    }
}
[download]

And this is the output:

$ cat output.txt
CAC40 1223. 1456.
[download]

I should add that this is not very robust code, it will probably break with any irregularity in the input data. But we would need to know more about the data (having at least three or four samples of line groups, instead of only one) to be able to do something more robust.

Update: See this post for a faster version.

Thank you for sharing batch scripting on Windows. Do not fear Perl. The following book is helpful for beginners.

Perl by Example 4th Edition.

Below, am sharing a parallel version using MCE. However, am using MCE mainly for its chunking abilities. Each record is handled individually. The record separator is anchored at the start of the line with "\nINTERPOLATED HYDROGRAPH". MCE detects this and handles automatically.

use strict;
use warnings;

use MCE::Loop;
use MCE::Candy;

my $input_file  = shift || 'input.txt';
my $output_file = shift || 'output.txt';

my $match_string = "INTERPOLATED HYDROGRAPH";

open my $ofh, ">", $output_file
   or die "cannot open '$output_file' for writing: $!\n";

MCE::Loop::init {
   use_slurpio => 1, chunk_size => 1, max_workers => 4,
   gather => MCE::Candy::out_iter_fh($ofh),
   RS => "\n${match_string}",
};

## Below, each worker receives one record at a time
## Output order is preserved via MCE::Candy::out_iter_fh

## line  1 CAC40 #  INTERPOLATED HYDROGRAPH AT CAC40
## line  2       #  blank line here
## line  3       #  PEAK FLOW TIME MAXIMUM AVERAGE FLOW
## line  4       #  6-HR 24-HR 72-HR 166.58-HR
## line  5       #  + (CFS) (HR)
## line  6       #  (CFS)
## line  7 1223. #  + 1223. 12.67 890. 588. 245. 106.
## line  8       #  (INCHES) .154 .408 .509 .509
## line  9 1456. #  (AC-FT) 441. 1166. 1456. 1456.
## line 10       #  CUMULATIVE AREA = 53.67 SQ MI

mce_loop_f {
   my ( $mce, $chunk_ref, $chunk_id ) = @_;

   ## Skip initial record containing header lines including *** ***

   if ( $chunk_id == 1 && $$chunk_ref !~ /^${match_string}/ ) {
      ## Gathering here is necessary when preserving output order,
      ## to let the manager process know chunk_id 1 has completed.
      MCE->gather( $chunk_id, "" );
      MCE->next;
   }

   ## Each record begins with INTERPOLATED HYDROGRAPH.

   my ( $k1, $k2, $k3 ) = ( "", "", "" );

   open my $ifh, "<", $chunk_ref;

   while ( <$ifh> ) {
      $k1 = $1 and next if $. == 1 && /(\S+)\s*$/;
      $k2 = $1 and next if $. == 7 && /^\S+\s+(\S+)/;
      $k3 = $1 and last if $. == 9 && /(\S+)\s*$/;
   }

   close $ifh;

   ## Gather values.

   MCE->gather( $chunk_id, "$k1 $k2 $k3\r\n" );

} $input_file;
[download]

Kind regards, Mario.

- use the match // operator to parse a string

- use 'split' to separate a line of input into columns

- use 'fork' to perform tasks in parallel

Update: Increased chunk size to 400.

Below, a parallel version with chunking enabled for the solution provided by monk Laurent_R. I ran against an input file containing 500k records.

Serial: 2.574 seconds. Parallel: 0.895 seconds, which includes the time to fork and reap children under a Unix environment. Afterwards, the output contains 500k lines.

The test machine is a 2.6 GHz Haswel Core i7 with RAM at 1600 MHz.

Optionally, the script can receive the input_file and output_file as arguments.

use strict;
use warnings;

use MCE::Loop;
use MCE::Candy;

my $input_file  = shift || 'input.txt';
my $output_file = shift || 'output.txt';

open my $ofh, ">", $output_file
   or die "cannot open '$output_file' for writing: $!\n";

MCE::Loop::init {
   use_slurpio => 1, chunk_size => 400, max_workers => 4,
   gather => MCE::Candy::out_iter_fh($ofh),
   RS => "\nINTERPOLATED HYDROGRAPH",
};

## Each worker receives many records determined by chunk_size.
## Output order is preserved via MCE::Candy::out_iter_fh

mce_loop_f {
   my ( $mce, $chunk_ref, $chunk_id ) = @_;

   open my $ifh, "<", $chunk_ref;
   my $output = "";

   while ( my $line = <$ifh> ) {
      chomp $line; # remove newline character from end of line
      if ( $line =~ /INTERPOLATED HYDROGRAPH AT (\w+)$/ ) {
         $output .= $1;
         $line = <$ifh> for 1..6; # skip 5 lines
         my $val2 = (split / /, $line)[1]; # get the second column
         $output .= " $val2";
         $line = <$ifh> for 1..2; # skip one line
         chomp $line;
         my $val3 = (split / /, $line)[-1]; # get the last column
         $output .= " $val3\r\n";
      }
   }

   close $ifh;

   MCE->gather( $chunk_id, $output );

} $input_file;

close $ofh;
[download]

Kind regards, Mario.

Quite interesting. Although I vaguely knew about the MCE module(s) before, I have never tried to use it/them, because I had the feeling that it would not bring much benefit when reading only one very large file (my input files are often gigabytes or even tens of GB large). It appears from your example that I was probably dead-wrong.

I should probably give it a try. Although one of my problem, presently, is that I am currently stuck with very old versions of Perl (5.8) (because of the AIX and VMS versions I am working on), so that dependencies might be fairly difficult to resolve.

But, having said that, we should move relatively soon (hopefully just a few months) to new hardware (blades) with much more recent versions of Linux, thus enabling much more recent versions of Perl. If not now, at least then, I might be able to take advantage of the MCE module(s).

Thank you for the information.

I do appreciate the help on this issue, it has been wonderful. I have been intrigued by Perl and what it can do. It is definitely quite different from anything that I have used before. I did have to dive into the solutions that were provided a little, because it was not working right out of the gate for me, and it came down to leading spaces in a line of the input in front of "Interpolated Hydrograph". For some reason I left that off on the original post.

I modified that, added a header row, and removed the \r on the MCE->gather so there are no blank lines. The time difference is night and day:

Batch (~275k lines) = ~3 hours

Perl <1 second

The final code that works like a charm that I am using:

use strict;
use warnings;

use MCE::Loop;
use MCE::Candy;

my $input_file  = shift || 'input.txt';
my $output_file = shift || 'output.txt';

my $match_string = "                         INTERPOLATED HYDROGRAPH A
+T   ";

open my $ofh, ">", $output_file
   or die "cannot open '$output_file' for writing: $!\n";
print $ofh "HEC1_ID,Q100,V100\n";
   
MCE::Loop::init {
   use_slurpio => 1, chunk_size => 1, max_workers => 4,
   gather => MCE::Candy::out_iter_fh($ofh),
   RS => "\n${match_string}",
};

## Below, each worker receives one record at a time
## Output order is preserved via MCE::Candy::out_iter_fh

## line  1 CAC40 #  INTERPOLATED HYDROGRAPH AT CAC40
## line  2       #  blank line here
## line  3       #  PEAK FLOW TIME MAXIMUM AVERAGE FLOW
## line  4       #  6-HR 24-HR 72-HR 166.58-HR
## line  5       #  + (CFS) (HR)
## line  6       #  (CFS)
## line  7 1223. #  + 1223. 12.67 890. 588. 245. 106.
## line  8       #  (INCHES) .154 .408 .509 .509
## line  9 1456. #  (AC-FT) 441. 1166. 1456. 1456.
## line 10       #  CUMULATIVE AREA = 53.67 SQ MI

mce_loop_f {
   my ( $mce, $chunk_ref, $chunk_id ) = @_;

   ## Skip initial record containing header lines including *** ***

   if ( $chunk_id == 1 && $$chunk_ref !~ /^${match_string}/ ) {
      ## Gathering here is necessary when preserving output order,
      ## to let the manager process know chunk_id 1 has completed.
      MCE->gather( $chunk_id, "" );
      MCE->next;
   }

   ## Each record begins with INTERPOLATED HYDROGRAPH.

   my ( $k1, $k2, $k3 ) = ( "", "", "" );

   open my $ifh, "<", $chunk_ref;

   while ( <$ifh> ) {
      $k1 = $1 and next if $. == 1 && /(\S+)\s*$/;
      $k2 = $1 and next if $. == 7 && /^\S+\s+(\S+)/;
      $k3 = $1 and last if $. == 9 && /(\S+)\s*$/;
   }
   close $ifh;

   ## Gather values.

   MCE->gather( $chunk_id, "$k1,$k2,$k3\n" );

} $input_file;
[download]

Thanks again. I hope to be learning more of this in the future.