Re: Script far too slow with large files

in reply to Script far too slow with large files - wisdom needed!

You've gotten some good pointers, but no-one has mentioned using 'index' instead of a regex. 'index' and 'substr' are extremely fast if used properly. (Untested example).

while (my $line = <MGF>)
{   my $si = index( $line, 'TITLE=' );          ## Find start of 'TITL
+E=' string from beginning
    if ( $si >= 0 )
    {   my $sj = index( $line, "\n", $si + 5 ); ## Get data after 'TIT
+LE='
        if ( $sj > $sj ) { $TI = substr( $line, $si+6, $sj-($si+6) ); 
+}
    }
  . . .
}
[download]

Remember 'index' returns '-1' if the string is not found, so you have to test for a positive number. ( Can't do 'if ( $si )' since '-1' is 'not 0' and is TRUE.

I've seen 10 to 50 times improvement on many-GByte files. YMMV.

Regards...Ed

"Well done is better than well said." - Benjamin Franklin

Comment on Re: Script far too slow with large files - wisdom needed! Download Code

Replies are listed 'Best First'.
Re^2: Script far too slow with large files - wisdom needed! by blahblahblah (Priest) on Jan 23, 2016 at 13:09 UTC
I've gotten good results from this too. In cases where you have a big regex which is too complicated to break down, you can do both: first, use index to do a simpler test. If that passes, then go on to use the full regex.	[reply]

In Section Seekers of Perl Wisdom