comment on

Hello biologistatsea,

You've gotten some good pointers, but no-one has mentioned using 'index' instead of a regex. 'index' and 'substr' are extremely fast if used properly. (Untested example).

while (my $line = <MGF>)
{   my $si = index( $line, 'TITLE=' );          ## Find start of 'TITL
+E=' string from beginning
    if ( $si >= 0 )
    {   my $sj = index( $line, "\n", $si + 5 ); ## Get data after 'TIT
+LE='
        if ( $sj > $sj ) { $TI = substr( $line, $si+6, $sj-($si+6) ); 
+}
    }
  . . .
}
[download]

Remember 'index' returns '-1' if the string is not found, so you have to test for a positive number. ( Can't do 'if ( $si )' since '-1' is 'not 0' and is TRUE.

I've seen 10 to 50 times improvement on many-GByte files. YMMV.

Regards...Ed

"Well done is better than well said." - Benjamin Franklin

In reply to Re: Script far too slow with large files - wisdom needed! by flexvault
in thread Script far too slow with large files - wisdom needed! by biologistatsea

Are you posting in the right place? Check out Where do I post X? to know for sure.
Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
<code> <a> <b> <big> <blockquote> <br /> <dd> <dl> <dt> <em> <font> <h1> <h2> <h3> <h4> <h5> <h6> <hr /> <i> <li> <nbsp> <ol> <p> <small> <strike> <strong> <sub> <sup> <table> <td> <th> <tr> <tt> <u> <ul>
Snippets of code should be wrapped in <code> tags not <pre> tags. In fact, <pre> tags should generally be avoided. If they must be used, extreme care should be taken to ensure that their contents do not have long lines (<70 chars), in order to prevent horizontal scrolling (and possible janitor intervention).
Want more info? How to link or How to display code and escape characters are good places to start.


Do you know where your variables are?
	PerlMonks