Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl-Sensitive Sunglasses

Re: Script far too slow with large files - wisdom needed!

by flexvault (Monsignor)
on Jan 22, 2016 at 08:52 UTC ( #1153357=note: print w/replies, xml ) Need Help??

in reply to Script far too slow with large files - wisdom needed!

Hello biologistatsea,

You've gotten some good pointers, but no-one has mentioned using 'index' instead of a regex. 'index' and 'substr' are extremely fast if used properly. (Untested example).

while (my $line = <MGF>) { my $si = index( $line, 'TITLE=' ); ## Find start of 'TITL +E=' string from beginning if ( $si >= 0 ) { my $sj = index( $line, "\n", $si + 5 ); ## Get data after 'TIT +LE=' if ( $sj > $sj ) { $TI = substr( $line, $si+6, $sj-($si+6) ); +} } . . . }

Remember 'index' returns '-1' if the string is not found, so you have to test for a positive number. ( Can't do 'if ( $si )' since '-1' is 'not 0' and is TRUE.

I've seen 10 to 50 times improvement on many-GByte files. YMMV.


"Well done is better than well said." - Benjamin Franklin

Replies are listed 'Best First'.
Re^2: Script far too slow with large files - wisdom needed!
by blahblahblah (Priest) on Jan 23, 2016 at 13:09 UTC
    I've gotten good results from this too. In cases where you have a big regex which is too complicated to break down, you can do both: first, use index to do a simpler test. If that passes, then go on to use the full regex.

Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://1153357]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others taking refuge in the Monastery: (7)
As of 2022-01-19 13:05 GMT
Find Nodes?
    Voting Booth?
    In 2022, my preferred method to securely store passwords is:

    Results (55 votes). Check out past polls.