Beefy Boxes and Bandwidth Generously Provided by pair Networks
XP is just a number
 
PerlMonks  

Re: Script far too slow with large files - wisdom needed!

by flexvault (Monsignor)
on Jan 22, 2016 at 08:52 UTC ( #1153357=note: print w/replies, xml ) Need Help??


in reply to Script far too slow with large files - wisdom needed!

Hello biologistatsea,

You've gotten some good pointers, but no-one has mentioned using 'index' instead of a regex. 'index' and 'substr' are extremely fast if used properly. (Untested example).

while (my $line = <MGF>) { my $si = index( $line, 'TITLE=' ); ## Find start of 'TITL +E=' string from beginning if ( $si >= 0 ) { my $sj = index( $line, "\n", $si + 5 ); ## Get data after 'TIT +LE=' if ( $sj > $sj ) { $TI = substr( $line, $si+6, $sj-($si+6) ); +} } . . . }

Remember 'index' returns '-1' if the string is not found, so you have to test for a positive number. ( Can't do 'if ( $si )' since '-1' is 'not 0' and is TRUE.

I've seen 10 to 50 times improvement on many-GByte files. YMMV.

Regards...Ed

"Well done is better than well said." - Benjamin Franklin

Replies are listed 'Best First'.
Re^2: Script far too slow with large files - wisdom needed!
by blahblahblah (Priest) on Jan 23, 2016 at 13:09 UTC
    I've gotten good results from this too. In cases where you have a big regex which is too complicated to break down, you can do both: first, use index to do a simpler test. If that passes, then go on to use the full regex.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1153357]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others chanting in the Monastery: (4)
As of 2020-09-19 02:11 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    If at first I donít succeed, I Ö










    Results (114 votes). Check out past polls.

    Notices?