Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl Monk, Perl Meditation
 
PerlMonks  

Re: Script far too slow with large files - wisdom needed!

by flexvault (Monsignor)
on Jan 22, 2016 at 08:52 UTC ( [id://1153357] : note . print w/replies, xml ) Need Help??


in reply to Script far too slow with large files - wisdom needed!

Hello biologistatsea,

You've gotten some good pointers, but no-one has mentioned using 'index' instead of a regex. 'index' and 'substr' are extremely fast if used properly. (Untested example).

while (my $line = <MGF>) { my $si = index( $line, 'TITLE=' ); ## Find start of 'TITL +E=' string from beginning if ( $si >= 0 ) { my $sj = index( $line, "\n", $si + 5 ); ## Get data after 'TIT +LE=' if ( $sj > $sj ) { $TI = substr( $line, $si+6, $sj-($si+6) ); +} } . . . }

Remember 'index' returns '-1' if the string is not found, so you have to test for a positive number. ( Can't do 'if ( $si )' since '-1' is 'not 0' and is TRUE.

I've seen 10 to 50 times improvement on many-GByte files. YMMV.

Regards...Ed

"Well done is better than well said." - Benjamin Franklin

Replies are listed 'Best First'.
Re^2: Script far too slow with large files - wisdom needed!
by blahblahblah (Priest) on Jan 23, 2016 at 13:09 UTC
    I've gotten good results from this too. In cases where you have a big regex which is too complicated to break down, you can do both: first, use index to do a simpler test. If that passes, then go on to use the full regex.