$IDS{$id}++;
...
if ($A+$B == 2 && (exists $IDS{$TI})){
[download]

Turning the second if into an elsif might also save some microseconds.

($q=q:Sq=~/;[c](.)(.)/;chr(-||-|5+lengthSq)`"S|oS2"`map{chr |+ord
}map{substrSq`S_+|`|}3E|-|`7**2-3:)=~y+S|`+$1,++print+eval$q,q,a,
[download]

Your MGF file seems to be divided into records (blocks). If the first lines of a record are always the same information in the same order, you can skip the useless lines. For that, you should read the file record by record instead of line by line (which would be better than your method for finding pairs, if for any reason the TITLE line doesn't match for one record, you'll match the RT of a record with the TITLE of the next, and so on until the next error). This can be done using the input record separator, or $/. If your records are paragraph (separated by at least one empty line, but contain no empty lines themselves) you can do something like:

use constant {
   TITLE => 3,
   RT    => 7
};
{ # extra block so that $/ retrieve its previous value after reading t
+his file
  local $/ = ""; # Paragraph mode
  while (my $record= <MGF>)
  {
    my @lines = split "\n", $record;
    $TI = "$1" if $lines[TITLE] =~ /^TITLE=(.*)/;
    $RT = "$1" if $lines[RT] =~ /^RTINSECONDS=(.*)/;
  }
}
[download]

If the records are not paragraphs, this could probably work by setting $/ = "SEARCH" if all records start with it.

If some of the fields in the records may be missing, and you can't use their position, reading record by record can still help you skip the lines at the end once you have found what you are looking for:

{
  local $/ = ""; # Paragraph mode
  RECORD: while (my $record = <MGF>)
  {
    my @lines = split "\n", $record ;
    $TI = undef;
    $RT = undef;
    for my $line (@lines)
    {
      $TI = "$1" if $line =~ /^TITLE=(.*)/;
      $RT = "$1" if $line =~ /^RTINSECONDS=(.*)/;
      next RECORD if defined $TI and defined $RT; # skip to next recor
+d
    }
  }
}
[download]

In any case, /^TITLE=(.*)/ is a little faster than /TITLE=(.*)/ because it tells perl to stop searching if TITLE isn't at the beginning of the line.

Edit : added missing end of code sections

The simplest way to speed up your second loop would be to skip over all the non-title lines:

my( $ti, $rt );
{
    {   local $/ = "TITLE="; <MGF>; }
    ( $ti ) = <MGF> =~ m[^(\S+)];
    {   local $/ = "RTINSECONDS="; <MGF> }
    ( $rt ) = <MGF> =~ m[^(\S+)];
    if( exists $IDS{ $ti } ) {
         $IDrtPairs{ $ti } = $rt;
    }
    redo unless eof( MGF );
}
[download]

Warning untested code that probably needs a couple of tweaks, but should explain the idea.

Using your "realistic time" data set profile your code (see Devel::Profile) and see where energy needs to be focused to improve performance. You can then measure performance improvements and change focus as appropriate.

Yet another guess at performance improvement is to simply read the entire file into memory. With multiple gigabytes of memory in most typical computers that really shouldn't be an issue. You can then run a regex over the whole thing to edit lines in place then print the result in one hit.

It might improve things a bit if you rearrange your if statements, you don't need to check ($A + $B == 2...) for each line of the input file, as you know that $B can only change when you've matched RTINSECONDS so you could do this instead :-

    if ($line =~ /TITLE=(.*)/) {
        $title=$1; 
    } 
    elsif
    ($line =~ /RTINSECONDS=(.*)/) { 
        $RT=$1;
        if ($title && grep {$_ eq $TI} @IDS) {
            $IDrtPairs{$TI}=$RT;
            $title = undef;
        }
    }
[download]

As already mentioned it will be better to use a hash, so you should do that too :)

The example input data looks like TITLE & RTINSECONDS are always at the beginning of the line so you could anchor your regexs too, so it can reject non matching lines quicker. i.e. /^TITLE=(.*)/

If this is not an one-off exercise use a database system.

Hello biologistatsea,

You've gotten some good pointers, but no-one has mentioned using 'index' instead of a regex. 'index' and 'substr' are extremely fast if used properly. (Untested example).

while (my $line = <MGF>)
{   my $si = index( $line, 'TITLE=' );          ## Find start of 'TITL
+E=' string from beginning
    if ( $si >= 0 )
    {   my $sj = index( $line, "\n", $si + 5 ); ## Get data after 'TIT
+LE='
        if ( $sj > $sj ) { $TI = substr( $line, $si+6, $sj-($si+6) ); 
+}
    }
  . . .
}
[download]

Remember 'index' returns '-1' if the string is not found, so you have to test for a positive number. ( Can't do 'if ( $si )' since '-1' is 'not 0' and is TRUE.

I've seen 10 to 50 times improvement on many-GByte files. YMMV.

Regards...Ed

I've gotten good results from this too. In cases where you have a big regex which is too complicated to break down, you can do both: first, use index to do a simpler test. If that passes, then go on to use the full regex.

You can gain some time by using a search for id that stops when it finds any match instead of continuing thru the whole list.

use List::Util qw/any/;
...
#    if ($A+$B == 2 && grep {$_ eq $TI} @IDS){
    if ($A && $B && any { $_ eq $TI } @IDS) {
[download]

Update: toolic's idea is better.

Greetings,
I second what GrandFather said about devel profile. You might want to have a look at FFI::Platypus It should allow you to rewrite the routine pointed at by the profiler in a more optimised way (namely as a C library or even in assembler I guess)
Keep in mind that first I'd try and find more patterns in the data (as proposed by other posters) so I can get as much leverage as I can before doing a rewrite at a lower level

Hello biologistatsea,

The following is a parallel demonstration for your use case. The comments are inlined inside the code. This helps address the concern with slow processing for the 2nd loop.

#!/usr/bin/perl

use strict;
use warnings;

use MCE::Flow;
use MCE::Shared;

my $usage  = "perl mgf_rt.pl [input mgf file] [mascot csv output] [out
+put file]\n";
my $mgfin  = shift or die $usage;
my $csvin  = shift or die $usage;
my $output = shift or die $usage;

my (%IDS, $CSV, $OUT);

# Instantiate a shared HASH for use by MCE workers.
my $IDrtPairs = MCE::Shared->hash();

# Obtain IDs from the CSV file.
open $CSV, '<', $csvin or die "open error ($csvin): $!\n";

while ( <$CSV> ) {
    # ?: means not to capture the 1st ( )
    # therefore, $1 refers to the 2nd ( )
    $IDS{ $1 } = 1 if /^(?:[^,]*,){30}"([^"]*)/;
}

close $CSV;

print "Finished collecting spectra names from CSV file\n";

# Process huge file by record separator. The "\nTITLE" RS is a
# special case which anchors "TITLE" at the start of the line.
# Workers receive records beginning with "TITLE" and ending in
# "\n". A chunk_size greater than 8192 means to read # of bytes.
# A worker completes reading the rest of the record before the
# next worker reads the next chunk.

mce_flow_f {
    max_workers => 4,
    chunk_size  => "2m",
    RS          => "\nTITLE",
},
sub {
    my ( $mce, $chunk_ref, $chunk_id ) = @_;
    my %pairs;

    # Collect pairs locally.
    for my $rec ( @{ $chunk_ref } ) {
        my @match = $rec =~ /^(?:TITLE|RTINSECONDS)=([^\n]+)/mg;
        $pairs{ $match[0] } = $match[1] if ( @match == 2 );
    }

    # Send pairs to the shared manager process.
    $IDrtPairs->mset(%pairs) if %pairs;

}, $mgfin;

# Shutdown the MCE workers.
MCE::Flow::finish;

print "Finished getting RT information from MGF file\n";

# Export/destroy the shared HASH into a local copy. Basically,
# to not involve the shared manager process for the next step.
$IDrtPairs = $IDrtPairs->destroy;

# Output to a new CSV file.
open $CSV, '<', $csvin  or die "open error ($csvin): $!\n";
open $OUT, '>', $output or die "open error ($output): $!\n";

while ( my $line = <$CSV> ) {
    chomp $line;
    if ( $line =~ /^(?:[^,]*,){30}"([^"]*)/ ) {
        my $reten = $IDrtPairs->get($1);
        print $OUT "$line,$reten\n";
    }
    else {
        print $OUT "$line\n";
    }
}

close $CSV;
close $OUT;
[download]

Perl is fun, Mario.