One thing that jumps out at me skimming your code is this pattern:

if ($_ =~ /^\s+total wire length\:\s+\d.*\d\s*/) {
                    ($NetLength) = $_ =~ /^\s+total wire length\:\s+(\
+d.*\d)\s*/;
[download]

There's no need to use two regexes here, and also no need to be explicit about $_ (it's the default match target). You can just do:

if (/^\s+total wire length\:\s+(\d.*\d)\s*/) {
  $NetLength = $1;
[download]

That should shave off a bit of time.

Another possibility to consider that profiling won't immediately suggest is parallellization. Do you have multiple CPUs or multiple machines at your disposal? If so, divide the file into pieces and hand them off to multiple processes to work on simulataneously.

-sam

The OP also wants to test if the text following the colon is a numeric value or something else. It still can be done with one regex, though:

if (/^\s+total wire length\:\s+(\d.*\d)?/) {
  if (defined($1)) {
    $NetLength = $1;
    $c++;
  } else {
    $NetLength = "NaN";
  }
}
[download]

That said, here are some small improvements to your code that could result in better performance:

1) Use elsif where possible, i.e.:

if (/wire capacitance/) {
  ...
} elsif (/wire resistance/) {
  ...
[download]

2) Use: if ($TotalNets % 50_000 == 0 && $TotalNets > 0) ... to test if you want to print out the trace message. It's simpler, cleaner and faster.

        if (($TotalNets == 50000) || ($TotalNets == 100000) || ($Total
+Nets == 250000) || ($TotalNets == 500000) || ($TotalNets == 1000000) 
+|| ($TotalNets == 1500000) || ($TotalNets == 2000000) || ($TotalNets 
+== 3000000)) {
[download]

        if ( $Nets{$TotalNets) ) {
[download]

Another suggestion would be to replace

            $_ =~ s/^ *//;
            $_ =~ s/ *$//;
            @DriverLine = split (/\s+/,$_);
[download]

       @DriverLine = split( /\s+/, 
                            (( /^\s*(\S.*)\s*$/ ) ? $1 : $_ ) );
[download]

The code hasn't been tested, and I can't swear to the efficiencies.

Split has a magical whitespace incantation:

$_ = '  foo  bar  baz   ';
print "'$_'\n" for split ' ';
[download]

So you don't need to trim leading or trailing whitespace to get the desired result.

As to the final suggestion, it might be more efficient to replace what is in effect two regexes (plus some logic) with just one:

>perl -wMstrict -e
"for (@ARGV) {
  my @s = m{ \S+ }xmsg;
  local $\" = ':';
  print qq('$_'  :@s: \n);
  }
"
""   " "   "  "   "   "   "    "
"foo"   "  foo"   "foo  "   "  foo  "
"foo bar"   "  foo bar"   "foo bar  "   "  foo bar  "
"foo bar baz"   "   foo bar baz"   "foo bar baz  "
"  foo bar baz  "
''  ::
' '  ::
'  '  ::
'   '  ::
'    '  ::
'foo'  :foo:
'  foo'  :foo:
'foo  '  :foo:
'  foo  '  :foo:
'foo bar'  :foo:bar:
'  foo bar'  :foo:bar:
'foo bar  '  :foo:bar:
'  foo bar  '  :foo:bar:
'foo bar baz'  :foo:bar:baz:
'   foo bar baz'  :foo:bar:baz:
'foo bar baz  '  :foo:bar:baz:
'  foo bar baz  '  :foo:bar:baz:
[download]

Or probably even more efficient, come to think of it, just to use the default  split parameters:

>perl -wMstrict -e
"for (@ARGV) {
  my @s = split;
  local $\" = ':';
  print qq('$_'  :@s: \n);
  }
"
""   " "   "  "   "   "   "    "
"foo"   "  foo"   "foo  "   "  foo  "
"foo bar"   "  foo bar"   "foo bar  "   "  foo bar  "
"foo bar baz"   "   foo bar baz"   "foo bar baz  "
"  foo bar baz  "
''  ::
' '  ::
'  '  ::
'   '  ::
'    '  ::
'foo'  :foo:
'  foo'  :foo:
'foo  '  :foo:
'  foo  '  :foo:
'foo bar'  :foo:bar:
'  foo bar'  :foo:bar:
'foo bar  '  :foo:bar:
'  foo bar  '  :foo:bar:
'foo bar baz'  :foo:bar:baz:
'   foo bar baz'  :foo:bar:baz:
'foo bar baz  '  :foo:bar:baz:
'  foo bar baz  '  :foo:bar:baz:
[download]

while (<>) { @_ = split; $wc += @_ }
print "counted $wc white-space-delimited tokens\n";
[download]

In any case, I'd be more inclined to look for ways to economize on the amount of code being written to accomplish the task. One thing that might simplify the logic a lot would be to do record-oriented input (rather than line-oriented input):

open (NETSTATS,"$input_file");
$/ = "\nnet '";

my @field_names = ( 'wire capacitance', 'wire resistance',
           'number of loads', 'number of pins', 'total wire length' );

while (<NETSTATS>) {
    chomp;  # removes $/ from end of string;
    warn "got record # $.\n" if ( $. % 50000 == 0 );  # this belongs o
+n STDERR, IMO
    next unless (/^([^']+)'/);
    $NetName = $1;

    my %field_val = ();
    for my $field ( @field_names ) {
        if ( /$field:\s+([\d.]+)/ ) {
            $field_val{$field} = $1;
        }
    }
    # ....
}
[download]

I'm not going to try reimplementing the whole thing, but just that little snippet should give you the basic idea of how I would go about it. The part I've shown replaces approximately the first 50 lines of code from the OP. As for the rest, instead of testing a bunch of distinct scalar variables in order to determine what to do with the record data, you are instead checking the keys and values of a hash, which can be done with less code.

Even if it ends up running a little slower than the original (though I doubt it would), there are other advantages in terms of clarity and maintainability of the code.

And with this sort of approach, it might be easier to find tricks that will speed it up -- e.g. the regex matches in the for loop might be quicker if done like this (because with each iteration, $_ becomes shorter, and the target string is near the beginning):

    for my $field ( @field_names ) {
        if ( s/.*?\s$field:\s+([\d.]+)//s ) {
            $field_val{$field} = $1;
        }
    }
[download]

Hi, All,

Thanks for all of the great suggestions. I have reduced the processing by a whopping 40% after rolling in a number of suggestions. The new result is below.

I don't know how to parallel process something like this. I think splitting the file will be too time-consuming. (I am running on a system with 32 CPUs, though, so it is tempting.) I think the file sizes will have to be much bigger before I consider that.

Again, thanks everyone!!

Fiddler42

open (NETSTATS,"$input_file");

$TotalNets = 0;
while (<NETSTATS>) {
    if (/^net \'(.*)'\:\s*$/) {
        $NetName = $1;
        $c = 1;
        $TotalNets++;
        if ($TotalNets % 100_000 == 0 && $TotalNets > 0) {
            print ("Parsed $TotalNets nets...\n");
        }
        do {
            if (/^\s+wire capacitance\:\s+(\d+.*\d*)\s*$/) {
                $NetCapRaw = $1;
                $NetCap = $CapMultiplier*$NetCapRaw;
                $c++;
            } elsif (/^\s+wire resistance\:\s+(\d+.*\d*)\s*$/) {
                $NetRes = $1;
                $c++;
            } elsif (/^\s+number of loads\:\s+(\d+)\s*$/) {
                $NetFanout = $1;
                $c++;
            } elsif (/^\s+total wire length\:\s+(\d+.*\d*)\s*/) {
                $NetLength = $1;
                $c++;
            }
            $_ = <NETSTATS>;
        } until ((/Driver Pins/) || ($_ eq "" ));
        if (/Driver Pins/) {
            $_ = <NETSTATS>;
            $_ = <NETSTATS>;
            ($FirstDriver) = $_ =~ /^\s*(\S.*)\s*/;
            $c++;
        }
        $AddToCustomTable = 0;        
        if (($NetName ne "") && (($NetCap ne "") && ($NetCap ne "NaN")
+) && (($NetRes ne "") && ($NetRes ne "NaN")) && (($NetFanout ne "") &
+& ($NetFanout ne "NaN")) && (($NetLength ne "") && ($NetLength ne "Na
+N")) && ($FirstDriver ne "") && ($c == 6)) {
            if ($NetFanout <= $UpperFanoutLimitOfTable) {
                if (($UseNetPattern == 0) && ($UseDriverCell == 0) && 
+($TopLevelOnly == 0)) {
                    $AddToCustomTable = 1;                
                } elsif (($UseNetPattern == 0) && ($UseDriverCell == 0
+) && ($TopLevelOnly == 1)) {
                    $DriverForwardSlashCount = $FirstDriver =~ s/(\/)/
+$1/gs; # Simple command to count characters...
                    $NetNameForwardSlashCount = $NetName =~ s/(\/)/$1/
+gs;
                    if (($DriverForwardSlashCount <= 1) && ($NetNameFo
+rwardSlashCount <= 1 )) {$AddToCustomTable = 1;}
                    if ($DebugMode == 1) {
                        print ("Adding net $NetName (driver = $FirstDr
+iver)...\n");
                        print DEBUG_VERBOSE ("$NetFanout $NetRes\n");
                    }                    
                } elsif (($UseNetPattern == 0) && ($UseDriverCell == 1
+) && ($TopLevelOnly == 0)) {
                    if ($FirstDriver =~ qr/$DriverPattern/x) {$AddToCu
+stomTable = 1;} # to regard variable as a regular expression...
                } elsif (($UseNetPattern == 1) && ($UseDriverCell == 0
+) && ($TopLevelOnly == 0)) {
                    if ($NetName =~ qr/$NetPattern/x) {$AddToCustomTab
+le = 1;}
                } elsif (($UseNetPattern == 1) && ($UseDriverCell == 1
+) && ($TopLevelOnly == 0)) {
                    if ($NetName =~ qr/$NetPattern/x) {
                        $AddToCustomTable = 1;
                    } elsif ($FirstDriver =~ qr/$DriverPattern/x) {$Ad
+dToCustomTable = 1;}
                }
                # These conditions are not allowed per input argument 
+parsing...
                #} elsif (($UseNetPattern == 0) && ($UseDriverCell == 
+1) && ($TopLevelOnly == 1)) {
                #} elsif (($UseNetPattern == 1) && ($UseDriverCell == 
+0) && ($TopLevelOnly == 1)) {
                #} elsif (($UseNetPattern == 1) && ($UseDriverCell == 
+1) && ($TopLevelOnly == 1)) {
                                
            }
            if ($AddToCustomTable == 1) {push @{$NetStats[ $NetFanout 
+] ||= []}, [ $NetName, $NetCap, $NetRes, $NetLength, $FirstDriver ];}
        } else {
            if ($DebugMode == 1) {
                print DEBUG_VERBOSE ("ERROR: Problem deriving stats fo
+r net $NetName!\n");
                print DEBUG_VERBOSE ("ERROR: c=$c NetName=$NetName Net
+Fanout=$NetFanout NetCap=$NetCap NetRes=$NetRes NetLength=$NetLength 
+FirstDriver=$FirstDriver\n\n");
            }
        }
    }
    $NetName = "";
    $NetCap = "NaN";
    $NetRes = "NaN";
    $NetFanout = "NaN";
    $NetLength = "NaN";
    $FirstDriver = "";
}
print ("Parsed $TotalNets nets...\n\n");
close (NETSTATS);
[download]

32 CPUs, wow! If you're interested, I think splitting the file should be very easy. To split it N ways I would:
1. $start = 0
2. seek() forward int($size / $N).
3. Search forward for the next "^net" delimiter line, capturing start position in $here.
4. Write out the chunk from $start to $here to a file "chunk.$N".
5. Set $start = $here
6. Loop to 2 until done.

Whether this is an overall win (it will take time) depends a lot on your disks. You'll have to tune $N to match the number of CPUs you can keep fed with data - set it too high and you'll go slower as your CPUs compete for disk access and slow each other down.

-sam

This:

        if (($TotalNets == 50000) ||
 ($TotalNets == 100000) || ($TotalNets == 250000) ||
 ($TotalNets == 500000) || ($TotalNets == 1000000) || 
($TotalNets == 1500000) || ($TotalNets == 2000000) || 
($TotalNets == 3000000)) {
[download]

do {
    for (0..N) {
        if ( my $line = <FH>) {
            ...
            do stuff here
            ...
        } else {
            last;
        }
    }
print "$Some_Total";
} until (eof );
[download]

You're processing every token three times here:

if ($_ =~ /wire capacitance/) {
                if ($_ =~ /^\s+wire capacitance\:\s+\d.*\d\s*$/) {
                    ($NetCapRaw) = $_ =~ /^\s+wire capacitance\:\s+(\d
+.*\d)\s*$/;
[download]

Actually I like the idea of tokenizing the whole file in a multi-pass interpreter; tokenize the file first replacing each token with a code-ref and each constant with an object that returns a constant. then execute the resulting file.

What does this do?

 if (($DriverForwardSlashCount == 0) && ($NetNameForwardSlashCount == 
+0)) {
                        $AddToCustomTable = 1;
[download]

$AddToCustomTable = 1 if ($DriverForwardSlashCount | $NetNameForwardSl
+ashCount <= 1 );
[download]

There are two time eaters in the code; reading the file and executing the regexes. I would try to separate those. Read the file in and split the fields, generating a hash of tokens and data ( note this is similar to the parsing idea above.) then process the hash for your data. This would seem like extra work but often when you refactor the code like this you see optimizations you wouldn't see with the code all in one mashup like it is.

I find that breaking these sort of jobs down into smaller tasks can offer benefits that out-weigh any speed penalties.

For instance you could consider doing the 'extracting' and the 'reporting' separately. First load each 'record' into a hash and then worry about what to do with it.

Long winded loops, complex ifs and elses and tricky 'bulk' regexes often leads, imo, to code that is difficult to write, read and maintain. And if the spec changes...

Building on the suggestions already this does the parsing.

update
Redundant line at the begining of process_part_two sub commented out.