Your algorithm is fundamentally flawed. Assume for a moment that the input file consisted of the following:

1 S AGT 3 S AGT 5 R AGT 7

Your algorithm would proceed as follows:

  1. The outer loop scans until it reads record 2;
  2. The inner loop then scans on until it reads record 6;
  3. Then you reset the pointer to the start of record 3; and exit to the outer loop;
  4. The outer loop scans on to record 4;
  5. The inner loop scans on to record 6;
  6. Then you reset the pointer to the start of record 5; and exit to the outer loop;
  7. The outer loop scans on to EOF.

Note that the above has matched both S records with the same R record.

Now imagine that your 100MB file has (say) 1000 S-records near the top of the file, and a single R-record 10 million lines further on. Then each of those 100 S-records would get matched against that single R-record; but you would have to re-read the 10 million intervening records over and over to do so.

1e3 * 10e6 == a very long time. It might well look like it had hung.

If the S-record/R-record pairs should appear sequentially:

... S AGT ... R AGT ... S AGT ... R AGT ...

Then you should not be resetting the pointer after you've found the matching R-record.

If however, the S-record/R-record pairs can be interleaved:

... S AGT #1 ... S AGT #2 ... R AGT #1 ... R AGT #2 ...

Then you would have to be maintaining two pointers: one telling you where to start looking for the next S-record; and one telling you where to start looking for the next R-record. Whilst this could be made to work, there are other, simpler approaches to this latter problem.

For example: a simple, single loop down the file looking for both types of record. When an S-record is found, you push it onto an array; when an R-record is found, you shift off the least recently found S-record from the array and do your calculations.

my @Srecs; while( <FILE> ) { if( /^S AGT/ ) { push @Srecs, $_; } elsif( /^R AGT/ ) { die "R-record with no corresponding S-record" unless @Srec; my @sRecBits = split ' ', shift @Srecs; my @rRecBits = split ' ', $_; ## Calculate stuff } } if( @Srecs) { printf "There were %d unmatched S-records\n"; scalar @Srec; } ## Other stuff

This single pass algorithm is far more efficient, less error prone and detects mismatches.


Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.

In reply to Re: Program Hangs by BrowserUk
in thread Program Hangs by anbarasans85

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":



  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.