Howdy there,

I am having an issue dereferencing a series of arrays (which are all in this case diifferent files taken in line by line) inside a subroutine. I do this for several arrays sequentially by applying a subroutine several times. The third array I dereference is rather large (constructed from a 100 Mb txt file), but only 25X as large as the next largest file. Unfortunately, this operation seems to take ~460 seconds, whereas the same operation for any other file takes at most between 1-2 seconds, as can be seen by my results (posted below). I have to do this many times repeatedly throughout my code and cannot really afford this huge lag time. Is there any way I can speed this up? Is there a reason why this is hanging? It should be noted that while it is hanging that the memory usage for perl slowly crawls from ~500,000k to ~700,000k. Thanks for your help!

Relevant code:

sub genLookupTable{
#begin subroutine genLookupTable
#ArrayRefs is a global variable wherein the references to the arrays o
+f interest are stored
 
my $lengRefs = @ArrayRefs
for(my $i = 0; $i<$lengRefs; $i++){
    print time()." is the time before\n";
    my @lines = @{$ArrayRefs[$i]};
    print time()." is the time after\n";
    my $why = @lines;
    print " The length of lines for $i is $why\n";
…

[download]

Relevant Results:


C:\Perl\bin>perl newIntermediate2.pl
The time is now 1244797376
files are opening...
files have opened and been put into arrays and those arrays have been made into references and put into an array...
The time is now 1244797384

Those arrays are being processed...

I am now in the subroutine.
1244797425 is the time before dereferencing array 1
1244797425 is the time after dereferencing array 1
The length of lines for 0 is 87544
1244797426 is the time before dereferencing array 2
1244797426 is the time after dereferencing array 2
The length of lines for 1 is 21573
1244797426 is the time before dereferencing array 3
1244797892 is the time after dereferencing array 3
The length of lines for 2 is 2250393
1244797893 is the time before dereferencing array 4
1244797893 is the time after dereferencing array 4
The length of lines for 3 is 12329
1244797893 is the time before dereferencing array 5
1244797893 is the time after dereferencing array 5
The length of lines for 4 is 83274
1244797893 is the time before dereferencing array 6
1244797893 is the time after dereferencing array 6
The length of lines for 5 is 66514
1244797893 is the time before dereferencing array 7
1244797893 is the time after dereferencing array 7
The length of lines for 6 is 7998
1244797893 is the time before dereferencing array 8
1244797893 is the time after dereferencing array 8
The length of lines for 7 is 2453

In reply to Slow Dereferencing and Not Sure Why by deequeue

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.