First thing's first. If you have sets embedded in your list, keep them as sets. What you're doing is nearly the same thing as pointer arithmetic in C, and is bad for many of the same reasons. ww mentioned an AoA, but as a question. I'll make it a statement: it should be an array of references. Personally, I'm a bit more partial to AoH, as it allows me to use names for keys to make sense of the inner data. An AoA would look like:

@in = ( [ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14 ], [ 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34 ], # ... );
but we still don't know what any of the numbers mean without more context. An AoH usually would take up a lot more memory, but would give the context implicitly:
@in = ( { user => 'Fighter2', id => 936570, xp => 34, writeups => 11, ... }, );
You can get hashes to take a lot less memory if you can define "default" values that cover most cases, reducing most of the hashes down to only unique values, at a slight cost to code complexity. I'll assume that's not an option (it rarely is, in my experience).

Once you do this, you would have $in[$i][12] or $in[$i]{attn} instead of $in[$i+12] Your outer for loop would be a simple for my $i (0..$#in) (you would want to go through the indexes, and the last one is the last index, which is what $# gives you). You wouldn't be creating an @Attn the length of your original array, but 1/14th the size, which would help both memory usage and speed. Also, your inner arrays would be arbitrary lengths - so if you regularly have less than 13 items in a set, you simply would not allocate the extra memory. And, should you need more than 14 items in a set, you can now handle that, kind of.

Here's a first crack:

#!/usr/bin/perl #Initialization my @in=( [...], [...], ); my $MaxAgc0 = "1.15503129132678e-005"; # MaxAgc0 is a constant #Interpolation if (@in) { for my $i (0..$#in) { my $ptr = $in[$i]; # convenience. # we don't need the first item to be a count of sectors anymor +e. # we also don't need this loop at all. #for my $j (0..$#$ptr) #{ # so we also reduce the numbers here. my $num = $ptr->[11] * $MaxAgc0; push @Attn, $num / $ptr->[9]; #} } for my $i (0..$#in) { push @powers, ($in[$i][$_] - $Attn[$i])/100 for (1..8); } } else { # @in is empty print "\n No data to map for Panda machine. Check if input logfile + is empty"; kill; }
Now, what I'd do from here is pull apart the sets even more. Say something like this:
( { attn => 5, # attenuation? agc => 6, # the MaxAgc0 multiplier sectors => [ all the sectors for this data point in this array ], # etc. }, # repeat for each item. )
I'd also change the output to match everything done above: return a list of arrays:
#!/usr/bin/perl #Initialization my @in=( {...}, {...}, ); my $MaxAgc0 = "1.15503129132678e-005"; # MaxAgc0 is a constant use List::MoreUtils qw(pairwise); #Interpolation if (@in) { my @Attn = map { my $num = $_->{agc} * $MaxAgc0; $num / $_->{attn}; } @in; @powers = pairwise { [ map { ( $a->{sectors}[$_] - $b ) / 100 } for @{$a->{sectors} +} ] } @in, @Attn; } else { # @in is empty print "\n No data to map for Panda machine. Check if input logfile + is empty"; kill; }
Now the return will keep the sectors grouped, there will be no extra data in the output. We can still improve this by getting rid of @Attn altogether:
#Initialization my @in=( {...}, {...}, ); my $MaxAgc0 = "1.15503129132678e-005"; # MaxAgc0 is a constant #Interpolation if (@in) { @powers = map { my $attn = $_->{agc} * $MaxAgc0 / $_->{attn}; [ map { ( $_->{sectors}[$_] - $attn ) / 100 } for @{$_->{secto +rs}} ] } @in; } else { # @in is empty print "\n No data to map for Panda machine. Check if input logfile + is empty"; kill; }
We got rid of a huge amount of wasted memory. That @Attn list was humongous in your code, and we've pared it down to 1/14th and now to a single scalar (which we calculate as we need it, then discard it once no longer needed). And we got rid of unnecessary items in your sets, though, in the interests of readability, I increased the memory usage of the input by making it an AoH's. That's probably a wash, if not slightly bigger now than it was.

Hope that helps.


In reply to Re: Can you help profile this? by Tanktalus
in thread Can you help profile this? by Fighter2

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":



  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.