- You create a list of IPs you find in @item;
- You then pass them as a list into uniq_ip(@item)
- Where you make a copy of that list my @list = @_;
- You then make another copy when you pass it to map map { $_ => 1 } @list
- Which you use to create an anonymous hash { map { $_ => 1 } @list }
- Then you create another list from its uniq keys keys %{{ map { $_ => 1 } @list }}
- Which you assign to an array my @uniq_ip = keys %{{ map { $_ => 1 } @list }};
- Which is returned as another list (last statement of the subroutine).
- Which you assign to another array @uniq_out = uniq_ip(@item);
- Which you iterate over foreach(@uniq_out)
That makes 9 or 10 copies of the list. It's no wonder you're running out of memory.
Try this. It avoids most of those copies:
use strict;
use warnings;
open(OUTNOMATCH, ">nomatch.out") or die "Couldn't write to file $!";
open(OUTMATCH, ">match.out") or die "Couldn't write to file $!";
sub match_internal { #Separate internal/external addresses
use Net::IP::Match::Regexp qw( create_iprange_regexp match_ip );
my $my_ip = $_[0];
my $regexp = create_iprange_regexp(
qw( 192.168.0.0/16 10.10.0.0/16 192.3.3.0/23 192.168.24.0/21 10
+.0.0.0/8 )
);
if (match_ip($my_ip, $regexp)) {
print OUTMATCH "$my_ip\n";
}
else {
print OUTNOMATCH "$my_ip\n";
}
}
sub sortme { # sort all addresses
my @array = @_;
my %hashTemp = map { $_ => 1 } @array;
my @array_out = sort keys %hashTemp;
}
sub main_loop { #main loop that performs all logic and munging
my %uniq;
while (<>) {
(my $field1, my $field2) = split /DST=/, $_;
if ($field2 =~ m/(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})/) {
$uniq{ $1 } = 1;
}
}
match_internal( $_ ) while $_ = each %uniq;
}
main_loop();
You're also re-creating this:
my $regexp = create_iprange_regexp(
qw( 192.168.0.0/16 10.10.0.0/16 192.3.3.0/23 192.168.24.0/21 10
+.0.0.0/8 )
);
for every uniq IP you check...which probably doesn't cost you in extra memory, but is hugely wasteful of cpu (time).
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.
Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
Read Where should I post X? if you're not absolutely sure you're posting in the right place.
Please read these before you post! —
Posts may use any of the Perl Monks Approved HTML tags:
- a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
| |
For: |
|
Use: |
| & | | & |
| < | | < |
| > | | > |
| [ | | [ |
| ] | | ] |
Link using PerlMonks shortcuts! What shortcuts can I use for linking?
See Writeup Formatting Tips and other pages linked from there for more info.