networkdude has asked for the wisdom of the Perl Monks concerning the following question:

Hi Team,

I have two files; a file with the IP addresses I am looking for (IPS.txt) and a file of 40K firewall rules (rules.txt).

I am trying to use a while to load the file and use an array for the IP addresses. The foreach loop should look up each IP address in the rules file and print each matching line.

It actually works if I search for static text, but as soon as I use my array to plug in the values, it finds nothing. I wonder if it is the formatting of IP addresses or something. I'm stuck.

Here's the script:

open ( IPS, '<', "GIS_DEV_IPS.txt" ) || die "can't open IPS!"; open ( RULES, '<', "rules.txt" ) || die "can't open file!"; @IPS = <IPS>; close (IPS); chomp (@IP); while($line = <RULES>) { foreach $IP (@IPS) { if ($line =~ $IP) { print $line; } } } exit;

Replies are listed 'Best First'.
Re: Sifting through firewall rules using a script
by NetWallah (Canon) on Dec 23, 2021 at 02:12 UTC
    As GrandFather says - "use strict;" would have told you that you are "chomp"ing the wrong variable name.

    In addition (besides other minor nits), your regex to match the rules could result in many false positives.

    I suggest

    $line =~ /\b\Q$IP\E\b/
    There are also optimizations possible depending on the number of IP addresses.
    If it is < ~4k, you can combine them into a single regex with alternation, or change which file you choose to read into memory.

    Feel free to ask specific (properly formatted) questions, using <code> tags for code (Writeup Formatting Tips).

                    "If you had better tools, you could more effectively demonstrate your total incompetence."

Re: Sifting through firewall rules using a script
by GrandFather (Saint) on Dec 23, 2021 at 01:46 UTC

    Add strictures (use strict; use warnings; - see The strictures, according to Seuss), fix the errors, then if you still have a problem come back and report. You might like to give us some sample data to run against your script if you do come back with an issue.

    Optimising for fewest key strokes only makes sense transmitting to Pluto or beyond
Re: Sifting through firewall rules using a script
by Fletch (Bishop) on Dec 23, 2021 at 13:29 UTC

    Depending on the number of IPs it might be slightly better performance if you used Regexp::Common and $RE{net}{IPv4} to extract out IP-address-looking-things from rule lines, then checked those against a hash for hits of interesting ones. Rather than running n different regular expressions over each firewall rule line n times, this runs one over only as many times as necessary to get out the IP-ish things in any given rule line and then you're doing a (much faster) hash existence test to see if that IP is a wanted one.

    use Regexp::Common qw( net ); my %interesting_ips; while( <IPS> ) { chomp; $interesting_ips{ $_ } = 1; } while( my $fw_line = <RULES> ) { while( my( $addr ) = $fw_line =~ m{($RE{net}{IPv4})}g ) { say qq{$.: found interesting address '$addr'} if exists $interesting_ips{ $addr }; } }

    The cake is a lie.
    The cake is a lie.
    The cake is a lie.

      Hi Team! I was on Christmas break so just got back and found your amazing answers! THANK YOU SO MUCH!

      I have adjusted the script according to your collective feedback and I also realised I need to explain a bit more what I am trying to do

      I have two files; one file with all my interesting IPs in it; and another file with all the firewall rules in it. I need to sift through the firewall rules and PRINT THE ENTIRE FW RULE LINE in the FW rules file if it matches any of the IPs in the interesting IP address file.

      My interesting IP file:

      1.1.1.1 2.2.2.2 10.198.0.0 3.3.3.3 10.198.1.0
      My FW rules file:
      133 bba33132-6192-51e8-4d78-c1b7bfd47251 any V072-AklC-DB MOSSACSQLAdm +inGroup 10.198.0.0/16 10.210.0.0/16 MOSS_SQLAD_10-208-22-1/28 accept + always RDP TCP17338 TCP18230 PING all change 125213 157309 + 136 a5ea4ee8-6192-51e8-0252-2017208af83d any V071-AklC-Web ACHenderson +RDPUsers 10.198.1.0/16 10.210.0.0/16 MOSS_InternalWeb_10-208-22-16/28 + accept always FTP all
      My adjusted script:
      use strict; use warnings; use Regexp::Common qw( net ); open ( IPS, '<', "TEST_IPS.txt" ) || die "can't open IPS!"; open ( RULES, '<', "test_rules.txt" ) || die "can't open file!"; my %interesting_ips; while( <IPS> ) { chomp; $interesting_ips{ $_ } = 1; } while( my $fw_line = <RULES> ) { while( my( $addr ) = $fw_line =~ m{($RE{net}{IPv4})}g ) { print ( $fw_line); } }
      When I run the script, two things happen:
      1. It only matches the first match and never moves on to the second match, and
      2. It does not exit and I have to ctrl+c to get it to stop filling the screen with the first match

      What am I doing wrong please?

      I can repay the help in honey from my backyard beehives. It's very very yummy!

        Minor stylistic nitpicks before anything substantive: lexical filehandles are better than bareword globals like you've used, and you want to include $! in the error message so you know why the open failed.

        open( my $ips, '<', 'TEST_IPS.txt' ) or die "Can't open IPS: $!\n";

        That out of the way you're close but you don't consult %interesting_ips in any way to tell if you have a match. You also want to pull the list of matching IPs out differently.

        #!/usr/bin/env perl use strict; use warnings; use Regexp::Common qw( net ); open ( my $ips_fh, '<', "TEST_IPS.txt" ) or die "can't open IPS: $!\n" +; open ( my $rules_fh, '<', "test_rules.txt" ) or die "can't open rules: + $!\n"; my %interesting_ips; while( <$ips_fh> ) { chomp; $interesting_ips{ $_ } = 1; } close( $ips_fh ); while( my $fw_line = <$rules_fh> ) { chomp( $fw_line ); my( @addresses ) = $fw_line =~ m{ ($RE{net}{IPv4}) }gx; for my $addr ( @addresses ) { next unless exists $interesting_ips{ $addr }; print qq{$.:interesting '$addr': $fw_line\n}; } } close( $rules_fh ); exit 0; __END__ $ perl pm_foo.plx 1:interesting '10.198.0.0': 133 bba33132-6192-51e8-4d78-c1b7bfd47251 a +ny V072-AklC-DB MOSSACSQLAdminGroup 10.198.0.0/16 10.210.0.0/16 MOSS_ +SQLAD_10-208-22-1/28 accept always RDP TCP17338 TCP18230 PING all + change 125213 157309 2:interesting '10.198.1.0': 136 a5ea4ee8-6192-51e8-0252-2017208af83d a +ny V071-AklC-Web ACHendersonRDPUsers 10.198.1.0/16 10.210.0.0/16 MOSS +_InternalWeb_10-208-22-16/28 accept always FTP all

        The cake is a lie.
        The cake is a lie.
        The cake is a lie.

        I almost got it to work but no dice. It matches everything. Must be the if line. What does the regex m{($RE{net}{IPv4})}g actually do here? I know what the line means from the documentation for that module, but what is the exact purpose in this script?

        use strict; use warnings; use Regexp::Common qw( net ); open ( IPS, '<', "TEST_IPS.txt" ) || die "can't open IPS!"; open ( RULES, '<', "test_rules.txt" ) || die "can't open file!"; my %interesting_ips; while( <IPS> ) { chomp; $interesting_ips{ $_ } = 1; } while( my $fw_line = <RULES> ) { if ($fw_line =~ m{($RE{net}{IPv4})}g ) { print ( $fw_line); } }
Re: Sifting through firewall rules using a script
by Discipulus (Canon) on Dec 23, 2021 at 08:51 UTC
    Hello networkdude and welcome to the monastery and to the wonderful world ofperl!

    Yes, beside strict and warnings (always use it!) I suggest you to adopt the lexical filehandle opening form: open(my $fh, "<", "input.txt")    or die "Can't open < input.txt: $!";

    Then I suspect that you problem arises in line

    if ($line =~ $IP) {

    because all elements of @IPS end with a \n newline: you can chomp inside the foreach loop too, just before the pattern matching.

    L*

    PS as Tux noted you are already chomping, but the wrong array: use strict; use warnings; always :)

    There are no rules, there are no thumbs..
    Reinvent the wheel, then learn The Wheel; may be one day you reinvent one of THE WHEELS.