How to grep matching IP address from a log file?

dotowwxo has asked for the wisdom of the Perl Monks concerning the following question:

Hi all, currently i have this script that looks for IP address that is in the netmask specified in $matcher and print that line from mylogFile.log


#!/usr/bin/perl

use strict;
use warnings;
use Net::Subnet;

my $filename = 'mylogFile.log';
open(my $fh, '<:encoding(UTF-8)', $filename)
  or die "Could not open file '$filename' $!";
my @cidr_list = <$fh>;

my $matcher = subnet_matcher qw(192.168.1.0/22);

my @grepIPs = grep(/\b(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\./,@cidr_
+list);

for (my $i=0; $i < scalar @grepIPs; $i++){    

    my $boolean = $matcher->($grepIPs[$i]);
    if ($boolean == 1){
        print $grepIPs[$i];
    }
    else{
        print "Nothing\n";
    }
}
[download]

However, $grepIPS only matches lines that only have IP address specified without any other strings in that particular line.

For e.g., my script only matches lines that only has IP address:

192.168.1.2
192.168.1.3
192.168.1.4
[download]

it does not matches lines that has other strings involved as shown below:

2017-12-08 07:01:39 <my source IP> GET /course-detail.aspx id=66&catCo
+lor=0 443 - 192.168.1.0 (e.g., IP) <server version etc. strings> 200 
+0 0 530
[download]

What I want as my output: I want to grep only the IP addresses (not the whole line which I can't feed into my for loop to check) from the line so that I can feed it into my for loop to see if the IP is in the netmask, and if yes, I would like to print the whole line. Can anyone guide me on how I can achieve this? any response is kindly appreciated, thank you.

 
Sample output that I want to achieve: 

2017-12-08 07:01:39 <my source IP> GET /course-detail.aspx id=66&catCo
+lor=0 443 - 192.168.1.1 (e.g., IP) <server version etc. strings> 200 
+0 0 530
2017-12-08 07:01:39 <my source IP> GET /course-detail.aspx id=66&catCo
+lor=0 443 - 192.168.1.2 (e.g., IP) <server version etc. strings> 200 
+0 0 530
2017-12-08 07:01:39 <my source IP> GET /course-detail.aspx id=66&catCo
+lor=0 443 - 192.168.1.3 (e.g., IP) <server version etc. strings> 200 
+0 0 530

#Note that I only want to grep IP addresses (192.168.1.1-3) that are i
+n the same netmask specified in $matcher
[download]

Comment on How to grep matching IP address from a log file? Select or Download Code

Replies are listed 'Best First'.
Re: How to grep matching IP address from a log file? by siberia-man (Friar) on Dec 19, 2017 at 09:21 UTC
Hi, I understand that gurus have responded you. Nevertheless, I'd like to add my 5 cents. What I've seen as a solution after I completed to read the thread. 1. grep input for any lines matching IP pattern 2. IPs after (1) should be tested for matching the IP list from the specified subnet Before posting my answer I created tiny toy proving correctness of my approach: #!/usr/bin/perl use strict; use warnings; use Net::Subnet; my $matcher = subnet_matcher qw(192.168.1.0/22); while ( <DATA> ) { my @ips = m/(?:(?:25[0-5]\|2[0-4][0-9]\|[01]?[0-9][0-9]?)\.){3}(?:25 +[0-5]\|2[0-4][0-9]\|[01]?[0-9][0-9]?)/g; next unless @ips; next unless grep { $matcher->($_) } @ips; print; } __DATA__ Sample output that I want to achieve: 2017-12-08 07:01:39 127.0.0.1 GET /course-detail.aspx id=66&catColor=0 + 443 - 127.0.0.1 (e.g., IP) <server version etc. strings> 200 0 0 530 2017-12-08 07:01:39 127.0.0.1 GET /course-detail.aspx id=66&catColor=0 + 443 - 192.168.1.2 (e.g., IP) <server version etc. strings> 200 0 0 5 +30 2017-12-08 07:01:39 127.0.0.1 GET /course-detail.aspx id=66&catColor=0 + 443 - 192.168.2.1 (e.g., IP) <server version etc. strings> 200 0 0 5 +30 2017-12-08 07:01:39 127.0.0.1 GET /course-detail.aspx id=66&catColor=0 + 443 - 192.168.1.3 (e.g., IP) <server version etc. strings> 200 0 0 5 +30 2017-12-08 07:01:39 127.0.0.1 GET /course-detail.aspx id=66&catColor=0 + 443 - 192.168.4.3 (e.g., IP) <server version etc. strings> 200 0 0 5 +30 #Note that I only want to grep IP addresses (192.168.1.1-3) that are i +n the same netmask specified in $matcher [download] Please pay attention that netmask `/22` corresponds to wide range of IPs `192.168.1.0` - `192.168.3.255`.	[reply] [d/l] [select]
Re^2: How to grep matching IP address from a log file? by dotowwxo (Acolyte) on Dec 19, 2017 at 09:42 UTC
Hey! That worked perfectly for me. Thank you so much! Really helped me a lot cause I'm a total newbie :')	[reply]
Re: How to grep matching IP address from a log file? by haukex (Archbishop) on Dec 19, 2017 at 07:46 UTC
You could use Regexp::Common::net - this will extract any IP's from the line: `use warnings; use strict; use Regexp::Common qw/net/; my $regex = qr{ (?<ip> \b $RE{net}{IPv4} \b ) }msx; my $filename = 'mylogFile.log'; open my $fh, '<:encoding(UTF-8)', $filename or die "$filename: $!"; while (my $line = <$fh>) { while ($line=~/$regex/g) { print "<", $+{ip}, ">\n"; } } close $fh;` [download] However, in your sample input you wrote `<my source IP>`. If this is an actual IP address, then that will be matched too. If you don't want that, you need to extend the regex to match the surrounding parts of the line. Also, if that's the case, you haven't shown what the actual lines look like, i.e. your input isn't representative, so we can't really help there. All I can say is that if you only want to match one IP per line, you can replace the inner `while` loop with a single `if`. Have a look at this for some advice on designing regexes. In the code you posted, you are reading the entire file into an array, which is a bit wasteful, and it'd probably be better if you used a regular `while (<$fh>)` loop instead, like what I showed above and as is explained in e.g. Files and I/O and I/O Operators. Also, note that with your current regex, you're just matching the first digits of an IP address, and you may get false positives.	[reply] [d/l] [select]
Re^2: How to grep matching IP address from a log file? by dotowwxo (Acolyte) on Dec 19, 2017 at 08:19 UTC
Hello, thank you so much for your reply. As you have mentioned, there will be 2 Ip addresses per line. One of it is my source IP, the other is the IP addresses I want to extract. These are the fields of the actual line: `date time s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c +-ip cs(User-Agent) sc-status sc-substatus sc-win32-status time-taken` [download] Sample logs 2017-12-08 07:01:39 <s-ip> GET /course-detail.aspx id=66&catColor=0 44 +3 - <c-ip> curl/7.19.7+(x86_64-redhat-linux-gnu)+libcurl/7.19.7+NSS/3 +.27.1+zlib/1.2.3+libidn/1.18+libssh2/1.4.2 200 0 0 530 2017-12-08 07:01:39 <s-ip> GET /course-listing.aspx - 443 - <c-ip> cur +l/7.19.7+(x86_64-redhat-linux-gnu)+libcurl/7.19.7+NSS/3.27.1+zlib/1.2 +.3+libidn/1.18+libssh2/1.4.2 200 0 0 140 2017-12-08 07:01:39 <s-ip> GET /course-detail.aspx id=24&catColor=0 44 +3 - <c-ip> curl/7.19.7+(x86_64-redhat-linux-gnu)+libcurl/7.19.7+NSS/3 +.27.1+zlib/1.2.3+libidn/1.18+libssh2/1.4.2 200 0 0 93 2017-12-08 07:01:40 <s-ip> GET /logistics.aspx - 443 - <c-ip> curl/7.1 +9.7+(x86_64-redhat-linux-gnu)+libcurl/7.19.7+NSS/3.27.1+zlib/1.2.3+li +bidn/1.18+libssh2/1.4.2 200 0 0 46 2017-12-08 07:01:40 <s-ip> GET /course-detail.aspx id=23&catColor=0 44 +3 - <c-ip> curl/7.19.7+(x86_64-redhat-linux-gnu)+libcurl/7.19.7+NSS/3 +.27.1+zlib/1.2.3+libidn/1.18+libssh2/1.4.2 200 0 0 140 [download] c-ip being the IP address I want to extract	[reply] [d/l] [select]
Re^3: How to grep matching IP address from a log file? by haukex (Archbishop) on Dec 19, 2017 at 13:18 UTC
hippo's reply hopefully makes clear: the sample input you have shown appears to still not really be representative of real-world data, because I assume there is a chance that the "`s-port`" may not always be `443`, and the user agent is unlikely to always start with `curl`, etc. Sorry, but what you have told us so far is still not enough information to build a robust parser. For example, if you assumed that the regex I showed will always match twice per line, and then your log file happens to contain an IP address in the `cs-uri-stem cs-uri-query` fields, and/or a hostname in one of the `s-ip c-ip` fields, code based on this assumption will break. The way I would suggest approaching this is to see if you can find out more specifications about the fields - in particular, whether any of the `date time s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip` fields may contain whitespace, and if they can, how they might be delimited, etc. If, for example, the specification says that none of these fields may contain whitespace, it may turn out that LanX's suggestion of split may be enough. But if not, you'll have to write a regex to parse the line.	[reply] [d/l] [select]
Re^3: How to grep matching IP address from a log file? by hippo (Archbishop) on Dec 19, 2017 at 09:02 UTC
c-ip being the IP address I want to extract Since you haven't put this actual address in, it cannot be searched for in the sample data. But that's OK because really you just want whatever is in its place, so this will suffice: #!/usr/bin/env perl use strict; use warnings; while (<DATA>) { my ($ip) = (/ 443 - (.+) curl/); print "Line $.: Found $ip\n"; } __DATA__ 2017-12-08 07:01:39 <s-ip> GET /course-detail.aspx id=66&catColor=0 44 +3 - <c-ip> curl/7.19.7+(x86_64-redhat-linux-gnu)+libcurl/7.19.7+NSS/3 +.27.1+zlib/1.2.3+libidn/1.18+libssh2/1.4.2 200 0 0 530 2017-12-08 07:01:39 <s-ip> GET /course-listing.aspx - 443 - <c-ip> cur +l/7.19.7+(x86_64-redhat-linux-gnu)+libcurl/7.19.7+NSS/3.27.1+zlib/1.2 +.3+libidn/1.18+libssh2/1.4.2 200 0 0 140 2017-12-08 07:01:39 <s-ip> GET /course-detail.aspx id=24&catColor=0 44 +3 - <c-ip> curl/7.19.7+(x86_64-redhat-linux-gnu)+libcurl/7.19.7+NSS/3 +.27.1+zlib/1.2.3+libidn/1.18+libssh2/1.4.2 200 0 0 93 2017-12-08 07:01:40 <s-ip> GET /logistics.aspx - 443 - <c-ip> curl/7.1 +9.7+(x86_64-redhat-linux-gnu)+libcurl/7.19.7+NSS/3.27.1+zlib/1.2.3+li +bidn/1.18+libssh2/1.4.2 200 0 0 46 2017-12-08 07:01:40 <s-ip> GET /course-detail.aspx id=23&catColor=0 44 +3 - <c-ip> curl/7.19.7+(x86_64-redhat-linux-gnu)+libcurl/7.19.7+NSS/3 +.27.1+zlib/1.2.3+libidn/1.18+libssh2/1.4.2 200 0 0 140 [download]	[reply] [d/l]
Re: How to grep matching IP address from a log file? by LanX (Saint) on Dec 19, 2017 at 07:42 UTC
Well you need to `split` the line into words and test the IP field. I can't tell what your field separator is (space or tab) and if the IP is always at the same position. So try `@fields = split /\t/, $line` and `print $line if grep { your_test($_) } @fields` ( tab separator, IP position unclear) Cheers Rolf _{(addicted to the Perl Programming Language and ☆☆☆☆ :) Wikisyntax for the Monastery}	[reply] [d/l] [select]
Re: How to grep matching IP address from a log file? by Anonymous Monk on Dec 19, 2017 at 21:41 UTC
Even if you cannot find a way to use the regular-expressions in Regexp::Common directly, by all means go to them as a source of authoritative expressions that work ... to be "cabbaged" in your source-code as needed.	[reply]