in reply to Re: Grep logs by start date and end date in different directories
in thread Grep logs by start date and end date in different directories

Hi, that helped me loads. Thanks a lot. However, is it possible if I put ->name('*.bz2') ? This is because all my log files are compressed into bz2 format. I have tested it but it didnt seem to work when I put *.bz2. It only works when I put it as *.log. Any idea why? Once again, thank you so much.
  • Comment on Re^2: Grep logs by start date and end date in different directories

Replies are listed 'Best First'.
Re^3: Grep logs by start date and end date in different directories
by Anonymous Monk on Jan 05, 2018 at 01:53 UTC
    Sorry to bother again but I am kinda urgent on this so I am working this on my own but at the same time I hope i get more insights from professionals which can allow me to do it in a better way. My current script also searches for IP in a network range from all the log file. This is the code that does what I've mentioned:
    use Net::Subnet; if (@ARGV){ while (<>) { my @ips = m/(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}( +?:25 +[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)/g; next unless @ips; next unless grep { $matcher->($_) } @ips; print $fh $_; }
    Do you know how I can implement this into your code? Thanks again
        Hi haukex, thank you so much for your reply. Managed to get it to work. However what happens if I want to read multiple bz2 files instead of 1 bz2 file? My current code is:
        my $filename = "debug.log.bz2"; my $fh = IO::Uncompress::Bunzip2->new($filename) or die "bunzip2 $filename: $IO::Uncompress::Bunzip2::Bunzip2Error" +; while(<$fh>){ #do something }

      Hello Anonymous Monk,

      Apologies for the late reply, but I just noticed your reply to my comment.

      It is very open your questions, I am not sure what do you mean with My current script also searches for IP in a network range from all the log file. network range can vary greatly. Give a bit more specific information e.g. 127.0.0.1 - 127.0.0.255 what is the range, how the IP will be imported? I mean you will import IP e.g. 127.0.0.1 and you want to check what IP are matching what the network, subnet, range? On your log files is this exact IP exists? Or are you looking for any number that consists of 1-255.1-255.1-255.1-255?

      We need sample of data in the files to see the format. For example you just mentioned that you are having bz2 files and fellow Monk haukex proposed a module and a few similar questions.

      So help us with more specific information to help you.

      Hope this helps, BR.

      Seeking for Perl wisdom...on the process of learning...not there...yet!
        Hi again, sorry for the lack of information. What I meant is when user provide a network address with a CIDR range such as 192.168.1.0/27, all IP address under that subnet will be printed. As for the log files, there are IP addresses in every line of different logs. I could not provide the data because it is confidential. Thank you again
        Update, I managed to get the IP address subnet range searching to work. However, to implement it into your code, I am unsure why when i put ->name('*.bz2') it doesn't work.. This is my code currently:
        use strict; use warnings; use IO::Uncompress::Bunzip2 (); use Net::Subnet; my $filename = "debug.log.bz2"; my $fh = IO::Uncompress::Bunzip2->new($filename) or die "bunzip2 $filename: $IO::Uncompress::Bunzip2::Bunzip2Error" +; my $matcher = subnet_matcher qw(72.46.130.0/24); while (<$fh>){ my @ips = m/(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25 +[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)/g; next unless @ips; next unless grep { $matcher->($_) } @ips; print; }
        And this is your code that I am unsure of how to search for all .bz2 files:
        sub get_files { my (@dirs) = @_; my $level = shift // 3; # level to dig into my @files = File::Find::Rule->file() ->name('*.bz2') #this line doesn't work ->maxdepth($level) ->in(@dirs); return @files; }
        Hi, please ignore all my replies above as I have already solved it myself (which I am amazed cause I suck at programming)... However, I have one last question, for this part of the code:
        sub get_files { my (@dirs) = @_; my $level = shift // 3; # level to dig into my @files = File::Find::Rule->file() ->name( '*.bz2' ) #can insert regex too ->maxdepth($level) ->in(@dirs); return @files; }
        This particular line:
        my (@dirs) = @_;
        When I change it to my own log file path, it will replace the <date>/<filename.bz2> which will returns nothing in result. How can I specify my own log path?