in reply to Process multiple directories

Hello Anonymous Monk,

I applied minor modifications on your code, sample bellow:

#!/usr/bin/perl
use strict;
use warnings;
use Net::Subnet;
use Date::Manip;
use Data::Dumper;
use File::Find::Rule;
use IO::Uncompress::Bunzip2 ();

my @LogDirs = ('/home/tinyos/Monks/TestDir',
           '/home/tinyos/Monks/TestDir2'); # add more here

#Find relevant files
sub get_files {
    my (@dates) = @_;
    my @dirss;
    foreach my $logDir (@LogDirs) {
    foreach my $dateDir (@dates){
        push @dirss, join("/", $logDir, $dateDir);
    }
    }
    my $level = shift // 3; # level to dig into
    my @files = File::Find::Rule->file()
    ->name( '*.bz2','*.log' ) #can insert regex too
    ->maxdepth($level)
    ->in(@dirss);
    return @files;
}

#Matches IP address only
sub searchForIP {
    my ($files, $ip) = @_;
    my @files = @$files;

    for my $file (@files){
    my $filename = $file;
    my $fh = IO::Uncompress::Bunzip2->new($filename)
        or die "bunzip2 $filename: $IO::Uncompress::Bunzip2::Bunzip2Er
+ror";

    while (<$fh>){
        print "$filename:$.:$_" if /$ip/;
    }
    }
}

#This portion contains some code for user input, I will leave this out
+ cause its not related to my problem
my $numdays = 2;
my $numberOfDays = $numdays .' days';

my $dateStart = ParseDate("01/01/2018"); # change that to the user inp
+ut format (d/m/y)
my $dateEnd = DateCalc($dateStart, $numberOfDays);

# To find the every day date1 to date2
my @dates = ParseRecur("0:0:0:1:0:0:0","",$dateStart, $dateEnd);
my @datesFormatted = map { UnixDate($_, '%Y-%m-%d') } @dates;

my @filess = get_files(@datesFormatted);
print Dumper \@filess;
my $ip = '127.0.0.1';
searchForIP(\@filess, $ip);

__END__

$ perl test.pl
$VAR1 = [
          '/home/tinyos/Monks/TestDir/2018-01-01/access.log',
          '/home/tinyos/Monks/TestDir/2018-01-01/sys.log',
          '/home/tinyos/Monks/TestDir/2018-01-02/access.log',
          '/home/tinyos/Monks/TestDir/2018-01-02/sys.log',
          '/home/tinyos/Monks/TestDir/2018-01-03/access.log',
          '/home/tinyos/Monks/TestDir/2018-01-03/sys.log',
          '/home/tinyos/Monks/TestDir2/2018-01-01/access.log',
          '/home/tinyos/Monks/TestDir2/2018-01-01/sys.log',
          '/home/tinyos/Monks/TestDir2/2018-01-02/access.log',
          '/home/tinyos/Monks/TestDir2/2018-01-02/sys.log',
          '/home/tinyos/Monks/TestDir2/2018-01-03/access.log',
          '/home/tinyos/Monks/TestDir2/2018-01-03/sys.log'
        ];
/home/tinyos/Monks/TestDir/2018-01-03/sys.log:1:127.0.0.1 This is insi
+dent 1 in 2018-01-05
/home/tinyos/Monks/TestDir/2018-01-03/sys.log:4:127.0.0.1 This is seco
+nd insident 4 in 2018-01-05
/home/tinyos/Monks/TestDir2/2018-01-03/sys.log:1:127.0.0.1 This is ins
+ident 1 in 2018-01-05
/home/tinyos/Monks/TestDir2/2018-01-03/sys.log:4:127.0.0.1 This is sec
+ond insident 4 in 2018-01-05

[download]

Since we do not have your input data and samples of your directories, we only guess. Also for example what is the dates that you are checking? For example the format of the directories (dates) is "2018-01-01" or different. Just by assuming all the time we will not be able to assist you fast enough.

Having said that, on the code above I modified the dates. I assume your code was failing to parse the dates because you had wrong format. Always see the expected input to the module and your input. I simplified it to the user input "01/01/2018" format (d/m/y). Based on this modification I assume it will match your directories.

Also observe that your @LogDirmy and @LogDir2 are strings based on your sample of code why you need to use Arrays? I have modified the code above to a sample of different directory paths that you could use.

I hope this sample of code resolves all your problems, alternatively update your question with more data so we can understand where the problem is coming from.

Hope this helps, BR.

Seeking for Perl wisdom...on the process of learning...not there...yet!

Replies are listed 'Best First'.
Re^2: Process multiple directories
by Anonymous Monk on Jan 09, 2018 at 01:04 UTC
    Hi again my saviour, that have completely solved my problem. Thank you so much! :D
[reply]

      Hello Anonymous Monk,

      I am glad it worked, keep up the good work experiment with your code and you will see at the end you will become very good.

      BR / Thanos

      Seeking for Perl wisdom...on the process of learning...not there...yet!
[reply]
[d/l]
[select]
Re^2: Process multiple directories
by Anonymous Monk on Jan 09, 2018 at 02:03 UTC
    Hi thanos1983, I am also quite interested in this topic, I am a new to programming hence my questions may be quite basic. May I ask what happens if user wants to enter multiple IP addresses instead of only 1 IP address? How can you tackle with that?
[reply]

      Hello Anonymous Monk,

      The fastest and minor modification to apply is simply concatenate the IP(s) with a column. For example: my $ip = '127.0.0.1|127.0.0.2'; you can easily add as many IP(s) you want following the same pattern. There are many ways to achieve what you want but this is the easiest and less modifications needed based on the code above. See the output bellow based on testing on my local configurations. If you need further assistance let us know.

      Output using the code above:

      $ perl test.pl
/home/tinyos/Monks/TestDir/2018-01-03/sys.log:1:127.0.0.1 This is insi
+dent 1 in 2018-01-05
/home/tinyos/Monks/TestDir/2018-01-03/sys.log:2:127.0.0.2 This is insi
+dent 2 in 2018-01-05
/home/tinyos/Monks/TestDir/2018-01-03/sys.log:4:127.0.0.1 This is seco
+nd insident 4 in 2018-01-05
/home/tinyos/Monks/TestDir2/2018-01-03/sys.log:1:127.0.0.1 This is ins
+ident 1 in 2018-01-05
/home/tinyos/Monks/TestDir2/2018-01-03/sys.log:2:127.0.0.2 This is ins
+ident 2 in 2018-01-05
/home/tinyos/Monks/TestDir2/2018-01-03/sys.log:4:127.0.0.1 This is sec
+ond insident 4 in 2018-01-05

      [download]

      Hope this helps, BR.

      Seeking for Perl wisdom...on the process of learning...not there...yet!
[reply]
[d/l]
[select]
Re^2: Process multiple directories
by dotowwxo (Acolyte) on Jan 09, 2018 at 03:00 UTC
    Hi thanos, I am the anon monk that posted the questions. Could not sign in to my account previously because most of the time I am using the company network to post the questions. If its not too much to ask for, I still have some questions in doubt.

    1. What happens if I have another directory, e.g., named TESTDIRECTORY that contains log files in this format: u_exYYMMDD.log, how can I also grep all my relevant IPs from these directories that does not have the date in the directory name? Here is the sample files in TESTDIRECTORY 

    u_ex171201.log u_ex171202.log u_ex171203.log u_ex171204.log

    [download]
    Note that the u_ex is a prefixed string.

    2. As the other anon monk have asked, how do I match with multiple IP addresses under this portion of the code? My sample input would be like 192.168.1.0,192.168.1.1,192.168.1.2 

    sub searchForIP { 
    my ($files, $ip) = @_; 
    my @files = @$files; 
    for my $file (@files){ 
        my $filename = $file; 
        my $fh = IO::Uncompress::Bunzip2->new($filename) or die "bunzi
+p2 $filename: $IO::Uncompress::Bunzip2::Bunzip2Error"; 
        while (<$fh>){ 
            print "$filename:$.:$_" if /$ip/; 
        } 
    } 
}

    [download]
    Any other response will be greatly appreciated and thank you so much y'all.. :)

    2018-01-09 Athanasius restored deleted content

[reply]
[d/l]
[select]
      I have already resolved these questions. Thank you everyone for all your help! Cheers!
[reply]

In Section Seekers of Perl Wisdom