ahuang14 has asked for the wisdom of the Perl Monks concerning the following question:

Hey guys,

I am trying to output logs in Snort and parse them in Perl in real time. I have gotten the parsing code down and the logging from Snort down. Now I am figuring out a way to do both at the same time. Since a log from Snort is similar to this: 

[**] hiiii [**]
06/09-13:41:55.785538 00:1A:A0:D3:BD:62 -> 00:1E:7A:20:C8:83 type:0x80
+0 len:0x1F0
192.17.123.101:49437 -> 131.118.254.38:80 TCP TTL:128 TOS:0x0 ID:25055
+ IpLen:20 DgmLen:482 DF
***AP*** Seq: 0xF6643B04  Ack: 0x7F9ACEDF  Win: 0x4074  TcpLen: 20
47 45 54 20 2F 35 2F 31 34 30 38 2F 31 33 38 38  GET /5/1408/1388
2F 32 30 30 35 31 31 30 34 30 33 2F 31 61 31 61  /2005110403/1a1a
31 61 64 39 34 38 62 65 32 37 38 63 66 66 32 64  1ad948be278cff2d
39 36 30 34 36 61 64 39 30 37 36 38 64 38 34 38  96046ad90768d848
62 34 31 39 34 37 61 61 31 39 38 36 2F 73 61 6D  b41947aa1986/sam
70 6C 65 5F 69 54 75 6E 65 73 2E 6D 6F 76 2E 7A  ple_iTunes.mov.z
69 70 20 48 54 54 50 2F 31 2E 31 0D 0A 41 63 63  ip HTTP/1.1..Acc
65 70 74 3A 20 69 6D 61 67 65 2F 6A 70 65 67 2C  ept: image/jpeg,
20 69 6D 61 67 65 2F 67 69 66 2C 20 69 6D 61 67   image/gif, imag
65 2F 70 6A 70 65 67 2C 20 2A 2F 2A 0D 0A 52 65  e/pjpeg, */*..Re
66 65 72 65 72 3A 20 68 74 74 70 3A 2F 2F 73 75  ferer: http://su
70 70 6F 72 74 2E 61 70 70 6C 65 2E 63 6F 6D 2F  pport.apple.com/
6B 62 2F 48 54 31 34 32 35 0D 0A 41 63 63 65 70  kb/HT1425..Accep
74 2D 4C 61 6E 67 75 61 67 65 3A 20 65 6E 2D 55  t-Language: en-U
53 0D 0A 55 73 65 72 2D 41 67 65 6E 74 3A 20 4D  S..User-Agent: M
6F 7A 69 6C 6C 61 2F 34 2E 30 20 28 63 6F 6D 70  ozilla/4.0 (comp
61 74 69 62 6C 65 3B 20 4D 53 49 45 20 38 2E 30  atible; MSIE 8.0
3B 20 57 69 6E 64 6F 77 73 20 4E 54 20 36 2E 31  ; Windows NT 6.1
3B 20 57 69 6E 36 34 3B 20 78 36 34 3B 20 54 72  ; Win64; x64; Tr
69 64 65 6E 74 2F 34 2E 30 3B 20 2E 4E 45 54 20  ident/4.0; .NET 
43 4C 52 20 32 2E 30 2E 35 30 37 32 37 3B 20 53  CLR 2.0.50727; S
4C 43 43 32 29 0D 0A 55 41 2D 43 50 55 3A 20 41  LCC2)..UA-CPU: A
4D 44 36 34 0D 0A 41 63 63 65 70 74 2D 45 6E 63  MD64..Accept-Enc
6F 64 69 6E 67 3A 20 67 7A 69 70 2C 20 64 65 66  oding: gzip, def
6C 61 74 65 0D 0A 48 6F 73 74 3A 20 61 31 34 30  late..Host: a140
38 2E 67 2E 61 6B 61 6D 61 69 2E 6E 65 74 0D 0A  8.g.akamai.net..
43 6F 6E 6E 65 63 74 69 6F 6E 3A 20 4B 65 65 70  Connection: Keep
2D 41 6C 69 76 65 0D 0A 0D 0A                    -Alive....

=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+
+=+=+

[download]

My current Perl code works for an output file consisting of many of these. I read @array = <SNORTFILE> so this would be in an array format. My question is, is there any way instead of using File::Tail in this way:

use File::Tail; $file=File::Tail->new("/some/log/file"); while (defined($line=$file->read)) { print "$line"; }

to read each line, can I read each alert and parse that? Or is there another method I am not thinking of? Reading line by line, I can't figure out a way to say anything after the hiiii and the GET, I should keep.

Hopefully this isn't too confusing. Thanks in advance.

Alvin

Update: This is what I have so far. When I run it, it gives me this output. I dont know why it doesn't write to a file nor gives me the GET string correctly. I have almost the same code running on a text file and it outputs perfectly.

CODE: 

#file to write to 
$writeFile = "C:\\Snort\\log\\test3\\write.log";
open WRITEFILE, ">$writeFile" or die "Could not open write file.";

use File::Tail;
$file= File::Tail -> new ("C:\\Snort\\log\\test3\\logto2.log");
@alert; 

while (defined ($linea = $file->read)){
    if($linea =~m/^\=\+\=/){
        process(@alert);
        @alert = ();
    }else{
        push @alert, $linea;
    }
}

sub process{
    @lines = @_;
    $foundGET = ();
    $foundHTTP = ();
    $foundAccept = (); 
    $counter = 0;
    @locations =();  
    
    #keeps track of where the separate alerts start 
    foreach $line (@lines){ 
        if ($line =~ /hiiii/){
            push (@locations, $counter);  
        }#end if
        $counter++;
    }#end foreach
    
    foreach $number (@locations){
    #for the timestamp
    $timeStamp = substr (@lines[$number+1], 0, 14);
    print "Time: $timeStamp\n"; 
    print WRITEFILE "Time: $timeStamp\n";
    
    #for source IP 
    $thirdLine = @lines[$number+2]; 
    $srcIPColonIndex = index ($thirdLine, ':');
    $sourceIP = substr ($thirdLine, 0, $srcIPColonIndex);
    print "Source IP: $sourceIP\n";
    print WRITEFILE "Source IP: $sourceIP\n";
    
    #for source port 
    $thirdLine = substr($thirdLine, $srcIPColonIndex+1);
    $srcPortSpace = index ($thirdLine, ' '); 
    $sourcePort = substr($thirdLine, 0, $srcPortSpace); 
    print "Source Port: $sourcePort\n";
    print WRITEFILE "Source Port: $sourcePort\n";
    
    # for destination IP 
    $thirdLine = substr($thirdLine, index($thirdLine, '>')+2); 
    $destIPColonIndex = index ($thirdLine, ':');
    $destIP = substr ($thirdLine, 0, $destIPColonIndex); 
    print "Destination IP: $destIP\n";
    print WRITEFILE "Destination IP: $destIP\n";
    
    #for destination port 
    $thirdLine = substr($thirdLine, $destIPColonIndex+1); 
    $destPortSpace = index ($thirdLine, ' '); 
    $destPort = substr($thirdLine, 0, $destPortSpace);
    print "Destination Port: $destPort\n";
    print WRITEFILE "Destination Port: $destPort\n";
    
    #for protocol
    $thirdLine = substr($thirdLine, $destPortSpace+1);
    $protocolSpace = index ($thirdLine, ' '); 
    $protocol = substr($thirdLine, 0, $protocolSpace); 
    print "Protocol: $protocol\n"; 
    print WRITEFILE "Protocol: $protocol\n";
    
    $counterGET=0;
    $counterHTTP=0;
    
    #for GET 
    while ($foundGET=~ ()){
        if(@lines[$counterGET]!~ m/GET/){
            $counterGET++;
        }#end if
        else {
            $foundGET = !();
        }#end else
    }#end while
    
    #for HTTP 
    while ($foundHTTP =~ ()){
        if(@lines[$counterHTTP]!~ m/Accept/ && @lines[$counterHTTP]!~ 
+m/HTTP/){
            $counterHTTP++; 
        }#end if
        else {
            $foundHTTP = !();
        }#end else
    }#end while 
    
    $fullString = "";
    @newArray = (); 
    
    #gets all the data packet lines needed 
    for ($count = $counterGET; $count <= $counterHTTP+1; $count++){
        push @newArray, @lines[$count];
    }#end for
    
    #chops the hex out of the get to http lines
    foreach $line3 (@newArray){
        $line3 = substr ($line3, 49);
    }#end foreach 
    
    chomp (@newArray);
    
    $fullString = join ("", @newArray);
        
    $indexOfAccept = index ($fullString, 'Accept'); 
    $fullString = substr($fullString, 0, $indexOfAccept-2);
    
    print "GET Trace: $fullString\n";
    print WRITEFILE "GET Trace: $fullString\n\n";
    print "\n";
    }#end foreach
}#end sub

[download]

Wrrong output: 

C:\Users\Administrator\Documents>c:\users\administrator\desktop\dummy.
+pl
Time: 06/23-14:34:16
Source IP: 172.17.116.101
Source Port: 49214
Destination IP: 131.118.254.38
Destination Port: 80
Protocol: TCP
GET Trace: 7A:20:C8:83 type:0x800 len:0x1

Time: 06/23-14:34:23
Source IP: 172.17.116.101
Source Port: 49214
Destination IP: 131.118.254.38
Destination Port: 80
Protocol: TCP
GET Trace:

Terminating on signal SIGINT(2)

C:\Users\Administrator\Documents>

[download]

Good output parsing a file that is already made: 

Time: 06/17-10:30:26
Source IP: 172.17.116.101
Source Port: 49279
Destination IP: 65.254.57.211
Destination Port: 80
Protocol: TCP
GET Trace: GET /ad.php?do=html&zid=14678&wd=728&ht=90&target=_top HTTP
+/1.1

Time: 06/17-10:30:27
Source IP: 172.17.116.102
Source Port: 49280
Destination IP: 65.254.57.211
Destination Port: 80
Protocol: TCP
GET Trace: GET /af.php?do=imp&zid=14678&aid=108360&auth=C26DDAD23C&wd=
+728&ht=90&t=1308321036 HTTP/1.1

[download]

If anyone has the time to go over this I would appreciate it. I don't know why my code doesn't write or get the right lines. Notice how the last packet doesnt even finish the GET string part.

Replies are listed 'Best First'.
Re: Help with Snort and File::Tail
by ikegami (Patriarch) on Jun 21, 2011 at 21:16 UTC 
    use File::Tail;

my $tail = File::Tail->new("/some/log/file");

my @alert;
while (defined(my $line = $file->read)) {
   if ($line =~ /^=\+=/) {
       process_alert(\@alert);
       @alert = ();
   } else {
       push @alert, $line;
   }
}

process_alert(\@alert) if @alert;

    [download]
[reply]
[d/l]
      Ah. This is pretty good. So in the process_alert, it would just be tailored to process ONE of the alerts rather than a whole group?
[reply]

        You asked how to "read each alert". You haven't been clear as to what you consider an alert, but I presume what you posted is an example of one. If that's the case, then I gave code that does what you ask.

        Even less clear is "I read @array = <SNORTFILE> so this would be in an array format.", which I took to mean you wanted the alert in an array, one element per line. It would just as easy to extract each alert as a (multiline) string, if that's what you prefer.

        You also asked how to parse the alert, but you didn't specify what information you wanted to extract, so I didn't touch that.

[reply]
[d/l]
      And this might be a really stupid question but what is the \ in process_alert(\@alert)? Is that how you pass in a parameter in Perl? (Sorry, total novice).
[reply]

        \@alert creates a reference to the array @alert, and the reference is indeed being passed as an argument to a to-be-provided sub process_alert.

[reply]
[d/l]
[select]

      Another probably stupid question.... What is the process_alert(\alert) if @alert;? Is this not a subroutine? can I just do

      sub process{ my @array = @_; }

[reply]
        Yes, you can do that. ikegami's code passed a reference to the array, which is better if the array is large, or necessary if you are passing multiple arguments (e.g. multiple arrays) to a function. I'm not sure it matters here. And then the sub would be something like:
        sub process {
  my $alert = shift;
  print "$_" for @$alert;
  # Or (since I'm not sure what you're doing with this)
  for my $line (@$alert) {
    # process $line of alert
  }
}

        [download]
        If you asking about the 'if @alert' part, then that's just like:
        if (@alert) {
  process(@alert)
}

        [download]
        and the array (@alert) is true in boolean context if it contains any elements (false if it has zero, of course).
[reply]
[d/l]
[select]
Re: Help with Snort and File::Tail
by BrowserUk (Patriarch) on Jun 21, 2011 at 20:46 UTC
    Or is there another method I am not thinking of?

    Can't you start snort from within your perl script using a piped open, so you read the file before it is written to disk?

    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
      I don't know off the top of my head how to do this since I just started Perl and I haven't looked at this specifically but I am certain you can. What do you mean though? I don't really care that the Snort log is in snort.log. I more care that I grab everything that is being written to it (lets say check every 20 seconds with File::Tail) and parse it with Perl, and output it somewhere else.
[reply]

        He is suggesting that if snort were to be configured to send output its output to stdout*, your script could act like a filter (like grep), so you wouldn't have to use File::Tail and the process would be more reliable.

        You'd still have to identify alerts, thought.

        * — I'm not familiar with snort. This could be trivial or impossible.

[reply]

Back to Seekers of Perl Wisdom