Suggestions:
   You are able to view the data in some way shape or form, so I would
+ either 
      A) figure out how to get that data out in a stream (I.e calling 
+an executable to provide that data on the fly for your parser) or 
      B) instead of pulling the binary data, convert it to text, compr
+ess that and munge the compressed text

   Determine what types of entries are in the file.
   Just looking at the data at hand I see at least 2 unique type of en
+tries and a whole slew of other things to help with a parser. 
      One msg from the kernel, and one msg from firewalld.
      Next we notice that the kernel is "Temporarily" blocking 
         does it also log permanent blocks? 
         does this line correlate to an earier firewalld line?

      The firewalld process is stating it denied a packet..
      There is all sorts of juicy bits in there..

      First off, the deny. 
         What other actions can it take?
      Then the interface.. 
         what other interfaces are there?
      The next number is interesting as I have no idea what its correl
+ated to 
         do all denies get stamped with 48? 
         or packets on eth0, 
         or tcp packets, 
         or tcp packets destined for X port?
      Next the type 
         what other types are coming through?..
[download]

Let me show what I am trying to pull out of the file:

03/13/03 16:44:56 kernel Temporarily blocking host 212.241.116.21

(This is where the inital block occurs, the firewall will then continue to block all attempts from this IP address.I would like to extract all entries in the firewall like this one.)

03/13/03 16:44:57 firewalld103 deny in eth0 48 tcp 20 117 212.241.11.21 209.126.xxx.xxx 4449 80 syn (LO-Proxied-HTTP)

(At this point, the firewall continues to block the attempt. I would like to extract all lines in the firewall that contain this as well...contains very useful information such as ports and packet sizes.)

With that in mind, can I ask for someone to help me build my script? I feel like I am butting my head against a wall. I know I have much to learn, but can learn a lot from seeing the script and breaking it down to see how it functions.

Thanks.

Tarballed

roughly...

...
my %lookfor;
# will look like this at the end
# %lookfor = (
#    192.168.254.1 => {
#                        first => '03/13/03 16:44:56',
#                        last  => '03/13/03 16:59:30',
#                        count => 12,
#                        bytes => 1234
#                     },
#   192.168.254.2 => {
#                           ....
#                    }
# );
#

while (<FWLOG>) {
  if (/(.*?) kernel Temporarily blocking host (.*)/) {
    $lookfor{$2}{first} = $1;
  }
  elsif (/(.*?) firewalld.*? deny in eth0 (\d+) (\w+) (\d+) (\d+) ([\d
+\.]+) (\d+) (\d+) (\w+) (.*)/) { 
    if (exists $lookfor{$6}) {
      $lookfor{$6}{count} += 1;
      $lookfor{$6}{last} = $1;
      $lookfor{$6}{bytes} += $7; # or whichever field is bytes
      # keep track of other info of interest
    }
    # else ignore the uninteresting deny lines
  }
  # else ingore lines we don't care about at all
}

printf "%15s %6s %18s %18s %s\n", qw/ ip count first last bytes /;
foreach my $badguy (keys %lookfor) {
  printf "%15s %6d %18s %18s %d\n",
    $badguy,
    $lookfor{$badguy}{count},
    $lookfor{$badguy}{first}, $lookfor{$badguy}{last},
    $lookfor{$badguy}{bytes};
}
[download]

Here is a snip from the firewall log, of what I am attempting to extract from the log:

03/13/03 16:44:56 kernel Temporarily blocking host 212.241.116.21
03/13/03 16:44:57 firewalld[103] deny in eth0 48 tcp 20 117 212.241.11
+6.21 209.126.xxx.xxx 4449 80 syn (LO-Proxied-HTTP)
[download]

This must mean that you are able somehow to read the logs: let's assume that you did so by opening the wgl-log file in some sort of editor (and not in a proprietary viewer).

There are strong chances then that the log is in (a variant of) ASCII and Perl will be able to read the log by opening a read-filehandle and inputting the log line-by-line through us of the <INPUT-FILEHANDLE> function.

If all log-lines start with a date-and-timestamp, you can extract all data following these and put it through some regular expressions to weed out the useless entries and keep the valuable ones, which you can then either output to another file, save in a database, calculate some statistics from or --in general-- mangle beyond all recognition to your (and Perl's) heart's content.

To parse the logfile, you might have a look at regexp-log, HTTPD-Log-Filter or Log-Detect. Even if you can't use these modules directly, they will certainly give you some good ideas on how to tackle your task!

CountZero

"If you have four groups working on a compiler, you'll get a 4-pass compiler." - Conway's Law

Yes, that is the correct. I have sent the logs over to my Linux server via syslogd. The files are now viewable with any text editor.

Also, I am planning on searching through the file, then outputting the results to another file and save in a database.(My list just continues to grow.)

I have to say, this task seems very daunting to me. If I could trouble someone, could you get me started a little? Maybe give me a few hints or examples.

I think once I have a few things under my belt, I will feel more more confident and where to proceed from that point.

Thanks.

tarballed

For starters, you can have a look at Basic Input and Output and File Input and Output or Re: How do I read the contents of a file?.

For regular expressions String matching and Regular Expressions and the items linked to it, are surely a must.

Of course, you have already read Common Beginner Mistakes and will heed its warnings.

CountZero

"If you have four groups working on a compiler, you'll get a 4-pass compiler." - Conway's Law

Just to link the 2 nodes together, the script resulting from the efforts of this node is posted at node id 249479, My first Perl script.

If the above content is missing any vital points or you feel that any of the information is misleading, incorrect or irrelevant, please feel free to downvote the post. At the same time, please reply to this node or /msg me to inform me as to what is wrong with the post, so that I may update the node to the best of my ability.

It may be possible, but the file type 'data' won't tell you that, data just means 'unidentified binary content', so you need to find out what the format of that content is to determine if/how you can read it.

I see. So basically, I need to identify the format of the file and once I have that, I can move forward?
Thanks. Tarballed

Now I need to work on the script. (And what a script it will be, being it will be my first true script.)

In answer to your questions:

Yes, I would like to extract the data like this one:
03/13/03 16:44:56 kernel Temporarily blocking host 212.241.116.21

03/13/03 16:44:57 firewalld103 deny in eth0 48 tcp 20 117 212.241.11 +6.21 209.126.xxx.xxx 4449 80 syn (LO-Proxied-HTTP)

Right now, deny, allow and log are current acceptions from the log. For interfaces, we have eth0 and eth1. Most packets will be coming in through eth0.(trying to at least)

Other parts of the log are basically bits and pieces of information about the packet; size, source and destination port. What I want to pull out is basically, the IP address that was attemtping access (In this case, 117.212.241.11) I get a lot of these during the day.

I was thinking that I could do a search some how looking for "Temporarily blocking Host." Then, grab that information and the IP as well. Grabbing the destination port would be nice as well. That is the very last number before the word syn.

That help?

Tarballed

To get the destination port out of a line which also contains "firewall" and "deny", you could use the following regular expression:

m/firewall.+deny.+ (\d+) syn/
[download]

Translated it means: match "firewall" followed by some characters, followed by "deny" followed by some more characters, followed by a space, some digits, another space and "syn"; also save the digits you found in the special variable $1.

CountZero

"If you have four groups working on a compiler, you'll get a 4-pass compiler." - Conway's Law

Here is my little script so far:

#!/usr/bin/perl
#This script will be run against our Firewall logs. It will extract the information we want and send it to a new log.
#Open the firewall script to allow Perl to read it and gather data
open(FWLOG,"firewall.txt");
while(<FWLOG>)
#Close the file
close FWLOG;
It is not much, but I am trying to grasp the basic and understand it all.

Tarballed