in reply to Perl Regex

Can I suggest that you use the perl debugger to help you write your regular expressions, as that way you can very quickly try out different approaches until you have found a regular expression that works.

Unlike graphical debuggers you may be used to from other programming environments, the perl debugger is a command line tool. You may find it unfamiliar, but it has the advantage that you get a perl shell that allows you try out things as much as you like without stepping of the current line.

You can either start your script in the debugger, and then stopping it at the point where the string you are trying to match is loaded into a variable, or you start the debugger without a script using the one liner:

perl -d -e 0

(This works because the -e argument says to run the script on the command line, and the zero is the shortest possible perl script.)

Once you have the debug prompt, you can put your string into a varable, and try out regular exprssons on it. eg:

perl -d -e 0

Loading DB routines from perl5db.pl version 1.32
Editor support available.

Enter h or `h h' for help, or `man perldebug' for more help.

main::(-e:1):    0
  DB<1> $message = q(some_fields msg="http_decoder: HTTP.Unknown.Tunne
+lling" some_fields)

  DB<2> x $message
0  'some_fields msg="http_decoder: HTTP.Unknown.Tunnelling" some_field
+s'
  DB<3> x $message =~ m/msg=\"+((?:([^:,]+):\s|)([^,]+?)\s*(?:\s*,.*?|
+)))\"+/
Unmatched ) in regex; marked by <-- HERE in m/msg=\"+((?:([^:,]+):\s|)
+([^,]+?)\s*(?:\s*,.*?|))) <-- HERE \"+/ at (eval 7)[/usr/share/perl/5
+.10/perl5db.pl:638] line 2.
 at (eval 7)[/usr/share/perl/5.10/perl5db.pl:638] line 2
    eval '($@, $!, $^E, $,, $/, $\\, $^W) = @saved;package main; $^D =
+ $^D | $DB::db_stop;
  $message =~ m/msg=\\"+((?:([^:,]+):\\s|)([^,]+?)\\s*(?:\\s*,.*?|)))\
+\"+/;

;' called at /usr/share/perl/5.10/perl5db.pl line 638
    DB::eval called at /usr/share/perl/5.10/perl5db.pl line 3444
    DB::DB called at -e line 1
  DB<4>

[download]

When I tried your regular expression it did not work and perl gave me the error message above. (It looks like your brackets don't match up).

At this point, if it where me, I would start by writing a simpler regular expression, that only captures one element that I am interested in. Once I have the first part working, I would build it up until I had a regular expression that matched everything I needed. The debugger is well suited to this sort of trial and error. When it is all working, I then just copy and paste the final regular expression into my script.

Replies are listed 'Best First'.
Re^2: Perl Regex
by cipher (Acolyte) on Nov 23, 2010 at 10:36 UTC
    Thanks for the Help. I already tried the given regex and it's not working. Here is my Perl script. 
    #!/usr/bin/perl

chdir("/tmp") or die "$!";
opendir (DIR, ".") or die "$!";
my @files = grep {/2010*/}  readdir DIR;
close DIR;
{   local @ARGV = @files;


foreach my $file (@files) 
{
   open(FILE,"/tmp/$file") or die "No file!";
        
###############################

while ($fields = <FILE>)
    {

    @logs = split (/ /,$fields); # split log fields by space.

            foreach $msg (@logs)

            { 
        
            #if ( $msg=~(/^msg="(.*?)"/))
            #if ( $msg=~(/msg=/))
            if ( $msg=~/\bmsg="[^"]*"/)
            #if ($message =~ /msg=\"+((?:([^:,]+):\s|)([^,]+?)\s*(?:\s
+*,.*?|))\"+/)
                {
                print "$msg\n";
                }
            }
   
        }
   close(FILE);

    }

}
close FILE;
exit(0);

    [download]

    And here is the log file format:
    20 Nov 17:43:1 10 28 2010 02:18:33: date=2010-10-28 time=00:27:54 log_id=2 type=ips subtype=signature pri=alert fwver=040002 severity=medium carrier_ep="N/A" profile="IPS" src=X.X.X.X dst=X.X.X.X src_int="wan1" dst_int="internal" policyid=2 status=detected proto=17 service=1434/udp vd="root" count=1 src_port=111 dst_port=80 attack_id=10328 sensor="IPS_sensor" ref="http://www.fortinet.com/ids/VID10328" user="N/A" group="N/A" incident_serialno=2004954881 msg="database: MS.SQL.Server.Resolution.Service.Stack.Overflow"

    Am I missing anything in my code, I tried debugging but could not do anything inside debugger.
[reply]
[d/l]

      Please define "not working". What do you expect to happen? What happens instead? Any warning or error messages?

      And here is the log file format:

      No, that's just some crap copied and pasted from elsewhere. Inside <code></code>, it may have been a little bit helpful, as an example, but not in this form. All of the possible whitespace characters have collapsed to a single space, due to the way HTML works. And it looks like you have editied the example, making things even worse. A proper specification of the log format would really be useful. If you don't have that, post several unmodified(!) log lines inside <code></code>. If the log comes from a well-known piece of software, tell us the name and version of that software.

      Alexander

      --
      Today I will gladly share my knowledge and experience, for there are no sweeter words than "I told you so". ;-)
[reply]
        It's not working means I do not get any output when I print $msg, I expect it to print the actual string from log, I have mentioned this in my first post. Here is the log file inside code: 
        20 Nov 17:43:1 10 28 2010 02:18:33: date=2010-10-28 time=00:27:54 log_
+id=2 type=ips subtype=signature pri=alert fwver=040002 severity=mediu
+m carrier_ep="N/A" profile="IPS" src=X.X.X.X dst=X.X.X.X src_int="wan
+1" dst_int="internal" policyid=2 status=detected proto=17 service=143
+4/udp vd="root" count=1 src_port=111 dst_port=80 attack_id=10328 sens
+or="IPS_sensor" ref="http://www.fortinet.com/ids/VID10328" user="N/A"
+ group="N/A" incident_serialno=2004954881 msg="database: MS.SQL.Serve
+r.Resolution.Service.Stack.Overflow"

        [download]

        All other fields are matching and working fine except this one.
        I tried debugging and regex matches the string but not when I run my perl script. 
          DB<10> $msg = q(some_fields msg="http_decoder: HTTP.Unknown.Tunnelli
+ng" some_fields)

  DB<11> x $ msg
0  'some_fields msg="http_decoder: HTTP.Unknown.Tunnelling" some_field
+s'
  DB<12> x $msg =~/msg=\"(.*?)\"/         
0  'http_decoder: HTTP.Unknown.Tunnelling'

  DB<13> x $msg =~ /msg=\"+((?:([^:,]+):\s|)([^,]+?)\s*(?:\s*,.*?|))\"
++/
0  'http_decoder: HTTP.Unknown.Tunnelling'
1  'http_decoder'
2  'HTTP.Unknown.Tunnelling'

        [download]
[reply]
[d/l]
[select]

In Section Seekers of Perl Wisdom