I also tried (/^msg=\"(.*?)\"+) This also gives me msg="http_decoder:

No, it does not. Whatever that captures must end with a '"'. A few possible reasons you might think you got what you say you got:

• Perhaps you are suffering from buffering? That is to say you may be viewing the output before all of it has been output.
• Or maybe you are outputting control characters which are affecting your terminal, making it look like that's what's being output.
• Maybe you are confusing your test runs, and you never got that output from that code.

my ($msg) = /^msg="(.*?)"/ should do the trick, but since I dislike using the "?" quantifier, I'd use my ($msg) = /^msg="[^"]*"/.

Can I suggest that you use the perl debugger to help you write your regular expressions, as that way you can very quickly try out different approaches until you have found a regular expression that works.

Unlike graphical debuggers you may be used to from other programming environments, the perl debugger is a command line tool. You may find it unfamiliar, but it has the advantage that you get a perl shell that allows you try out things as much as you like without stepping of the current line.

You can either start your script in the debugger, and then stopping it at the point where the string you are trying to match is loaded into a variable, or you start the debugger without a script using the one liner:

(This works because the -e argument says to run the script on the command line, and the zero is the shortest possible perl script.)

Once you have the debug prompt, you can put your string into a varable, and try out regular exprssons on it. eg:

perl -d -e 0

Loading DB routines from perl5db.pl version 1.32
Editor support available.

Enter h or `h h' for help, or `man perldebug' for more help.

main::(-e:1):    0
  DB<1> $message = q(some_fields msg="http_decoder: HTTP.Unknown.Tunne
+lling" some_fields)

  DB<2> x $message
0  'some_fields msg="http_decoder: HTTP.Unknown.Tunnelling" some_field
+s'
  DB<3> x $message =~ m/msg=\"+((?:([^:,]+):\s|)([^,]+?)\s*(?:\s*,.*?|
+)))\"+/
Unmatched ) in regex; marked by <-- HERE in m/msg=\"+((?:([^:,]+):\s|)
+([^,]+?)\s*(?:\s*,.*?|))) <-- HERE \"+/ at (eval 7)[/usr/share/perl/5
+.10/perl5db.pl:638] line 2.
 at (eval 7)[/usr/share/perl/5.10/perl5db.pl:638] line 2
    eval '($@, $!, $^E, $,, $/, $\\, $^W) = @saved;package main; $^D =
+ $^D | $DB::db_stop;
  $message =~ m/msg=\\"+((?:([^:,]+):\\s|)([^,]+?)\\s*(?:\\s*,.*?|)))\
+\"+/;

;' called at /usr/share/perl/5.10/perl5db.pl line 638
    DB::eval called at /usr/share/perl/5.10/perl5db.pl line 3444
    DB::DB called at -e line 1
  DB<4>
[download]

When I tried your regular expression it did not work and perl gave me the error message above. (It looks like your brackets don't match up).

At this point, if it where me, I would start by writing a simpler regular expression, that only captures one element that I am interested in. Once I have the first part working, I would build it up until I had a regular expression that matched everything I needed. The debugger is well suited to this sort of trial and error. When it is all working, I then just copy and paste the final regular expression into my script.

Thanks for the Help. I already tried the given regex and it's not working. Here is my Perl script.

#!/usr/bin/perl

chdir("/tmp") or die "$!";
opendir (DIR, ".") or die "$!";
my @files = grep {/2010*/}  readdir DIR;
close DIR;
{   local @ARGV = @files;


foreach my $file (@files) 
{
   open(FILE,"/tmp/$file") or die "No file!";
        
###############################

while ($fields = <FILE>)
    {

    @logs = split (/ /,$fields); # split log fields by space.

            foreach $msg (@logs)

            { 
        
            #if ( $msg=~(/^msg="(.*?)"/))
            #if ( $msg=~(/msg=/))
            if ( $msg=~/\bmsg="[^"]*"/)
            #if ($message =~ /msg=\"+((?:([^:,]+):\s|)([^,]+?)\s*(?:\s
+*,.*?|))\"+/)
                {
                print "$msg\n";
                }
            }
   
        }
   close(FILE);

    }

}
close FILE;
exit(0);
[download]


And here is the log file format:
20 Nov 17:43:1 10 28 2010 02:18:33: date=2010-10-28 time=00:27:54 log_id=2 type=ips subtype=signature pri=alert fwver=040002 severity=medium carrier_ep="N/A" profile="IPS" src=X.X.X.X dst=X.X.X.X src_int="wan1" dst_int="internal" policyid=2 status=detected proto=17 service=1434/udp vd="root" count=1 src_port=111 dst_port=80 attack_id=10328 sensor="IPS_sensor" ref="http://www.fortinet.com/ids/VID10328" user="N/A" group="N/A" incident_serialno=2004954881 msg="database: MS.SQL.Server.Resolution.Service.Stack.Overflow"

Am I missing anything in my code, I tried debugging but could not do anything inside debugger.

Please define "not working". What do you expect to happen? What happens instead? Any warning or error messages?

And here is the log file format:

No, that's just some crap copied and pasted from elsewhere. Inside <code></code>, it may have been a little bit helpful, as an example, but not in this form. All of the possible whitespace characters have collapsed to a single space, due to the way HTML works. And it looks like you have editied the example, making things even worse. A proper specification of the log format would really be useful. If you don't have that, post several unmodified(!) log lines inside <code></code>. If the log comes from a well-known piece of software, tell us the name and version of that software.

Alexander

--
Today I will gladly share my knowledge and experience, for there are no sweeter words than "I told you so". ;-)

Regex given by Ikegami should do the trick, but in the example log you have mentioned some_fields before and after the required message. In that case you should remove the caret (^) to allow it matching anywhere in the string. E.g. /\bmsg="[^"]*"/

The example you posted has syntax errors. You have left the concatenation operator (.) while printing. I hope you are not redirecting your error output to /dev/null while running the script.

Thanks all for your help, I learned new things like basic perl debugger and the checking regex on notepad.
I will take a different approach for parsing different log fields.
Thanks again!