perladdict has asked for the wisdom of the Perl Monks concerning the following question:

Hi monks,
I am trying to extract the web page report log, which will genarate when i clicked any of the link in a web page.
The request and response will be in xml format. There are no of clicks from different ipaddress, i want to see
the request log which genarated from a specific ipaddress. At present i am using perl one liner command which i got from the perl monks.
perl -ane 'print if /<request>/../</request>/' | tail -f logfile
By this i am able to see the xml log from the <request> to </request> tag.It is very difficult for
searching the request log from a specific ip address.The log format is as below
<?xml version="1.0" encoding="UTF-8"?><status>SUCCESS</status>
/***********For QA only ****************/
<?xml version="1.0" encoding="UTF-8"?>
<request>
<visitorID>a4379158-2fb5-494b-a195-ee2cc44df2a2</visitorID>
<reportSuiteID>aolmobilewebdev</reportSuiteID>
<pageName>wpt: help_pv</pageName>
<pageURL>http://mqa.aol.com/portal/default/help.do?icid=ft_help</pageU
+RL>
<ipAddress>64.12.218.80</ipAddress>
<userAgent>ApacheBench/2.0.41-dev</userAgent>
<scXmlVer>1.0</scXmlVer>
<timezone>-5</timezone>
<channel>us.wptportal</channel>
<referrer></referrer>
<events>event10</events>
<server>qawire-ql26.tweb.aol.com</server>
<eVar16>ft_help</eVar16>
<prop1>wpt: portal</prop1>
<prop2>wpt: Help Page</prop2>
<prop3>gmt_5</prop3>
<prop10>ApacheBench/2.0.41-dev</prop10>
<prop12>http://mqa.aol.com/portal/default/help.do</prop12>
<prop13>non-authenticated</prop13>
<prop14>no referrer</prop14>
<prop24>uaid_na</prop24>
<prop49>xml api</prop49>
</request>

<?xml version="1.0" encoding="UTF-8"?><status>SUCCESS</status>
/***********For QA only ****************/
<?xml version="1.0" encoding="UTF-8"?>
<request>
<visitorID>c8b26ed4-3370-4402-abd7-b0519c51d80c</visitorID>
<reportSuiteID>aolmobilewebdev</reportSuiteID>
<pageName>wpt: help_pv</pageName>
<pageURL>http://mqa.aol.com/portal/default/help.do?icid=ft_help</pageU
+RL>
<ipAddress>10.146.163.43</ipAddress>
<userAgent>ApacheBench/2.0.40-dev</userAgent>
<scXmlVer>1.0</scXmlVer>
<timezone>-5</timezone>
<channel>us.wptportal</channel>
<referrer></referrer>
<events>event10</events>
<server>qawire-ql26.tweb.aol.com</server>
<eVar16>ft_help</eVar16>
<prop1>wpt: portal</prop1>
<prop2>wpt: Help Page</prop2>
<prop3>gmt_5</prop3>
<prop10>ApacheBench/2.0.40-dev</prop10>
<prop12>http://mqa.aol.com/portal/default/help.do</prop12>
<prop13>non-authenticated</prop13>
<prop14>no referrer</prop14>
<prop24>uaid_na</prop24>
<prop49>xml api</prop49>
</request>

<?xml version="1.0" encoding="UTF-8"?><status>SUCCESS</status>
/***********For QA only ****************/
<?xml version="1.0" encoding="UTF-8"?>
<request>
<visitorID>420ea1fe-c93e-4209-bee9-e548de8ea1c5</visitorID>
<reportSuiteID>aolmobilewebdev</reportSuiteID>
<pageName>wpt: help_pv</pageName>
<pageURL>http://mqa.aol.com/portal/default/help.do?icid=ft_help</pageU
+RL>
<ipAddress>64.12.218.64</ipAddress>
<userAgent>ApacheBench/2.0.41-dev</userAgent>
<scXmlVer>1.0</scXmlVer>
<timezone>-5</timezone>
<channel>us.wptportal</channel>
<referrer></referrer>
<events>event10</events>
<server>qawire-ql26.tweb.aol.com</server>
<eVar16>ft_help</eVar16>
<prop1>wpt: portal</prop1>
<prop2>wpt: Help Page</prop2>
<prop3>gmt_5</prop3>
<prop10>ApacheBench/2.0.41-dev</prop10>
<prop12>http://mqa.aol.com/portal/default/help.do</prop12>
<prop13>non-authenticated</prop13>
<prop14>no referrer</prop14>
<prop24>uaid_na</prop24>
<prop49>xml api</prop49>
</request>

[download]
I want to see the log </request> to </request> which is from the ipaddress:10.146.163.43

Appreciate your assistance!
  • Comment on How i can extract the part of server report log based on Ipaddress
  • Download Code

Replies are listed 'Best First'.
Re: How i can extract the part of server report log based on Ipaddress
by Anonymous Monk on Aug 17, 2010 at 12:29 UTC
    I don't see why you're using xml to begin with, but why aren't you using an xml parser? XML::Twig or XML::LibXML, or even XML::Simple (i'd probably use XML::Simple to dump the reports into a sqlite database ...)
[reply]
      To be able to use an XML parser, the OP would have to make the logfile a valid XML (i.e. remove repeated <?xml... lines and add a root element). Then, the desired request could be found by this simple XPath: 
      /root/request[ipAddress="10.146.163.43"]

      [download]
[reply]
[d/l]
[select]
Re: How i can extract the part of server report log based on Ipaddress
by dasgar (Priest) on Aug 17, 2010 at 14:54 UTC

    My first thought would be to use an XML parser (as previously suggested), but choroba has good point about XML format. Although it's not a one liner, I believe that the untested code below should do the trick.

    use strict;

my $ip = '10.146.163.43';
my $logfile = "logfile.log";
my $log;

open(LOG,"<",$logfile) || die "Unable to open file '$logfile':  $!\n";
{
  local $/;
  $log = <LOG>; # slurping file into $log
}
close(LOG);

$log =~ m/(<request>.*?$ip.*?\/request>)/is;
print "Log for IP address $ip:\n\n$1\n";

    [download]

    If you're on a *nix OS, you'll probably need to add the appropriate #! line at the top. I do most of my Perl stuff on Windows and haven't gotten in the habit of adding that line to my code.

[reply]
[d/l]
Re: How i can extract the part of server report log based on Ipaddress
by suhailck (Friar) on Aug 17, 2010 at 14:36 UTC
    A sed solution,

    sed -n '/<request>/,/<\/request>/{/<request>/!{H;/<ipAddress>10.146.163.43<\/ipAddress>/{x;p;:a;n;p;/<\/request>/!ba}};/<request>/{h} }' <filename>
[reply]
[d/l]
      Hi suhailck,
      THanks a lot,i tried my task by your sed code, it is working perfectly.Can you please explain in detail about this command, i gone through the sed online tutorial, still i unable to understand some of the thigs in this code
      in your sed code. If you provide some cool sed tutorial stuff link, it will be a greate help for me in future.
[reply]

Back to Seekers of Perl Wisdom