Help Parsing lwp browser request content

mmittiga17 has asked for the wisdom of the Perl Monks concerning the following question:

Hi All, I am working on a script to automate downloading files from a website. I am using LWP::UserAgent. I need to parse the output from the returned content to get the value of the TOKEN: <input name="org.somescript.html.TOKEN" type="hidden" value="f152a40cd9234c57542ee5b0c057cb0b" />


my $browser = LWP::UserAgent->new();
$browser->cookie_jar( {} );

#Post to some URL with some data
$url="https://somesite.com/LoginAction.do";
my $req0 = HTTP::Request->new(GET => $url);

my $resp0 = $browser->request($req0);
# Response can be read like this:
print $resp0->content;
[download]

the print $resp0->content; line returns the entire html code from the site. I have tried endlessly to parse the content for just the TOKEN Value. Can any one recommend a method? Thanks!

Comment on Help Parsing lwp browser request content Download Code

Replies are listed 'Best First'.
Re: Help Parsing lwp browser request content by wfsp (Abbot) on Mar 13, 2008 at 17:58 UTC
This uses HTML::TokeParser::Simple #!/usr/local/bin/perl use strict; use warnings; use HTML::TokeParser::Simple; # load your html into a string my $content = do{local $/;<DATA>}; my $p = HTML::TokeParser::Simple->new(\$content); while (my $t = $p->get_tag(q{input})){ my $name = $t->get_attr(q{name}); my ($token) = $name =~ /^org\.somescript\.html\.(.*)$/; print $token; # prints TOKEN } __DATA__ <!-- lots of html --> <input name="org.somescript.html.TOKEN" type="hidden" value="f152a40cd9234c57542ee5b0c057cb0b" /> <!-- lots more html --> [download] other html parsers are available	[reply] [d/l]
Re: Help Parsing lwp browser request content by Roy Johnson (Monsignor) on Mar 13, 2008 at 18:01 UTC
A quick and dirty way might be `my $whole_page = $resp0->content; my ($token) = $whole_page =~ /<input[^>]?value="(.?)"/;` [download] A more robust solution might be to use WWW::Mechanize and search through the input fields it parses. Or what wfsp said. :) Caution: Contents may have been coded under pressure.	[reply] [d/l]


Think about Loose Coupling
	PerlMonks

Help Parsing lwp browser request content

other html parsers are available