Trying to step through a web page with WWW::Mechanize one line at a time

jclh has asked for the wisdom of the Perl Monks concerning the following question:

Hello,

I need to step through a web page one line at a time, and once I match a string, grab the next N lines...

My first thought was to load the content of the page into an array and each line would be an element:


$browser->get( $url );
my @array = $browser->content;
print scalar @array;
[download]

The problem is, that loads the whole page into a single array element not allowing me to do this:


while($Element=shift(@array)) {
if($Element =~ m/SomeonesName/) { print "$Element\n"; }
}
[download]

So, my question is: what is the best way to step through the page one line at a time?

Thank you!

Comment on Trying to step through a web page with WWW::Mechanize one line at a time Select or Download Code

Replies are listed 'Best First'.
Re: Trying to step through a web page with WWW::Mechanize one line at a time by Corion (Patriarch) on Jun 04, 2013 at 07:06 UTC
I recommend using a parser that returns the page as a tree, like HTML::TreeBuilder. For data extraction, I like the approach of using CSS selectors or XPath queries to specify the elements to extract. See HTML::TreeBuilder::XPath and HTML::Selector::XPath. App::scrape provides a blueprint of how to use the two to extract data from a website.	[reply]
Re: Trying to step through a web page with WWW::Mechanize one line at a time by hdb (Monsignor) on Jun 04, 2013 at 08:08 UTC
The question is: do you want to go through one line at a time as displayed in a browser or one line at a time throught the source code? The former is not a well defined question as the term "line" depends among other things how wide the browser window is. For the latter, `split` at line breaks: `my @array = split /\n/, $browser->content;` [download]	[reply] [d/l] [select]
Re^2: Trying to step through a web page with WWW::Mechanize one line at a time by jclh (Initiate) on Jun 05, 2013 at 01:32 UTC
Both helped me out... thanks! Yes, I want to step through the raw html one line at a time... I'm going to play around with the Tree code though too.	[reply]