Strange way a string or array works

lampros21_7 has asked for the wisdom of the Perl Monks concerning the following question:

Hi there, i have been testing some code i found in this website and i 've realized it works a bit funny when i change some things on it and i dont understand why. The initial code is this:

use WWW::Mechanize;
use HTML::TokeParser;

my $webcrawler = WWW::Mechanize->new();
$webcrawler->get("http://www.google.com");
my $content = $webcrawler->content;
my $parser = HTML::TokeParser->new(\$content);
while($parser->get_tag){
    print $parser->get_trimmed_text(),"\n";
}
[download]

It gives me the output i want, stripping almost all the HTML from a web page and leaving the contents.Now, if i don't have the webpage specified here and i try to get it with the <STDIN> command when i try to compile and run the program with the perl command then the script gets stuck. It won't even let me type a URL in.

$url_name = <STDIN>; # The user inputs the URL to be searched
 $webcrawler->get($url_name);
[download]

The 3rd version is when the while statement starts i want to save the $parser variable instead of printing it. I tried to save it as an element of an array and a variable on its own.So at the it would change to:

while($parser->get_tag){
   my $stripped_html = $parser->get_trimmed_text(),"\n";
    #print $parser->get_trimmed_text(),"\n";
   print $stripped_html;
    }
[download]

But this gets stuck again for some reason and looking at the proccesses it takes a lot of my CPU power so it probably gets stuck somewhere although i dont know where.This is the same if i initialize an array and use the first element of that array to write the $parser in. Any ideas?Thanks in advance

Comment on Strange way a string or array works Select or Download Code

Replies are listed 'Best First'.
Re: Strange way a string or array works by Eimi Metamorphoumai (Deacon) on Aug 02, 2005 at 11:32 UTC
As far as your first question goes, what exactly do you mean by "It won't even let me type a URL in"? That is, I've never seen a program physically prevent you from typing, so the question is what does it do when you do try to type into it? I'd suggest trying to `chomp $url_name` before using it. For your second question, I think the problem you're facing is that `print` allows multiple arguments, and you're passing it two (the text to print, and then a newline). That's fine for print, but you can't assign both of those to a single scalar. If you change that to `my $stripped_html = $parser->get_trimmed_text()."\n"; print $stripped_html;` [download] it should do what you want.	[reply] [d/l] [select]
Re: Strange way a string or array works by CountZero (Bishop) on Aug 02, 2005 at 08:20 UTC
I changed your program as follows: `use strict; use WWW::Mechanize; use HTML::TokeParser; use Data::Dumper; my $webcrawler = WWW::Mechanize->new(); $webcrawler->get("http://www.marsh.com"); my $content = $webcrawler->content; my $parser = HTML::TokeParser->new(\$content); my @text_data; while($parser->get_tag){ push @text_data, $parser->get_trimmed_text(),"\n"; } print Dumper(\@text_data);` [download] And it works as advertized. BTW, You better not crawl/scrape Google as they don't like it very much: see "No automated querying" (external link). CountZero "If you have four groups working on a compiler, you'll get a 4-pass compiler." - Conway's Law	[reply] [d/l]
Re: Strange way a string or array works by dtr (Scribe) on Aug 02, 2005 at 11:30 UTC
Try using `Term::ReadLine` to prompt you for a URL. Or, use `GetOpt::Long` and read the url in from the command line. Either of these options will be better than reading from STDIN.	[reply] [d/l] [select]
Re: Strange way a string or array works by TedPride (Priest) on Aug 02, 2005 at 08:16 UTC
I don't know what would be causing the problem in your last section of code, but in the second section you aren't removing the newline from <STDIN> before feeding it to the crawler. This could be the problem.	[reply]
Re^2: Strange way a string or array works by Anonymous Monk on Aug 02, 2005 at 09:20 UTC
... in the second section you aren't removing the newline from <STDIN> before feeding it to the crawler. This could be the problem. That was my first guess as well, but his problem is It won't even let me type a URL in. Which means, there's nothing to feed the crawler - the program doesn't get this far. Unless the OP described his problem very poorly.	[reply]