donno20 has asked for the wisdom of the Perl Monks concerning the following question: 

use WWW::Mechanize;
my $mech = WWW::Mechanize->new();
$mech->quiet(1);
$mech->get("http://www.thesite.com/sample.asp");
my $content = $mech->content();
print $content;

[download]
the content only javacript but no html form inside. When I use my web browser to view the same address, I can get the html form with view source.
anybody experience this before? Thank you ^_^

Replies are listed 'Best First'.
Re: My Mechanize script cannot work ?
by Corion (Patriarch) on Aug 07, 2003 at 06:43 UTC

    WWW::Mechanize does not execute any JavaScript. You will have to look at the returned JavaScript and then look what the next URL to fetch will be, and fetch that URL then from the script :

    use strict;
use WWW::Mechanize;
my $mech = WWW::Mechanize->new();
$mech->quiet(1);
$mech->get("http://www.thesite.com/sample.asp");
my $content = $mech->content();
$mech->content =~ qr(\.location = "(http://.*?)")
  or die "Couldn't find anything matching '.location = \"http://...\" 
+in $content";
my $new_location = $1;
print $new_location;
$mech->get($new_location);
print $content;

    [download]

    If you want to see the "real" behaviour in your browser as well, you have to switch off JavaScript in your browser.

    Perl does not yet have a real JavaScript interpreter with the browser DOM, but there is JavaScript.pm and Scripting::JavaScript, both by Claes Jacobson.

    perl -MHTTP::Daemon -MHTTP::Response -MLWP::Simple -e ' ;    # The  
$d = new HTTP::Daemon and fork and getprint $d->url and exit;#spider
($c = $d->accept())->get_request(); $c->send_response( new   #in the
HTTP::Response(200,$_,$_,qq(Just another Perl hacker\n))); ' #  web

    [download]
[reply]
[d/l]
[select]
      Thank you. I manage to get to the right login page.
      However I get this return whil submit the form.
      Unexpected field value http://www.thesite.com/members/login.asp at (eval 5) line 1
      What is going on ? I dump the filled in form and return:

      uid=<UNDEF> (hidden)
      tuid=username
      password=pass123 (password)
      cb= (hidden)
      vcid=dummy (hidden)
      submit1=Login now (submit)

      Thanks again. ^_^
[reply]

        Without seeing your code, it is very hard to guess what happens there. I guess from your description that the script prints the message Unexpected field value http://www.thesite.com/members/login.asp at (eval 5) line 1 while running. I don't know what might cause this message to be printed, so I guess that it has something to do with how you are passing form values to the $mech object. 

        perl -MHTTP::Daemon -MHTTP::Response -MLWP::Simple -e ' ;    # The  
$d = new HTTP::Daemon and fork and getprint $d->url and exit;#spider
($c = $d->accept())->get_request(); $c->send_response( new   #in the
HTTP::Response(200,$_,$_,qq(Just another Perl hacker\n))); ' #  web

        [download]
[reply]
[d/l]
      The code is pretty simple as the following 
      use WWW::Mechanize;
my $mech = WWW::Mechanize->new();
$mech->get("http://www.thesite.com/members/login.asp");
my $content = $mech->content();

$mech->form_number(1);
$form = $mech->current_form();

$mech->field('username', 'username');
$mech->field('password', 'pass123');
print $form->dump();

$result = $mech->submit();

      [download]
      So any comments ?
[reply]
[d/l]

        try this...

        # ...
my $mech = WWW::Mechanize->new();
$mech->agent('Mozilla/4.0 (compatible; MSIE 4.01; Windows 98)');   # l
+ie through your teeth.
# ...

        [download]
[reply]
[d/l]

        As far as I can see, there is nothing wrong with the script, so the problem possibly lies with the HTML from that site. Save the HTML of the site into a file (with $mech->getstore("http://www.thesite.com/members.login.asp", "login.html") and then try to reproduce the problem with a page as small and simple as possible. To load a page ino WWW::Mechanize, set the content like this :

          open HTML, "<", "login.html"
    or die "Couldn't read 'login.html' : $!\n";
  my $html = do { local $/; <HTML> };
  $mech->{content} = $html;
  $mech->_reset_page();
  if ( $mech->is_html ) {
        $mech->{forms} = [ HTML::Form->parse($mech->{content}, "http:/
+/www.thesite.com/members/") ];
        $mech->{form}  = $self->{forms}->[0];
        $mech->_extract_links();
    };

        [download]

        This is all I can do for you at the moment and I will be away over the weekend.

        perl -MHTTP::Daemon -MHTTP::Response -MLWP::Simple -e ' ;    # The  
$d = new HTTP::Daemon and fork and getprint $d->url and exit;#spider
($c = $d->accept())->get_request(); $c->send_response( new   #in the
HTTP::Response(200,$_,$_,qq(Just another Perl hacker\n))); ' #  web

        [download]
[reply]
[d/l]
[select]
Re: My Mechanize script cannot work ?
by cfreak (Chaplain) on Aug 07, 2003 at 02:37 UTC

    Hmm ... I've never used WWW::Mechanize so I can't really comment directly on the exact problem, though I can offer a few suggestions. 1: My guess is that $mech->quiet(1); is going to suppress any errors that might get returned, while your debugging it would probably be a good idea to set it to 0 or comment out the line (assuming WWW::Mechanize returns errors by default) so you can see if you get an error. Secondly I know that your site is an example but if the page you're trying to get is an ASP page, be sure that there isn't some kind of IE specific things on it that would cause it not to return correctly for your agent. And finally I would make sure that $mech->content() returns a SCALAR value.

    Hope that helps

    Lobster Aliens Are attacking the world!
[reply]
[d/l]
[select]
Re: My Mechanize script cannot work ?
by WhiteBird (Hermit) on Aug 07, 2003 at 02:34 UTC
    When I try to browse to http://www.thesite.com it takes me to http://www.techtv.com/thesite/ and I get a 403 Forbidden error. Which is exactly what I get when I run your script through perl. Check your web address? HTH
[reply]
[d/l]
[select]
      The site given is a dummy example.
[reply]

Back to Seekers of Perl Wisdom