in reply to Re: Re: Re: Getting result page using WWW::Mechanize
in thread Getting result page using WWW::Mechanize

Hi Corion, Thank you for helping me to write the basic steps to implement the module. Still i didn't get any idea if i go for second level of form (First level is to get the inputs and submit and it will display a page. The second level is, in the display page if i have a text field say "age" and two buttons "continue" and "stop". I would like to input in the text field and click on the continue button . How can i add these steps in my previous code? It will be very useful for me, if you post that part. Thanks a lot.
  • Comment on Re: Re: Re: Re: Getting result page using WWW::Mechanize

Replies are listed 'Best First'.
Re: Re: Re: Re: Re: Getting result page using WWW::Mechanize
by Corion (Patriarch) on Jul 04, 2003 at 11:11 UTC

    As long as you don't post the URL of the site you are trying to automate, I can only poke around in the dark - an exercise not very rewarding. I suggest that you install my module WWW::Mechanize::Shell, which allows you to explore a website interactively and afterwards spit out a complete Perl script that recreates what you did manually.

    My guess is, that the following commands in WWW::Mechanize::Shell should be sufficient in getting you to your goal :

    get http://www.example.com/ value login "username" value password "secret" click "Login" # You get a page back which is a frame page links # look at the links open "/data_frame/" value age 99 click "continue" content

    If you absolutely do not get WWW::Mechanize::Shell to install, I will try to translate that script into Perl manually, but please try ::Shell first, as it can display the current page in your browser for easy review.

    perl -MHTTP::Daemon -MHTTP::Response -MLWP::Simple -e ' ; # The $d = new HTTP::Daemon and fork and getprint $d->url and exit;#spider ($c = $d->accept())->get_request(); $c->send_response( new #in the HTTP::Response(200,$_,$_,qq(Just another Perl hacker\n))); ' # web

      Hi Corion,

      Once again thanks for your explanation.

      I had modified my previous code like this to get the second level of the page... It's working fine.

      $first = $agent->submit(); $agent->get($first); print $agent->content;

      Before i did was a test program to use this module.

      I would like to connect a mail service to get a page from there.. this is i am trying now.

      URL of the site you are trying to automate is http://www.attachmail.com/

      • UserId is test01
      • Passwd is s
      • Click on Go image (name=imageField). It will display a useraccount page.
      • Click on the Addressbook button from the page. It will display Address book page contains 4 buttons (Add,Delete,Edit,close).
      • Click on Add button, it will display a page contains two text fields name and email address , Add and Close button.
      • Click on the Add button will display a page. This page i required.

      How can i do that?

      Hope you can help me to solve this problem. Thanks a lot and waiting for your great help.

      Regards,
      ps

      20030705 Edit by Corion: Added formatting

        First of all, please take a look at the Writeup Formatting Tips. They tell you how to make your node look appealing and not look like an unstructured mess.

        Second, I told you to try WWW::Mechanize::Shell, a tool to generate scripts for WWW::Mechanize. Did you try it? If so, where were the problems you encountered?

        If you did try it, but did not get far, then my solution below will not be of much help, since you will have to learn about JavaScript and HTML before you can automate this website.

        Using WWW::Mechanize::Shell, I went to that website and navigated through the pages, emulating the JavaScript by hand. Here is the full transcript of that session, with comments :

        # Fake IE ua "Mozilla/4.0 (compatible; MSIE 4.01; Windows 98)" get http://www.attachmail.com/ # Navigate to login frame open bott # Login value UserId test01 value Passwd s click imageField # Got empty page. Why? content # Aaah - it is a META refresh page # Extract the target url : eval $self->agent->content =~ /URL=(.*)'>/; qq{http://www.attachmail.c +om/cg-bin/$1} # Jump to that page get http://www.attachmail.com/cg-bin/userpagedisplay.cgi?user_no=10573 +48162&domain=attachmail.com&dtexpiry=19-Jul-2003&status=0&spacebal=4. +998&bounce=&flmove= # Multi-frame page, go to the content frame open mailatt open /AddressBook/ open /AddressBook/ # Still no action - must be JavaScript. # Looking at the JavaScript, I know now that I need the # user number : eval $self->agent->uri eval $self->agent->uri=~/userno=(\d+)/; $1 # Go to the page referenced from the JavaScript get http://www.attachmail.com//cg-bin/am_bring_Addbook.cgi?userno=1057 +348162&flg=Disp # Find the correct frame : open mainFrame back open bottomFrame # More JavaScript interaction: get http://www.attachmail.com/cg-bin/add_entry.cgi?username=1057348162 # Fill in the values and submit the form value nam foo value email bar@example.com submit

        Then I looked at the code that was generated by my actions, and cleaned it up with the stuff I know about this website :

        #!D:\Programme\indigoperl-5.6\bin\perl.exe -w use strict; use WWW::Mechanize; use URI::URL; my $agent = WWW::Mechanize->new(); $agent->env_proxy(); $agent->get('http://www.attachmail.com/'); $agent->form(1) if $agent->forms and scalar @{$agent->forms}; $agent->agent('Mozilla/4.0 (compatible; MSIE 4.01; Windows 98)'); # navigate to the login frame $agent->follow('bott'); { local $^W; $agent->current_form->value('UserId', 'test01'); }; { local $^W; $agent->current_form->value('Passwd', 's'); }; $agent->click('imageField'); # The login page is a redirect die "Unknown page received" unless $agent->content =~ /URL=(.*)'>/; my $redirect = qq{http://www.attachmail.com/cg-bin/$1}; $agent->get($redirect); $agent->form(1) if $agent->forms and scalar @{$agent->forms}; $agent->follow('mailatt'); die "Couldn't retrieve user number" unless $agent->uri=~/userno=(\d+)/; my $userno = $1; $agent->get("http://www.attachmail.com//cg-bin/am_bring_Addbook.cgi? +userno=$userno&flg=Disp'); $agent->form(1) if $agent->forms and scalar @{$agent->forms}; $agent->follow('bottomFrame'); $agent->get("http://www.attachmail.com/cg-bin/add_entry.cgi?username +=$userno"); $agent->form(1) if $agent->forms and scalar @{$agent->forms}; { local $^W; $agent->current_form->value('nam', 'foo'); }; { local $^W; $agent->current_form->value('email', 'bar@example.com') +; }; $agent->submit();

        This is the complete code and should work as is, but it will not help you much unless you actually try to understand the website and how this code interacts with it.

        perl -MHTTP::Daemon -MHTTP::Response -MLWP::Simple -e ' ; # The $d = new HTTP::Daemon and fork and getprint $d->url and exit;#spider ($c = $d->accept())->get_request(); $c->send_response( new #in the HTTP::Response(200,$_,$_,qq(Just another Perl hacker\n))); ' # web