Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

I am using WWW::Mechanize in order to get some information from a web application. My problem is that between request page and result page there are 3 intermediate pages that just indicate that the server accepted the request, is working on it and has termined. Then it comes the result page. The intermediate pages dont have any forms or links on them. $mech gets the first page it can see after submit so it stores the page that indicates, that the server accepted the request.
How can I make $mech to store the last and interesting page?

Edited by Arunbear: Changed title from 'WWW::Mechanize', as per Monastery guidelines

  • Comment on Navigating with WWW::Mechanize across intermediate pages

Replies are listed 'Best First'.
Re: Navigating with WWW::Mechanize across intermediate pages
by monarch (Priest) on Jun 22, 2005 at 07:31 UTC
    It is possible that the server you are visiting is sending "redirect" headers.

    Check out the $mech->redirect_ok() option, which instructs your virtual browser (WWW::Mechanize) to follow redirects.

    update 1: a simpler approach may be to parse the header yourself looking for a URI to follow. Try the following on one of your magically redirecting pages:

    use WWW::Mechanize; my $mech = WWW::Mechanize->new( quiet => 1 ); $mech->get( "http://www.mysite.com/index.html" ); die( "Error getting page" ) if ( ! $mech->success() ); print( $mech->response->headers_as_string );

    or alternatively if you want to see the entire message, headers included, for debugging purposes, replace ->headers_as_string with ->as_string..

      Mech already follows valid redirects automatically. These are probably JavaScript or Meta-Refresh tags.
Re: Navigating with WWW::Mechanize across intermediate pages
by tlm (Prior) on Jun 22, 2005 at 09:40 UTC

    Can you tell us the URL of the page you're trying to access via WWW::Mechanize?

    If you're on Windows, you may have better luck with Win32::IE::Mechanize. See this recent thread for some discussion of this module.

    the lowliest monk

      Thank you for your answers. Here comes the Link:
      http://igs-server.cnrs-mrs.fr/Tcoffee/tcoffee_cgi/index.cgi?action=Make%20a%20multiple%20alignment::Advanced&stage1=1
        You will need a dummy file:

        >mel
        gtaagttttgagcatttaaatggaagtaaaaggcctactagatgggta--------atttgtgttt-acaaagtg---ccagctagggttaccat-gtcgactttgtgttttttccgcatccact-tagcatcagatgatagcattttt-gcaaatttcaagttttacggcgactgtacgagcgcgttccggtccacagcgcactttttccatacgaccgcagtgcgt---ttgtatgcgtttgtgtgtgtatgtacctgatgtctatcttcctcttctgtgtgtttttg---ttgctttttcgttcggggcgaaagtgctactactg--cggtcgttcggaaaaagtgctgctctcaggca--gacgcctcgccgacagtggccttaaaatttgtattcc--acatagcgtggcaaataattactttttcaacg-tgctcataattgcgatttttgatccatag
        >sim
        gtatgttttgatcacttaaatt-aagtaaaagacctactagatggcta--------atttgtgatt-acaaagtg---ccagctaggcttaccac-gccgactttgtgttttttccgcatccactctagcatcagatgatagcattttttgcaaatttcaagttttacggcgactgtacgagcgcgttccggtccacagcgcactttt-ccatacgaccgcagtgcgc---gtgcatgcgt--gtgtgtgtatgtacctgatatctctcttcctcttctatgtgattctg---ttgctttttcgttcggggcgaaagtgcgaccactg--cggtcgttc-ggaaaagtgctgctctcaaaca--gacgcctcgccgacagtggccgtaaaatttgtattcc--gcatagcgtggcaaataattactttttcaagg-tgctcataattgtgatttttgatccctag
        >yak
        gtaagttttgtgctcttatgtt-aagtaaaataccttctagccgggcg--------atttgtgatt-acaaagtg---tcagctatgactaccac-gccgactttgtgtttttttcgcatccgct-tagcgtcagatgatagcattttt-gcaaatttcaagttttacggcgactgtacgagcgcgttccggttaacagcgcacttttcccatacgaccgcagtgcgc---ttgtatgcgt--gtgcttgtatgtacctgatgtctctcttcctcttctttgtgttgttg---ctgccttttcgttcggggcgaaagtgcgaccgctg--cggtcgttcgggaaaagtgatgctctccagca--gacgcctcgccgactgtggccgtaaaatgtgtatttc--gcatagcgtggcatataattacaattccaagg-tactcataatcgcgatttttgatccccag
        >pse
        gtaagtttt-tgca--taaaat-aactaaaaa---ttgtacgcttt-acagcagatatttatgatttac----cggcccccgcagtgtttttcggagccggaggcaaatttttttcgggtccatt-cggcggcggatgatagca-tttt-gcgaatttctagttatacggcgactgtacgagcgcgtttcg--tcagagctcacttttcagtgacaaccgcagagcggattttggtggtgt--gtgtgtat-----------gcttaagtgtg-cgcgtatgtacctgtatgtctgc-ttt--gttc--ggcgaaagtgt-act-ctgaagggttgttgtcgaaaagtgc-gctctgc-gcatggacgcttcagcgaccgtggcggtaaaattt--attttctgcatagcgtggcaattgctcattttttcgccaat-ctaacccttgatatttctgctctccag

        That are the options I have chosen:

        $mech->form_number(1);
        $mech->set_fields(email => "", -uploaded_file => $upload, -case => "upper", -seqnos => "off", -outorder => "input",);
        $mech->tick(-in => "Mlalign_id_pair");
        $mech->tick(-in => "Mfast_pair");
        $mech->tick(-in => "Mclustalw_aln");
        $mech->tick(-output => "fasta_aln");
        $mech->tick(-output => "score_html");
        $mech->submit();