Dear all,

Im using WWW::Mechanize to scrape a online library of citations. It seems to be working well, and easily. However, this online library needs a session id, persistent throughout its search, which is present in a hidden input field.

Here's my code so far:

#!/usr/bin/perl -w 

require strict;

use WWW::Mechanize;

my $mech = WWW::Mechanize->new( autocheck => 1 );
$mech->get("http://isiknowledge.com/?DestApp=WOS&Func=Frame");
$mech->follow_link(tag => 'frame', n => 2);
$mech->form_number(1);
$mech->click('General Search', [2,2]);
$mech->form_number(1);
$mech->field("topic","emergence AND complexity");
$mech->click();
$mech->form_number(1);
$mech->field('mark_from',1);
$mech->field('mark_to',500);
$mech->field('mark_selection','range');
my $res=$mech->click('Submit');
print $res->content;
[download]

Its pretty straight forward, and for the first couple of pages, it goes well, but on the last 'Submit', the response page claims I need to start a new session!! I've encountered this before, and I actually did try extracting the session id from previous pages, and inserting it into the hidden field in the last page, but WWW::Mechanize wont let me, as it says that field is 'read-only' I assume because it is hidden.

The funny thing is, I stopped using HTTP::Request::Form, because there was a bug concerning multiple checkboxes of the same name, however doing this same thing with HRF, I could get through to the next page, so WWW::Mechanize seems to be dropping the session id, what's going on?

Is it possible that there's a redirect that it's following automatically WITHOUT the session id?

Thanks
Seaver