in reply to Re: Re: Re: Requesting webpages which use cookies and session ids. (rev)
in thread Requesting webpages which use cookies and session ids. (rev)

  1. If the server says you need either MSIE or Netscape ver x.x you have to fake you use it: $ua->agent('Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0)');
  2. You do not have to fiddle with cookies that much. After you give the $ua the cookie jar you do not have to worry.
  3. The parameters should tno be POSTed to the script, they should be passed in the query string: 
    use CGI::Enurl;
my $request = GET $url_search . '?' . enurl {
        QRY => 'res',
        BV_SessionID => $sessID,
        BV_EngineID => $engID,
        new_search => 'true',
        NAM => 'Griffin',
        GIV => '',
        LOC => '',
        STR => '',
        PCD => 'BT',
        limit => '50',
        CallingPage => 'Homepage',
    };

    [download]
  4. The site sends a malformated cookie that is not remembered by LWP::UserAgent. Therefore you have to send it with the last request explicitely:
    $request = GET $url_search . '?' . enurl {
        QRY => 'res',
        NAM => 'Griffin',
        lci => '0',
        PCD => 'BT',
        start_id => '50',
        CallingPage => 'Homepage',
    },
    'Set-Cookie' => "'BV_IDS=$engID:$sessID; path=;";

    [download]
  5. The URL for the searching is not ".../dq_locationfinder.jsp". It's the ".../dq_home.jsp" itself. ".../dq_locationfinder.jsp" only serves for finding the location on a graphical map.

    6. After all these changes the script worked fine for me. You may find it at http://jenda.krynicky.cz/perl/BT_search.pl.txt.

    HTH, Jenda

In Section Seekers of Perl Wisdom