Re: Re: Re: Requesting webpages which use cookies and session ids. (rev)

just one of the two afore mentioned error messages subsequent Sorry Jenda, I missed your post....I commented out that conversion line and now I'm getting

To use this service you will need either an Internet Explorer (IE) browser or Netscape 4.7 and above.

for the first search page now aswell as for subsequent pages - at least theres some consistency there. :)

Comment on Re: Re: Re: Requesting webpages which use cookies and session ids. (rev)

Replies are listed 'Best First'.
Re: Re: Re: Re: Requesting webpages which use cookies and session ids. (rev) by Jenda (Abbot) on Aug 05, 2002 at 13:51 UTC
If the server says you need either MSIE or Netscape ver x.x you have to fake you use it: `$ua->agent('Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0)');` You do not have to fiddle with cookies that much. After you give the $ua the cookie jar you do not have to worry. The parameters should tno be POSTed to the script, they should be passed in the query string: `use CGI::Enurl; my $request = GET $url_search . '?' . enurl { QRY => 'res', BV_SessionID => $sessID, BV_EngineID => $engID, new_search => 'true', NAM => 'Griffin', GIV => '', LOC => '', STR => '', PCD => 'BT', limit => '50', CallingPage => 'Homepage', };` [download] The site sends a malformated cookie that is not remembered by LWP::UserAgent. Therefore you have to send it with the last request explicitely: `$request = GET $url_search . '?' . enurl { QRY => 'res', NAM => 'Griffin', lci => '0', PCD => 'BT', start_id => '50', CallingPage => 'Homepage', }, 'Set-Cookie' => "'BV_IDS=$engID:$sessID; path=;";` [download] The URL for the searching is not ".../dq_locationfinder.jsp". It's the ".../dq_home.jsp" itself. ".../dq_locationfinder.jsp" only serves for finding the location on a graphical map. After all these changes the script worked fine for me. You may find it at http://jenda.krynicky.cz/perl/BT_search.pl.txt. HTH, Jenda	[reply] [d/l] [select]

Replies are listed 'Best First'.

Re: Re: Re: Re: Requesting webpages which use cookies and session ids. (rev)
by Jenda (Abbot) on Aug 05, 2002 at 13:51 UTC

If the server says you need either MSIE or Netscape ver x.x you have to fake you use it: $ua->agent('Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0)');
You do not have to fiddle with cookies that much. After you give the $ua the cookie jar you do not have to worry.

The parameters should tno be POSTed to the script, they should be passed in the query string:

use CGI::Enurl;
my $request = GET $url_search . '?' . enurl {
        QRY => 'res',
        BV_SessionID => $sessID,
        BV_EngineID => $engID,
        new_search => 'true',
        NAM => 'Griffin',
        GIV => '',
        LOC => '',
        STR => '',
        PCD => 'BT',
        limit => '50',
        CallingPage => 'Homepage',
    };
[download]

The site sends a malformated cookie that is not remembered by LWP::UserAgent. Therefore you have to send it with the last request explicitely:

$request = GET $url_search . '?' . enurl {
        QRY => 'res',
        NAM => 'Griffin',
        lci => '0',
        PCD => 'BT',
        start_id => '50',
        CallingPage => 'Homepage',
    },
    'Set-Cookie' => "'BV_IDS=$engID:$sessID; path=;";
[download]

The URL for the searching is not ".../dq_locationfinder.jsp". It's the ".../dq_home.jsp" itself. ".../dq_locationfinder.jsp" only serves for finding the location on a graphical map.

After all these changes the script worked fine for me. You may find it at http://jenda.krynicky.cz/perl/BT_search.pl.txt.

HTH, Jenda

[reply]
[d/l]
[select]