Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

When submitting a particular form with netscape, I get a large number of results, but when submitting what I *think* is the same form with LWP, I get no results (the request succeeds, but the web server says "You must specify some information to search on."). The form is the "Search for Jobs" thing at the bottom of http://www.BrilliantPeople.com/jobsearch.asp I can with netscape select Information Technology, and leave the keyword field blank (I have javascript disabled, so it doesn't complain about this), and press the "Go ->" submit button. This works. But when I try to recreate the request with LWP: 
   sub searchreq($) {
      my $page = 0 + shift;
      return HTTP::Request->new(POST =>
         "http://www.BrilliantPeople.com/jobsearch.asp",
         undef,
         # Search for Information Technologies stuff.
         "hdnOccupationSelectionCodes=".join(",",5118..5137)
         ."&txtKeywords="
         #."&numberperpage=100".
         #."&currentpage=" . $page
     );
   }

[download]
I get: "You must specify some information to search on." Any suggestions? The more I pound my head on this problem, the less like a BrilliantPerson I feel.

Replies are listed 'Best First'.
Re: Mimicing browser with LWP
by Abigail-II (Bishop) on Jun 04, 2002 at 14:56 UTC
    If I leave the keyword fields blank, also with Netscape, I get the 'You must specify some information to search on' error. Did you try using LWP and filling out some keywords?

    Also, years ago, when I could still be bothered doing webbish stuff, I used a simple debugging proxy, just a handful of lines with with LWP and some networking code, that would dump all traffic to STDERR.

    Very useful if you want to know what exactly your browser or client is sending and receiving.

    Abigail

[reply]
      When I ran it in netscape, I disabled javascript so that I could leave the keywords field blank.

      Anyway, adding keywords with LWP doesn't help any:

      When I use category Information Technology, and keyword programmer, netscape gives 75 matches, but with LWP it still says, "You must specify some information to search on."


      I guess your idea of writing a debugging proxy is probably what I'll need to figure this out. Do you have it? I could write my own, I guess, but it's nice to recycle :)

[reply]
        I guess your idea of writing a debugging proxy is probably what I'll need to figure this out. Do you have it? I could write my own, I guess, but it's nice to recycle :)

        If I still have it somewhere (which I doubt), it's on a tape for which I currently don't have a drive to read.

        Abigail

[reply]
        I prefer to use an ethernet sniffer for network protocol debugging. Look into tcpdump, which is free. There are also some very expensive but very good windows sniffers available. Iris is a particularly sexy one which comes to mind. 
           MeowChow                                   
               s aamecha.s a..a\u$&owag.print
[reply]
Re: Mimicing browser with LWP
by Aristotle (Chancellor) on Jun 05, 2002 at 09:01 UTC
    You could make your life a bit simpler than constructing the post data yourself, using HTTP::Request::Common: 
    return POST("http://www.BrilliantPeople.com/jobsearch.asp", [
    undef,
    # Search for Information Technologies stuff.
    hdnOccupationSelectionCodes => join(",",5118..5137),
    txtKeywords => 'programmer',
    # numberperpage => 100,
    # currentpage => $page,
]);

    [download]
    But that's just a sidenote. What I wanted to ask is did they change their site in the space of half a day? I can't find any form with a "hdnOccupationSelectionCodes" parameter there.

    Makeshifts last the longest.

[reply]
[d/l]

Back to Seekers of Perl Wisdom