LWP script for Dice metro search form

Plankton has asked for the wisdom of the Perl Monks concerning the following question:

Friends,

I am trying to write a LWP script that I can use to download the search reults from Dice's Metor search page. I am not sure if my script has a problem or if Dice is just making it difficult (or impossible) for my script to work. Here's what I have so far.

#!/usr/bin/perl -w

use strict;
...

my $debug_level = 1;
my $ACTIONURL="http://seeker.dice.com/jobsearch/servlet/JobSearch";
getopts( "U:d:h" );
if ( $opt_h ) { &usage(); exit 0; }
if ( $opt_d ) { $debug_level = $opt_d; }

my $browser;

my $rc = do_LWPsetup( \$browser );

if ($rc==0) {
    die __FILE__ . "[" . __LINE__ . "] Failure: do_LWPStuff failed\n";
}

do_getSearchResults ( $browser );

sub do_getSearchResults {
    my $browser        = shift;

        my $resp  = $browser->post("$ACTIONURL",
                        [
                            'op' => '1000',
                            'METRO_SEARCH' => '1',
                            'rel_code' => '1102',
                            'metro_id' => '1',
                'secret' => '1',
                'FREE_TEXT' => 'Perl',
                'SKILLS' => "",
                'RADIO'  => 'AND',
                'RADIO'  => 'OR',
                'RADIO'  => 'BOOL',
                'AREA_CODES'  => '408',
                'TAXTERM'  => 'ALL',
                'SORT_TYPE' => "", 
                'REQ_TRAVEL' => '1', 
                'TELECOMMUTE' => "", 
                'SORTSPEC' => '0',
                'DAYSBACK' => '30', 
                'NUM_PER_PAGE' => '50', 
                'DETAILED_RESULTS' => '1', 
                            'HOTRESULTS' => ""
#                            'Submit' => 'HOTRESULTS'
                        ],
                        'Content_Type' => 'form-data'
                        );
    if (do_errorcheck($resp,"Search")) { return 0; }
    return 1;
}

...
[download]

This is all I get ...

$ getdice.pl -d2

Begin HTML:

End HTML
[download]

Can anyone give me a clue????
Thanks

Plankton: 1% Evil, 99% Hot Gas.

Comment on LWP script for Dice metro search form Select or Download Code

Replies are listed 'Best First'.
Re: LWP script for Dice metro search form by b10m (Vicar) on Jan 08, 2004 at 20:58 UTC
I never use dice, but on one of my many quests for knowledge through the CPAN maze, I stumbled across thes modules: WWW::Search::Dice, WWW::Search::Scraper::Dice, and WWW::Scraper::Dice. Maybe these can help you? -- b10m	[reply]
Re: LWP script for Dice metro search form by Anonymous Monk on Jan 08, 2004 at 20:59 UTC
http://seeker.dice.com/assets/docs/provider/terms_and_conditions.html CODE OF CONDUCT While using the Site or Site-related services, you agree not to do any of the following without our prior written authorization: ... • Use any search engine, software, tool, agent or other device or mechanism, including without limitation browsers, spiders, robots, avatars or intelligent agents (other than those made available by Dice on the Site or other generally available third party web browsers, e.g., Netscape Navigator or Microsoft Internet Explorer), to navigate or search the Site. ... • Use any robot, spider, site search/retrieval application, or other manual or automatic device or process to retrieve, index, "data mine," or in any way reproduce or circumvent the navigational structure or presentation of the Site or its contents. Anywho, the site sets a cookie. I don't know if the fields change but a couple were different from the fields you had listed. I'd recommend looking at Firebird with the Live HTTP Headers extension installed.	[reply]
Re: Re: LWP script for Dice metro search form by bassplayer (Monsignor) on Jan 08, 2004 at 21:21 UTC
Use any search engine, software, tool, agent or other device or mechanism, including without limitation browsers, spiders, robots, avatars or intelligent agents (other than those made available by Dice on the Site or other generally available third party web browsers, e.g., Netscape Navigator or Microsoft Internet Explorer), to navigate or search the Site. I'm no lawyer, but it seems from this statement as though you can only use a browser if it is 'generally available'. What is LWP but another browser? It is generally available -- anyone can download and use it if they like. Just because it doesn't have a GUI shouldn't change anything. Of course, I will never subject myself to the legal headache of trying to find out. Besides, I've never had great luck with dice anyway... bassplayer	[reply]

CODE OF CONDUCT