These are some routines from a simple script to test a search engine. Since I am not able to post the whole script (sorry, sensitive data), I deleted the sensitive parts and retained what I thought of as the core routines in the context of a script skeleton.

The original script proceeds as follows: It fetches a web page that contains several boxes with select options. It parses out the name and options for each box and submits a request for each box and each option, printing out a simple OK / NO RESULTS statement for it, using a call like print format_results($box, $option, $action).

Since there is nothing special about a select box with just one option to select in terms of CGI parameters, these routines should generalize easily to other field types.

Hope to have been of help. 

#!/usr/bin/perl -w

use strict;

# --------------------------------------------------
# fragments from a web testing script
# to give away some helper routines 
# and to show use of the LWP module
# Author: Christian Lemburg, 2000-11-30
# --------------------------------------------------

use Getopt::Std;
use LWP::UserAgent;
use HTTP::Request;
use URI::Escape;

# --------------------------------------------------
# setup and globals
# --------------------------------------------------

$| = 1;

my %opts;

getopts('u:p:d:v', \%opts);

my $user = $opts{'u'} || 'foo';
my $password = $opts{'p'} || 'bar';

my $VERBOSE = $opts{'v'};
my $agent_delay = $opts{'d'} || 2;

# [ ... snip ... ]
# other argument processing - whatever you need

my $ua = LWP::UserAgent->new();
$ua->agent('YourAgent/1.0');
$ua->env_proxy();

# --------------------------------------------------
# action
# --------------------------------------------------

# [ ... snip ... ]

# in here, set:
# 1) $action to the URL of a search script
# to call, with ###BOX### as a placeholder for the name
# of the select box and ###OPTION### as a placeholder for
# the value of the selected option, 
# 2) $box to the name parameter of the select box,
# 3) $option to the value of the selected option,

# then call format_results($box, $option, $action)
# to output a statement on the result of a search

# --------------------------------------------------
# subs
# --------------------------------------------------

sub format_results {
    my ($box, $option, $action) = @_;
    my $box_param = uri_escape($box);
    my $option_param = uri_escape($option);
    $action =~ s|###BOX###|$box_param|g;
    $action =~ s|###OPTION###|$option_param|g;
    my $result = get_result($action);
    return "Selection Box '" . $box . "'" .
    ", Option '$option': " . $result . 
    "\n";
}


sub get_result {
    my ($action) = @_;
    sleep($agent_delay);
    print "Sub Agent: Processing $action ... \n" if $VERBOSE;
    my $sub_ua = LWP::UserAgent->new();
    $sub_ua->agent('YourAgent/1.0');
    $sub_ua->env_proxy();
    my $request = HTTP::Request->new('GET', $action);
    $request->authorization_basic($user, $password);
    my $response = $sub_ua->request($request);
    if ($response->is_success) {
        my $html = $response->content;
        return evaluate_search_result($html);
    } else {
        return 'ERROR';
    }
}


sub evaluate_search_result {
    my ($html) = @_;
    if ($html =~ m|$no_results_indicator{$index_language}|) {
        return 'NO RESULTS';
    } elsif ($html =~ m|$have_results_indicator{$index_language}|) {
        return 'OK';
    } else {
        return 'UNCERTAIN - CHECK RESULT INDICATORS';
    }
}


sub usage {
    return << "EOU";

Usage: $0 [-u][-p][-d][-v] args 

Options:

    u: user - name of user for htaccess authentification
    p: password - password for htaccess authentification
    d: delay - delay between page fetches in seconds, default 2
    v: verbose output

Note:
    If you are located behind a firewall, please set the
    'http_proxy' environment variable to something like
    'http://myproxy.mydomain.com:myport'.

EOU
}

[download]

Christian Lemburg
Brainbench MVP for Perl
http://www.brainbench.com

In reply to Re: Web Automation and POST Confusing - need help by clemburg
in thread Web Automation and POST Confusing - need help by hhalpin

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.