Shizzle has asked for the wisdom of the Perl Monks concerning the following question:

Hello fellow Monks,

USPS.com recently made a change to their Zip Lookup program. I tried to automate a query with the following script:

#!/opt/gnu-irix/bin/perl

use strict;

use HTTP::Request::Common qw(POST);
use LWP::UserAgent;

my $ua = new LWP::UserAgent;

my $url = 'http://www.usps.com/zip4/zip4_response.jsp';

my $data = [
        'Selection' => '1',
        'address1' => $ARGV[0],
        'address2' => '',
        'city' => '',
        'state' => '',
        'zipcode' => $ARGV[1],
        'Submit' => ''
    ];

$ua = new LWP::UserAgent;
my $resp = $ua->request(POST $url, $data);

if ($resp->is_success())
{
  print $resp->content();
}
else
{
  print "OOPS\n", $resp->as_string();
}

[download]

At the command prompt, I typed:

address.pl '1713 62nd Avenue' '20785'

[download]

And got this response:

OOPS
HTTP/1.1 500 (Internal Server Error) Server Error
Connection: close
Date: Fri, 16 Aug 2002 15:20:52 GMT
Server: Netscape-Enterprise/6.0
Content-Length: 305
Content-Type: text/html
Client-Date: Fri, 16 Aug 2002 15:20:45 GMT
Client-Peer: 56.0.78.66:80
Title: Server Error

<HTML><HEAD><TITLE>Server Error</TITLE></HEAD>
<BODY><H1>Server Error</H1>
This server has encountered an internal error which prevents it from f
+ulfilling your request. The most likely
cause is a misconfiguration. Please ask the administrator
to look for messages in the server's error log.
</BODY></HTML>

[download]

I urgently need an explanation for this. My initial thought is that LWP cannot interface with Java servlets (.jsp does indicate a java servlet, right?). Your feedback is greatly needed and appreciated.

Replies are listed 'Best First'.
Re: Has USPS banned LWP? (no - your code is incorrect)
by Ovid (Cardinal) on Aug 16, 2002 at 16:11 UTC

    Go look at the page. You'll see, amongst other things, that there are some hidden fields that you left out. I ran a form parsing script on the page and received the following output:

    my $tainted_selection    = $q->param( 'Selection' )    || ''; # select
my $tainted_urbanization = $q->param( 'urbanization' ) || ''; # hidden
my $tainted_firm         = $q->param( 'firm' )         || ''; # hidden
my $tainted_address      = $q->param( 'address' )      || ''; # hidden
my $tainted_address1     = $q->param( 'address1' )     || ''; # text
my $tainted_address2     = $q->param( 'address2' )     || ''; # text
my $tainted_city         = $q->param( 'city' )         || ''; # text
my $tainted_state        = $q->param( 'state' )        || ''; # select
my $tainted_zipcode      = $q->param( 'zipcode' )      || ''; # text
my $tainted_submit       = $q->param( 'Submit' )       || ''; # image

    [download]

    A quickly constructed query string based upon that data succeeded with the warning that they couldn't find the fake address:

    http://www.usps.com/zip4/zip4_response.jsp?Selection=1&urbanization=&f
+irm=&address=&address1=222%20Washington%20Drive&address2=&city=Portla
+nd&state=OR&zipcode=&97212

    [download]

    However, as soon as I left out the hidden "urbanization" field, the script generated an error. Admittedly, this is poor programming on their part, but it you had viewed the source of their HTML, you probably would have caught this.

    Cheers,
    Ovid

    Join the Perlmonks Setiathome Group or just click on the the link and check out our stats.

[reply]
[d/l]
[select]
      Thanks for the explanation. Where can I download the form parsing script you used?
[reply]
[reply]
        Where can I download the form parsing script you used?

        If your browser supports "show source", then you're 98% of the way there. The last 2% is isolating the form in the HTML.

[reply]
Re: PLEASE ADVISE: Has USPS banned LWP???
by thraxil (Prior) on Aug 16, 2002 at 15:36 UTC
    My initial thought is that LWP cannot interface with Java servlets (.jsp does indicate a java servlet, right?)

    .jsp usually indicates a java server page, which in a roundabout way implies servlets.

    but, this is totally irrelevant. the server-side technology used makes no difference on what browsers or user agents can access a site. if the server speaks HTTP, any client that speaks HTTP can interface with it just as easily.

    your problem lies elsewhere.

    anders pearson

[reply]
Re: PLEASE ADVISE: Has USPS banned LWP???
by Arien (Pilgrim) on Aug 16, 2002 at 15:37 UTC

    The explanation for the response is clearly stated: This server has encountered an internal error which prevents it from fulfilling your request..

    LWP happily communicates with JSP the same way your browser does: using HTTP.

    — Arien

[reply]
Re: PLEASE ADVISE: Has USPS banned LWP???
by BorgCopyeditor (Friar) on Aug 16, 2002 at 15:38 UTC

    I don't know anything about the service you're talking about, and your description isn't exactly clear ... but, if this thing is available via a URI then there is always a way of getting LWP to get it. You're using LWP::UserAgent; why not try "dressing up" as Internet Explorer?

    Then again, it could just be that their script is hosed.

    BCE
    --Your punctuation skills are insufficient!

[reply]
      Anders/Ariel/BCE - thanks for the input. That makes sense - if the browser can work with servlets, so can LWP.
[reply]
Are you within their terms of use?
by Anonymous Monk on Aug 17, 2002 at 14:52 UTC
    Read their terms of use. If you are doing this for your company then the odds are very good that you are violating their terms, and if they improve monitoring of their use then you could be shut down on no notice.
[reply]

Back to Seekers of Perl Wisdom