Shizzle has asked for the wisdom of the Perl Monks concerning the following question:

After reading some wisdom on this site about
simulating POST with LWP, I tried it myself.
I was hoping to check an address on the USPS
site using this code:


#!/opt/gnu-irix/bin/perl

use strict;

use HTTP::Request::Common qw(POST);
use LWP::UserAgent;

my $ua = new LWP::UserAgent;

my $req = POST 'http://www.usps.com/cgi-bin/zip4/zip4inq2',
          [ 'Delivery Address' => '3601 Fallstone Court',
            'Zip Code' => '20744'
          ];

my $resp = $ua->request($req);

if ($resp->is_success())
{
  print $resp->content();
}
else
{
  print "Problem\n";
}

[download]
When I run the program, it prints "Problem". Any
thoughts why my scripts fails to retrieve the data?
Thanks in advance for your help.

Replies are listed 'Best First'.
Re: Yet another question on POSTing with LWP
by mdillon (Priest) on Apr 22, 2002 at 21:13 UTC
    It looks like the other fields are required as part of the form submission, even if they are left blank. Changing the my $req line to the following worked for me: 
    my $req = POST 'http://www.usps.com/cgi-bin/zip4/zip4inq2',
          [
            'Firm' => '',
            'Urbanization' => '',
            'Delivery Address' => '3601 Fallstone Court',
            'City' => '',
            'State' => '',
            'Zip Code' => '20744',
            'Submit' => 'Process',
          ];

    [download]
[reply]
[d/l]
[select]
      Tried your suggestion this morning when I got to to office. It worked perfectly. Thanks for the tip and big ups for responding so quickly!!!
[reply]
Re: Yet another question on POSTing with LWP
by Corion (Patriarch) on Apr 22, 2002 at 21:14 UTC

    I added some more reporting to the error case of your script, mostly 

        print $resp->as_string()

    [download]
    , and what I got back was an 500 internal server error. This means most likely, that you didn't supply all parameters needed for the request. Whenever I want to automate using a website, I set up a proxy to log all transfer going back and forth between the two connection points. This makes it easier for me to replicate the whole situation and not to miss out on any field that might be necessary.

    When I look at http://www.usps.com/ncsc/lookups/lookup_zip+4.html, the original webpage, there are many more fields to fill out : 

    <FORM NAME="Table1FORM" ACTION="/cgi-bin/zip4/zip4inq2" METHOD=POST>
<INPUT TABINDEX="1" ID="Firm" TYPE="text" NAME="Firm" VALUE="" SIZE=30
+ MAXLENGTH=30 >
<INPUT TABINDEX="2" ID="Urbanization" TYPE="text" NAME="Urbanization" 
+VALUE="" SIZE=30 MAXLENGTH=30>
<INPUT TABINDEX="3" ID="Delivery Address" TYPE="text" NAME="Delivery A
+ddress" VALUE="" SIZE=30 MAXLENGTH=40 >
<INPUT TABINDEX="4" ID="City" TYPE="text" NAME="City" VALUE="" SIZE=30
+ MAXLENGTH=30>
<INPUT TABINDEX="5" ID="State" TYPE="text" NAME="State" VALUE="" SIZE=
+2 MAXLENGTH=2 >
<INPUT TABINDEX="6" ID="Zip Code" TYPE="text" NAME="Zip Code" VALUE=""
+ SIZE=10 MAXLENGTH=10 >
<INPUT TABINDEX="7" TYPE="submit" NAME="Submit" VALUE="Process" ID="Fo
+rmsButton2"> 
</FORM>

    [download]

    I would send all of them, even with empty content, because the CGI might be very badly written and expect all those fields to be present. But the best bet is always to use a dump of that session via a small proxy to see what fields with which content are actually sent.

    perl -MHTTP::Daemon -MHTTP::Response -MLWP::Simple -e ' ;    # The  
$d = new HTTP::Daemon and fork and getprint $d->url and exit;#spider
($c = $d->accept())->get_request(); $c->send_response( new   #in the
HTTP::Response(200,$_,$_,qq(Just another Perl hacker\n))); ' #  web

    [download]
[reply]
[d/l]
[select]
      You're exactly right! I won't take as_string() for granted anymore. Big ups for getting back to me promptly.
[reply]
Re: Yet another question on POSTing with LWP
by jsprat (Curate) on Apr 22, 2002 at 21:20 UTC
    mdillon helped with your POST problem.
    Have you thought about using Data::Address::Standardize to get the zip codes? I used it for a large mailing I did for a fund raiser - worked quite well with partial addresses and so forth.

    Update: ehdonhon points out that D::A::S is deprecated! Been a while since I used it, but id did it's job nicely. Partial addresses magically became complete, wrong addresses became right.

[reply]

      That module appears to be deprecated. I found the following in its pod:

      NOTE: This code is (nearly) obsolete. It has been gutted, and the guts now reside in Scrape::USPS::ZipLookup. This module persists to maintain the old interface, but use of this module now requires that Scrape::USPS::ZipLookup be installed.
[reply]
      If our firm has Scrape::USPS::ZipLookup installed, I just might try using D::A::S. The LWP method will work, but this seems a little easier. Thanks for the great suggestion and prompt feedback!
[reply]
      I should have known somebody was smart enough to write a program to interface with UPSP's service. Unfortunately, my firm doesn't have Scrape::USPS::ZipLookup installed, so I'm going to have to work a little harder with LWP. Great suggestion, and thanks for the prompt feedback!
[reply]

Back to Seekers of Perl Wisdom