When I convert the POST to a GET, I get an error:

Error 400: 

HTTP Web Server: Unknown Command Exception

So my guess is that they really want you to POST your data.

For such tasks, WWW::Mechanize is usually my preferred choice, for it makes stuff so easy. A sample script like this would get you started:

use strict;
use WWW::Mechanize;

my $mech = new WWW::Mechanize;
$mech->get('http://www.stat-usa.gov/nct_all.nsf/Search');
$mech->submit_form(
   form_name    => '_Search',
   fields       => {
        Query   => 'your search term',
   }
);
print $mech->content;
[download]

Thanks for the advice. Does it work similarly to the LWP::UserAgent?

It uses LWP::UserAgent, yes, only (as you can see), it a lot easier to work with. The code above is basically all you need ;-)

--
b10m

All code is usually tested, but rarely trusted.

Could be any number of things. My two best guesses would be:

• Maybe the form processer only accepts POSTS. Why not try POSTing the request instead.
• The "2d58b7a34bbaa3838525703f004f804e" part of your URL looks like it might be a session ID. Perhaps that session has expired.

Another useful tip in situations like this is to install Firefox's LiveHTTPHeaders extension and to see exactly what the HTTP interaction is. You might be missing important headers.

Your second idea, about the session ID, was one worth pursuing. So I tried the URL manually, and I got a search page. I tried removing the "session ID" and I got a page with just 2 links: to a plain search page, and to an advanced search page. Apparently it's the latter the OP has been using, and its canonical URL is http://www.stat-usa.gov/nct_all.nsf/advSearch.

And when I looked in this page's source, the form's action attribute was /nct_all.nsf/2d58b7a34bbaa3838525703f004f804e?CreateDocument: the exact same strange weird ID. So no, apparently it's not variable, but likely, generated by their web site creation tool.

Do note the part after the question mark: "CreateDocument". I propose the OP tries it using POST with this part appended — and obviously, this wouldn't work with GET.

I did try the OP's code as posted at this time, with just this changed (and the broken up words reassembled), and it works for me.

Sorry about the 'GET', it should have been 'POST', althought the result is the same.
I tried it with newer session ids, and got the same result.
I will use your advice for the HTTP headers.
Do you know if there is another way that a website stores info besides cookies and session IDs?

I'm going to guess that that long hex string is some sort of session identifier and you're probably missing a temporary cookie that's set to expire when your browser quits, or perhaps coming from a different machine than the session was started from.

In either case, you're probably going to be better off using WWW::Mechanize to go through the site's login page then navigate to whatever data you're trying to retrieve. Or see if they don't have some sort of SOAP/XMLRPC/REST-y interface for queries that you may can hit directly.

I heartily second the advice given you by both Fletch and b10m: use WWW::Mechanize.

More specific to your stated problem: I would try using a module such as HTTP::Recorder or WWW::Mechanize::Shell to record a successful manual form submission. The output of HTTP::Recorder, for instance, can be "dropped" right into your WWW::Mechanize scripts. (leira's article "Web Testing with HTTP::Recorder" contains an excellent example of how you might want to do this.)

Another important tool for finding out what is really happening behind the scenes between server and browser is a protocol analyzer such as Ethereal.

HTH,

Thank you for pointing out HTTP::Recorder! This looks like exactly the solution to a problem I've been having getting a script to collect multiple pages out of some search results. Thank you!