Win32::Internet and TerraServer

Dr. Mu has asked for the wisdom of the Perl Monks concerning the following question:

Back in 2001 I wrote a Perl/Tk script to download tracks from a GPS unit and overlay them onto maps and aerial photos from Microsoft's TerraServer. I used Win32::Internet (I needed the headers to tell GIFs from JPEGs.) to connect to and make requests from terraserver.microsoft.com via the following subroutines:

sub StartInternet {
    $Internet = Win32::Internet->new
}

sub Get {
    my ($url, $req) = @_;
    my $http;
    my $result = $Internet->HTTP($http, $url, 'anonymous', '');
    if ($result) {
        my ($status, $header, $content) = $http->Request("/$req");
        $http->Close;
        return ($header, $content)
    } else {
        return undef
    }
}
[download]

Typical values for $url and $req would be:

$url = 'terraserver.microsoft.com';
$req = 'tile/tile.asp?T=1&S=13&X=321&Y=3329&Z=10'
[download]

and the script worked just fine.

Now, however, Microsoft has changed the url to terraserver-usa.com and seems to be herding everyone into using their .NET RPCs. Nonetheless, the website is still able to retrieve individual tiles using addresses like:

terraserver-usa.com/tile.ashx?T=1&S=13&X=321&Y=3329&Z=10
[download]

Yet, when reconfigured for the new addresses, my script no longer works. I get a "400 Bad request" each time. So my question is, what is my browser (Opera 6 on Windows) sending to the server that Win32::Internet isn't, or vice-versa, and how can I turn a bad request into a good one? (I'd prefer not to use SOAP. It seems like overkill just for accessing static content.)

Comment on Win32::Internet and TerraServer Select or Download Code

Replies are listed 'Best First'.
Re: Win32::Internet and TerraServer by NetWallah (Canon) on Feb 17, 2004 at 19:49 UTC
The most authoritative to answer your question is to compare the packets are sent and received by each method. To see that you need a network protocol analyzer like Ethereal (Free). "Experience is a wonderful thing. It enables you to recognize a mistake when you make it again."	[reply]
Re: Re: Win32::Internet and TerraServer by Dr. Mu (Hermit) on Feb 17, 2004 at 21:25 UTC
Good idea! I used Ethereal, and here is the result. Win32::Internet sent the following request: `GET /tile.ashx?T=1&S=13&X=321&Y=3329&Z=10 HTTP/1.1\r\n Accept: , , , , , , M\360\017\204-\002\r\n User_Agent: Perl-Win32::Internet/0.081\r\n Host: terraserver-usa.com\r\n \r\n` [download] And Opera sent this: GET /tile.ashx?T=1&S=13&X=321&Y=3329&Z=10 HTTP/1.1\r\n User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows 98) Opera 7.23 +[en]\r\n Host: terraserver-usa.com\r\n Accept: text/html, application/xml;q=0.9, application/xhtml+xml;q=0.9, + image/png, image/jpeg, image/gif, image/x-xbitmap, /;q=0.1\r\n Accept-Language: en\r\n Accept-Charset: windows-1252, utf-8, utf-16, iso-8859-1;q=0.6, ;q=0.1 +\r\n Accept-Encoding: deflate, gzip, x-gzip, identity, ;q=0\r\n Cache-Control: no-cache\r\n Connection: Keep-Alive, TE\r\n TE: deflate, gzip, chunked, identity, trailers\r\n \r\n [download] This is an obvious answer to my first question. I guess I need to expand that question to, "What part of this makes the difference?" And, again, my second question, "How do I make Win32::Internet do that? Update: And to answer that I tried specifying the MIME types I would accept, as follows: `sub Get { my ($url, $req) = @_; my ($http, $request); my $result = $Internet->HTTP($http, $url, 'anonymous', 'none', ); if ($result) { my ($status, $header, $content) = $http->Request("/$req", 'GET +', 'HTTP/1.1', 'none', "image/gif\0image/jpeg"); $http->Close; return ($header, $content) } else { return undef } }` [download] And that, apparently, was all the server needed. The script now works as well as it did before. Thanks to NetWallah for pointing me to Ethereal!	[reply] [d/l] [select]