Re: LWP::Simple::get($url) does not work for some urls

I think this is the problem (in server response to your request):

HTTP/1.x 200 OK
Date: Sun, 06 Jul 2008 09:00:12 GMT
Server: Apache
X-Powered-By: PHP/5.2.5
Cache-Control: private, s-maxage=0, max-age=0, must-revalidate
Content-Language: en
Vary: Accept-Encoding,Cookie
X-Vary-Options: Accept-Encoding;list-contains=gzip,Cookie;string-contains=enwikiToken;string-contains=enwikiLoggedOut;string-contains=enwiki_session;string-contains=centralauth_Token;string-contains=centralauth_Session;string-contains=centralauth_LoggedOut
Last-Modified: Sat, 05 Jul 2008 05:40:32 GMT
Content-Encoding: gzip
Content-Length: 19469
Content-Type: text/html; charset=utf-8
Age: 53724
X-Cache: HIT from sq30.wikimedia.org, MISS from sq21.wikimedia.org, MISS from mail.irkutsk.2gis.ru
X-Cache-Lookup: HIT from sq30.wikimedia.org:3128, MISS from sq21.wikimedia.org:80, MISS from mail.irkutsk.2gis.ru:3128
Via: 1.0 sq30.wikimedia.org:3128 (squid/2.6.STABLE18), 1.0 sq21.wikimedia.org:80 (squid/2.6.STABLE18)
Proxy-Connection: keep-alive

As you can see it uses gziped response so you must have something which unzip it for you.

Comment on Re: LWP::Simple::get($url) does not work for some urls

Replies are listed 'Best First'.
Re^2: LWP::Simple::get($url) does not work for some urls by tinita (Parson) on Jul 07, 2008 at 12:56 UTC
As you can see it uses gziped response so you must have something which unzip it for you. usually the server sends its content gzipped only if the client says it can handle it. So if you don't send an Accept-Encoding: gzip, you will get the plain content.	[reply]