I've scraped a lot of websites with LWP and later WWW::Mechanize, and always relied on $response->content (or $mech->content), and i've just been hit for the first time with with a case where content returns binary garbage (my console beeping at me). It was gzipped html, and decoded_content returns the html I expected ( $mech->response()->decoded_content() ). 

Can't locate object method "decoded_content" via package "WWW::Mechani
+ze"

[download]
Took me about 20 minutes to figure it out.

Replies are listed 'Best First'.
Re: HTTP::Response content or decoded_content
by Jenda (Abbot) on Jun 01, 2008 at 21:55 UTC

    Data::Dumper is your friend. I've been hit by the same thing just last week and spent a few minutes on that as well. I first thought I received some image or something so I dumped the whole HTTP::Response object and noticed the gzip encoding in the headers. Luckily I noticed the decoded_content() sooner than I wasted time searching for and reading docs of a gzip module and decompressing it myself.

    I think the two methods should have been named differently. content() should have been raw_content() and decoded_content() should have been content(). This can't be helped now, but I think the examples should be changed to use decoded_content().

    Jenda
    Support Denmark!
    Defend the free world!

[reply]

Back to Meditations