There are two kinds of encoding at play here: Transfer Encoding and Character Encoding.

Transfer encoding allows the content to be compressed during transit among other things. You want the actual content, not the temporary version used for transit, which is why you want to use ->decoded_content. ->content returns the representation of the content during transit, something that's useless on its own.

Character encoding is what allows characters to be represented as bytes. For example, the character encoding US-ASCII associates byte 0x41 with character LATIN CAPITAL LETTER A. The same character is associated with bytes 00 41 using encoding UTF-16be.

In files, characters can only appear in their encoded form. Internally, the same is true for memory, but Perl allows you to work with characters directly instead of the underlying bytes that form it.

That means that you can decode 00 41 to A, but you need to encode A back to into bytes if you want to save it to disk. (print expects bytes, not characters, unless you told it what to do with characters by using binmode :encoding.)

->decoded_content will also decode the character encoding for you if the web server specifies the content is some kind of text (incl HTML and XML) and specifies its character encoding. That can actually be bad, so you can disable that feature by specifying charset => 'none',

You seem to have assumed that the being encoded using UTF-16be is a problem. It's not necessarily, and trying to "fix" it could actually break it. For example, if the file is an XML document, it's not safe to change it's encoding since the document's encoding is specified in the document. The same is often the case for HTML documents as well.

Some background info on what you are trying to do would be useful.

In reply to Re^7: Cannot access HTTP::Response content properly by ikegami
in thread Cannot access HTTP::Response content properly by URAvgDeveloper101

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.