Good day bros. I am decoding HTML pages with mod Encode and am having a couple of pesky problems.

The first is when the page doesn't declare its encoding and it is actually utf-8. Since I don't know the encoding I can't name it for Encode's decode method. When I later try to decode entities with HTML::Entities it complains Parsing of undecoded UTF-8 will give garbage when decoding entities at C:/Perl/site/lib/LWP/Protocol.pm line 114. Is there some way to detect the encoding of a response object if it is undeclared?

The second quandry is what to do when a page lists its encoding as, e.g., Windows 125x (where x is some number). I think it's an error to declare that as an encoding because it's really a charset, but that's the web for you. Anyway I still have the problem of what to name as the actual encoding for the benefit of the decode method. Does anyone have experience with this?

TIA....Steve

In reply to Pesky html encoding problems by cormanaz

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.