(or possibly 8 Unicode if they were automatically converted for the comparison), but I get 5 non-Unicode and 3 Unicode?? What gives?
No, as far as perl is concerned, you start with 4 Unicode strings and get 8 Unicode strings... in different storage formats.
utf8 flag says pretty much nothing about "Unicodeness".
Is that a problem that encode('utf-8', $_) returns what is indistinguishable from "Unicode string" (as people usually understand it)? Yes, it's a problem in practice.
Think about it this way: "1" in perl is struct PV, 1 is struct IV, "1" + 1 is PVIV (if i remember correctly). Now, what would happen if, say, the string concatenation operator was '+' (plus)? How would you determine what $x + $y actually do? What if cmp did the same thing as <=>, ge was just like =? How would you sort numbers?
That's the situation with "Unicode" and "binary" strings in Perl, pretty much. As Ricardo Signes said:
Right now, you can write programs in Perl that handle all this correctly, using only one tool: extreme vigilance. Or, more likely, two tools: vigilance and a
debugger.
I personally
Devel::Peek instead of debugger :)
Oh, and here's an example of a non-Unicode string:
"\x{FFFF_FFFF_FFFF}"
(Unicode doesn't have such a big "codepoint")