in reply to Re^8: Mixed Unicode and ANSI string comparisons?
in thread Mixed Unicode and ANSI string comparisons?

Given that description, any sense of "sorting" seems pretty meaningless. Is there some other term that might better describe a sequencing of elements that is better than random?

If the overall data is (close to) what you describe, my first inclination would be to partition or segregate the data, by checking for the following conditions in the order shown:

  1. chunks that contain null bytes (these are probably UTF16 or UCS2)
  2. chunks that are entirely comprised of 7-bit ASCII
  3. chunks with some non-ASCII that are properly utf8 encoded
  4. chunks with some non-ASCII that are not proper utf8
  5. chunks that are not utf8 but are entirely mostly comprised of bytes in the range 128-255, except for carriage-returns and line-feeds and maybe tabs (some pre-Unicode Asian encodings could behave this way, even though all such encodings could also accommodate ASCII bytes interspersed with non-ASCII byte pairs that make up 16-bit characters).

Obviously, you have to start by using plain old binmode to read the input as raw bytes. In case you didn't look it up yet, the test for step 3 is: 

eval { decode("utf8",$input,Encode::FB_CROAK) };

[download]
If the eval succeeds, it's utf8 data.

Default sorting within some of those partitions would make sense. For the others, it's not so much a matter of making sense, but rather just behaving in some consistent, predictable way.

Note that group 2 could actually qualify as a subset of groups 3-5 - and that's a good reason to keep it distinct from those others.

Apart from that, if there's some desire to "classify" or "cluster" the non-ASCII, non-Unicode strings, statistics on byte ngrams can help a fair bit with that (but it remains a bit of a research task, with some training of models required for classification).

(updated to amend the conditions for set 5)

Replies are listed 'Best First'.
Re^10: Mixed Unicode and ANSI string comparisons?
by BrowserUk (Patriarch) on Dec 16, 2015 at 11:55 UTC

    Thanks for taking the time to consider the implications of my question seriously.

    Given that description, any sense of "sorting" seems pretty meaningless.

    Hm. They want a unified index. Generally, they want similar things to be found roughly together; and given something specific to look for, a rough idea of where to start looking. 'Ordered'? 'Collated'? I'm not sure that any other term is much better?

    Apart from that, if there's some desire to "classify" or "cluster" the non-ASCII, non-Unicode strings, statistics on byte ngrams can help a fair bit with that (but it remains a bit of a research task, with some training of models required for classification).

    With enough time and knowledge and money, I've no doubt that something along those lines could be done, but they do not have the money to fund such a project. They were looking for a quick fix and I was basically thinking aloud when I asked my question. I didn't anticipate the hostility people would have to answering such a simple question.

    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority". I knew I was on the right track :)
    In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
      Its the end of days

      of the year

      an auspicious time to view questions as accusation and let inferiority complex flare

[reply]

In Section Seekers of Perl Wisdom