I suppose that it is perhaps possible that you may wish to change the weighting of the variably-weighted elements such that they are not ignored the first time around. The constructor’s variable named-argument accepts four possible values to alter how otherwise-ignored code points are to be treated. Per the manpage:

blanked

Variable elements are made ignorable at levels 1 through 3; considered at the 4ᵗʰ level.

non-ignorable

Variable elements are not reset to ignorable.

shifted

Variable elements are made ignorable at levels 1 through 3 their level 4 weight is replaced by the old level 1 weight. Level 4 weight for Non-Variable elements is 0xFFFF.

shift-trimmed

Same as shifted, but all FFFF’s at the 4ᵗʰ level are trimmed.

You may also want to set upper_before_lower => 1.

tchrist,

A "common" practice for handling duplicate names in a database is to append non-printable characters after the name, in the order of insertion. This is like using base 32 (numbers 0 to 31 ) for appended characters. This allows duplicates and retains the order of insertion. You don't have a limit since when you fill the first character, you just add another as "\0" and continue from there. That would be broken with Unicode::Collate.

The implication in the article was that you could replace 'sort' with 'Unicode::Collate'.

Thank you

"Well done is better than well said." - Benjamin Franklin

The implication in the article was that you could replace 'sort' with 'Unicode::Collate'.

And that seems to be the real problem. sort isn't broken (that's just link baiting), and neither is Unicode::Collate. They just do different things.

The article does say

Fortunately, you don't have to come up with your own algorithm for dictionary sorting, because Perl provides a standard class to do this for you: Unicode::Collate

So despite its title, it doesn't mandate UC to be a universal replacement for sort, but just for one application.

Perl 6 - second systems done right

A "common" practice for handling duplicate names in a database is to append non-printable characters after the name, in the order of insertion. This is like using base 32 (numbers 0 to 31 ) for appended characters. This allows duplicates and retains the order of insertion. You don't have a limit since when you fill the first character, you just add another as "\0" and continue from there. That would be broken with Unicode::Collate.

The implication in the article was that you could replace 'sort' with 'Unicode::Collate'.

I’m afraid you’ve swapped my implication with your inference, as I implied no such thing — and what you’ve inferred in no way follows from what I wrote. Quoting myself, I wrote:

If you have code that purports to sort text that looks like this:

@sorted_lines = sort @lines;

Then all you have to get a dictionary sort is write instead:

use Unicode::Collate;
@sorted_lines = Unicode::Collate::->new->sort(@lines);
[download]

See the red part? Clearly, you do not have ‘code that purports to sort text’! Therefore, nothing I wrote applies to you.

You have code that blindly does a mindless numeric sort on code points, not an alphabetic sort on text. What you are doing is not an alphabetic sort. Plus sorting of textual representations of numbers is specifically outside the scope of the UCA.

Of course it’s trivial to modify the UCA sort to take care of your weirdo situation, such that it does a proper text sort on the text and a weirdo binary sort on the binary. But you have to tell it to do that. It doesn’t play mind games with you; here as always, one has to know what one is doing, and why.

A "common" practice for handling duplicate names in a database is to append non-printable characters after the name, in the order of insertion.

What you need is an invisible letter in Unicode. Just such a letter was proposed several years ago by typographer Michael Everson. His proposed name for the character was INVISIBLE LETTER. Unfortunately, the Unicode Consortium rejected his proposal. See Proposal to add INVISIBLE LETTER to the UCS and Every character has a story #11: U+???? (The Invisible Letter)

If there were such an invisible Unicode character, you could do something like this:

#!perl

use strict;
use warnings;
use open qw( :std :encoding(UTF-8) );
use charnames qw( :full );

use Unicode::Collate;

my $DISAMBIGUATOR_CHARACTER = "\N{LATIN SMALL LIGATURE FFL}"; # U+FB04

my %president_number_by; # President number by president name
my %seen;

while (<DATA>) {
    chomp;

    my ($name, $number) = split m/,/, $_, 2;

    $seen{$name} = exists $seen{$name}
                 ? $seen{$name} . $DISAMBIGUATOR_CHARACTER
                 : $name
                 ;

    $president_number_by{$seen{$name}} = $number;
}

my $collator = Unicode::Collate->new();

for my $name ($collator->sort(keys %president_number_by)) {
    my $number = $president_number_by{$name};

    $name =~ s/$DISAMBIGUATOR_CHARACTER+$//;

    print "$name,$number\n";
}

exit 0;

__DATA__
Washington,1
Adams,2
Jefferson,3
Madison,4
Monroe,5
Adams,6
Jackson,7
Van Buren,8
Harrison,9
Tyler,10
Polk,11
Taylor,12
Fillmore,13
Pierce,14
Buchanan,15
Lincoln,16
Johnson,17
Simpson,18
Hayes,19
Garfield,20
Arthur,21
Cleveland,22
Harrison,23
Cleveland,24
McKinley,25
Roosevelt,26
Taft,27
Wilson,28
Harding,29
Coolidge,30
Hoover,31
Roosevelt,32
Truman,33
Eisenhower,34
Kennedy,35
Johnson,36
Nixon,37
Ford,38
Carter,39
Reagan,40
Bush,41
Clinton,42
Bush,43
Obama,44
Bush,45
[download]

This script produces this output:

Adams,2
Adams,6
Arthur,21
Buchanan,15
Bush,41
Bush,43
Bush,45
Carter,39
Cleveland,22
Cleveland,24
Clinton,42
Coolidge,30
Eisenhower,34
Fillmore,13
Ford,38
Garfield,20
Harding,29
Harrison,9
Harrison,23
Hayes,19
Hoover,31
Jackson,7
Jefferson,3
Johnson,17
Johnson,36
Kennedy,35
Lincoln,16
Madison,4
McKinley,25
Monroe,5
Nixon,37
Obama,44
Pierce,14
Polk,11
Reagan,40
Roosevelt,26
Roosevelt,32
Simpson,18
Taft,27
Taylor,12
Truman,33
Tyler,10
Van Buren,8
Washington,1
Wilson,28
[download]

(For the purpose of demonstrating more than two presidents with the same last name, I had to assume Barack Obama is re-elected in 2012 and Jeb Bush is elected in 2016. I'm sorry if this prospect offends you.)

This is a pure Unicode solution to the problem. There's no commingling of Unicode characters or graphemes with binary data. Unfortunately, however, there isn't a Unicode character with the general property L (Letter) that's guaranteed to be invisible. If there were, it would be just the right character to use for this "weirdo" purpose.

Why did I use the Unicode character LATIN SMALL LIGATURE FFL in the demo script? I don't know exactly. Maybe because it's a character that collates high and seems impossibly unlikely ever to occur in real data.

Jim