tchrist,

A "common" practice for handling duplicate names in a database is to append non-printable characters after the name, in the order of insertion. This is like using base 32 (numbers 0 to 31 ) for appended characters. This allows duplicates and retains the order of insertion. You don't have a limit since when you fill the first character, you just add another as "\0" and continue from there. That would be broken with Unicode::Collate.

The implication in the article was that you could replace 'sort' with 'Unicode::Collate'.

Thank you

The implication in the article was that you could replace 'sort' with 'Unicode::Collate'.

And that seems to be the real problem. sort isn't broken (that's just link baiting), and neither is Unicode::Collate. They just do different things.

The article does say

Fortunately, you don't have to come up with your own algorithm for dictionary sorting, because Perl provides a standard class to do this for you: Unicode::Collate

So despite its title, it doesn't mandate UC to be a universal replacement for sort, but just for one application.

Perl 6 - second systems done right

moritz,

But all the references in the article are related to data in databases. I goggled ASCII and UTF-8, and found many times "...UTF-8 uses one byte for any ASCII characters, which have the same code values in both UTF-8 and ASCII encoding...", so why are the 0 - 127 characters being redefined? I understand the complexity of the subject, but the designers of UTF-8 knew better than to mess with ASCII, and that is why UTF-8 enhances ASCII.

'Unicode::Collate' is core, so it could be used a lot in the future, as it should be. But a lot of production environments will be affected if they don't know in advance that the code points of ASCII have been redefined.

My hope was that someone would say 'ASCII => 1' will work like Perl 'sort' for ASCII characters and UTF-8, etc for anything above 127.

Thank you

"Well done is better than well said." - Benjamin Franklin

A "common" practice for handling duplicate names in a database is to append non-printable characters after the name, in the order of insertion. This is like using base 32 (numbers 0 to 31 ) for appended characters. This allows duplicates and retains the order of insertion. You don't have a limit since when you fill the first character, you just add another as "\0" and continue from there. That would be broken with Unicode::Collate.

The implication in the article was that you could replace 'sort' with 'Unicode::Collate'.

I’m afraid you’ve swapped my implication with your inference, as I implied no such thing — and what you’ve inferred in no way follows from what I wrote. Quoting myself, I wrote:

If you have code that purports to sort text that looks like this:

@sorted_lines = sort @lines;

Then all you have to get a dictionary sort is write instead:

use Unicode::Collate;
@sorted_lines = Unicode::Collate::->new->sort(@lines);
[download]

See the red part? Clearly, you do not have ‘code that purports to sort text’! Therefore, nothing I wrote applies to you.

You have code that blindly does a mindless numeric sort on code points, not an alphabetic sort on text. What you are doing is not an alphabetic sort. Plus sorting of textual representations of numbers is specifically outside the scope of the UCA.

Of course it’s trivial to modify the UCA sort to take care of your weirdo situation, such that it does a proper text sort on the text and a weirdo binary sort on the binary. But you have to tell it to do that. It doesn’t play mind games with you; here as always, one has to know what one is doing, and why.

tchrist,

See the red part?

I re-checked and you are correct about the red part, and I was wrong for quoting you out of context. I apologize.

Of course it’s trivial to modify the UCA sort to take care of your weirdo situation, such that it does a proper text sort on the text and a weirdo binary sort on the binary. But you have to tell it to do that. It doesn’t play mind games with you; here as always, one has to know what one is doing, and why.

Do I understand you correctly that it can be done? I have read the docs on CPAN and the perldoc on my system, and I don't see how to do this. I know you think my request is "...weirdo binary sort on the..." ASCII, but I could give many instances of real-life uses where both text and binary co-exist and require sorting. One example: a desktop calendar program where all events are in a database server. The key part of key/value pair, would contain binary ASCII data(time, duration, etc) as well as the title for the event and possible sequencing information (base 32). The data value would be a description of the event. No sorting required for that and it could be UTF-nn or ASCII. The database engine doesn't care about the data portion, only the key matters.

It would be wonderful if the database engine could sort the key information so the language of the title was handled correctly and the ASCII portion is also handled correctly.

Thank you

"Well done is better than well said." - Benjamin Franklin

A "common" practice for handling duplicate names in a database is to append non-printable characters after the name, in the order of insertion.

What you need is an invisible letter in Unicode. Just such a letter was proposed several years ago by typographer Michael Everson. His proposed name for the character was INVISIBLE LETTER. Unfortunately, the Unicode Consortium rejected his proposal. See Proposal to add INVISIBLE LETTER to the UCS and Every character has a story #11: U+???? (The Invisible Letter)

If there were such an invisible Unicode character, you could do something like this:

#!perl

use strict;
use warnings;
use open qw( :std :encoding(UTF-8) );
use charnames qw( :full );

use Unicode::Collate;

my $DISAMBIGUATOR_CHARACTER = "\N{LATIN SMALL LIGATURE FFL}"; # U+FB04

my %president_number_by; # President number by president name
my %seen;

while (<DATA>) {
    chomp;

    my ($name, $number) = split m/,/, $_, 2;

    $seen{$name} = exists $seen{$name}
                 ? $seen{$name} . $DISAMBIGUATOR_CHARACTER
                 : $name
                 ;

    $president_number_by{$seen{$name}} = $number;
}

my $collator = Unicode::Collate->new();

for my $name ($collator->sort(keys %president_number_by)) {
    my $number = $president_number_by{$name};

    $name =~ s/$DISAMBIGUATOR_CHARACTER+$//;

    print "$name,$number\n";
}

exit 0;

__DATA__
Washington,1
Adams,2
Jefferson,3
Madison,4
Monroe,5
Adams,6
Jackson,7
Van Buren,8
Harrison,9
Tyler,10
Polk,11
Taylor,12
Fillmore,13
Pierce,14
Buchanan,15
Lincoln,16
Johnson,17
Simpson,18
Hayes,19
Garfield,20
Arthur,21
Cleveland,22
Harrison,23
Cleveland,24
McKinley,25
Roosevelt,26
Taft,27
Wilson,28
Harding,29
Coolidge,30
Hoover,31
Roosevelt,32
Truman,33
Eisenhower,34
Kennedy,35
Johnson,36
Nixon,37
Ford,38
Carter,39
Reagan,40
Bush,41
Clinton,42
Bush,43
Obama,44
Bush,45
[download]

This script produces this output:

Adams,2
Adams,6
Arthur,21
Buchanan,15
Bush,41
Bush,43
Bush,45
Carter,39
Cleveland,22
Cleveland,24
Clinton,42
Coolidge,30
Eisenhower,34
Fillmore,13
Ford,38
Garfield,20
Harding,29
Harrison,9
Harrison,23
Hayes,19
Hoover,31
Jackson,7
Jefferson,3
Johnson,17
Johnson,36
Kennedy,35
Lincoln,16
Madison,4
McKinley,25
Monroe,5
Nixon,37
Obama,44
Pierce,14
Polk,11
Reagan,40
Roosevelt,26
Roosevelt,32
Simpson,18
Taft,27
Taylor,12
Truman,33
Tyler,10
Van Buren,8
Washington,1
Wilson,28
[download]

(For the purpose of demonstrating more than two presidents with the same last name, I had to assume Barack Obama is re-elected in 2012 and Jeb Bush is elected in 2016. I'm sorry if this prospect offends you.)

This is a pure Unicode solution to the problem. There's no commingling of Unicode characters or graphemes with binary data. Unfortunately, however, there isn't a Unicode character with the general property L (Letter) that's guaranteed to be invisible. If there were, it would be just the right character to use for this "weirdo" purpose.

Why did I use the Unicode character LATIN SMALL LIGATURE FFL in the demo script? I don't know exactly. Maybe because it's a character that collates high and seems impossibly unlikely ever to occur in real data.

Jim

Jim,

Thank you for you input. You seem to know quite a bit about Unicode.

What I tried to ask in the original post was why 'use Unicode::Collate;' changed the meaning of characters 0..31? Everything I have read, talked about not changing the meaning of 7bit ASCII.

History of the question:

I don't know if you are familiar with the NoSQL database engine BerkeleyDB (now owned by Oracle), but I have written a pure perl replacement that performs as well. In some cases where the data portion of the key/value pair are very large, it outperforms BerkeleyDB.

Most people on this forum, believe that BerkeleyDB is free. Oracle has added some conditions that make it very expensive( our law firm's counsel ). One example: If a company employee downloads BerkeleyDB and installs it, that's okay. But as a software vendor, if I download it and install it, the company owes Oracle a fee based on number of cores and type of box. For a power7 IBM p-series with 32 cores, the license fee is $ 48,000. for the "free" BerkeleyDB.

Most of our products sell for under $ 5,000. Hard to ask a company to pay an additional $48K.

Since the PurePerlDB already exists, I was looking at adding a feature to use Unicode::Collate, but it broke other features of PurePerlDB. Unfortunately, my only solution now was to put the burden on the software developer to handle Unicode and duplicates, which is the same as BerkeleyDB.

Thanks again for your input...Ed

"Well done is better than well said." - Benjamin Franklin