Hi,

I'am facing a problem with storing UTF8 keys via Perl's BerkeleyDB. On the command line the code works as expected: 

$ perl -Mstrict -Mutf8 -MBerkeleyDB -MEncode -MData::Dumper -le '
  unlink "xx.db";
  tie my %h, "BerkeleyDB::Btree", -Filename=>"xx.db", -Flags=>DB_CREAT
+E;
  my $db=tied %h; $Data::Dumper::Useqq=1;
  $db->filter_fetch_key(
    sub {
      warn ">>fetch: ".Dumper($_);
      $_=decode("utf8", $_);
      warn "<<fetch: ".Dumper($_);
    });
  $db->filter_store_key(
    sub {
      warn ">>store: ".Dumper($_);
      $_=encode("utf8", $_);
      warn "<<store: ".Dumper($_);
    });
  $h{"ä"}=1;
  my @l=keys %h'
>>store: $VAR1 = "\x{e4}";
<<store: $VAR1 = "\303\244";
>>fetch: $VAR1 = "\303\244";
<<fetch: $VAR1 = "\x{e4}";

[download]
The store filter gets a utf8 string and converts it into a byte string. The fetch filter gets a byte string and internalizes it.

When I use the same filters in my program I get this output:

While filling the database (2 keys are stored "äü" and "ää"): 

>>store_key: $VAR1 = "\x{e4}\x{fc}";
<<store_key: $VAR1 = "\303\244\303\274";
>>store_key: $VAR1 = "\x{e4}\x{e4}";
<<store_key: $VAR1 = "\303\244\303\244";

[download]
But reading back fails: 
>>store_key: $VAR1 = "\x{e4}";
<<store_key: $VAR1 = "\303\244";
>>fetch_key: $VAR1 = "\x{e4}\x{e4}";
<<fetch_key: $VAR1 = "\x{fffd}\x{fffd}";

[download]
The perl snippet that produces this output looks like: 
  my $check=qr/\A\Q$prefix\E(.)?/;

  $k=$prefix;
  if( ($rc=$cursor->c_get($k, $v, DB_SET_RANGE))==0 and $k=~$check ) {
    do {
      if( defined $1 ) {
        ...
      }
    } while( ($rc=$cursor->c_get($k, $v, DB_NEXT))==0 and $k=~$check )
+;
  }

[download]
$prefix is initially "ä". So the store filter called from the first c_get sees this utf8 string and converts it correctly into a byte string.

Then the fetch filter should be passed the byte string "\303\244\303\244" but it gets the utf8 string "\x{e4}\x{e4}".

So, what is wrong here?

Why do I read a byte string from the database in one case (command line) and a character string in the other?

Thanks,
Torsten

In reply to BerkeleyDB + UTF8 by tfoertsch

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.