I decided to join the 21st century and learn how to work with Unicode and UTF-8 encoding. I've included here a test Perl program that both contains literal UTF-8 text, and writes it into a sqlite DB, reads it back in, and prints it to STDOUT.

The advice given by brian d foy (https://stackoverflow.com/a/47946606/522385) and others has been to include the following two pragmas: 

use utf8; # expect UTF-8 text in this source code
use open qw(:std :encoding(UTF-8)); # do UTF-8 encoding on I/O and STD
+*

[download]
But the following code produces garbage output on STDOUT unless I comment out both pragmas!

The garbage I sometimes see is: Åke Lindström. If I paste and pipe that into `hexdump` I get 

  00000000  c3 83 c2 85 6b 65 20 4c  69 6e 64 73 74 72 c3 83  |....ke 
+Lindstr..|
  00000010  c2 b6 6d                                          |..m|

[download]

Other possibly-relevant info: I'm on MacOS Sequoia. I'm working in iTerm2.app, and I get the same behavior in Terminal.app.

I've read perlunitut and https://perldoc.perl.org/open, probably not enough times.

(Please note: I'm having trouble using UTF-8 text in this post, so unfortunately it's not going to look right. The text I'm trying to use, shown as a hex string, is "c3856b65204c696e64737472c3b66d". I fervently hope that, even without working code, someone can identify what I'm doing wrong.) 

#!/usr/bin/env perl
use v5.40;

# brian d foy recommends using these settings
# (https://stackoverflow.com/a/47946606/522385):

# (1) recognize UTF-8 in this source code:
use utf8; 
# (2) do the right things for writing and reading UTF-8, including to 
+STD*:
use open qw(:std :encoding(UTF-8)); 

my $utf8_text1 = "Åke Lindström"; # contains UTF8 chars:

say "A variable set to a UTF8 literal within perl program";
show ($utf8_text1);

use DBI;
my $dbh = DBI->connect( "dbi:SQLite:dbname=:memory:", "", "",
          { RaiseError => 1, AutoCommit => 1 } );
$dbh->do('CREATE TABLE names (name_id CHAR PRIMARY KEY, name CHAR)');
$dbh->do(qq{INSERT INTO names VALUES("nm0512537", "$utf8_text1")});
my $aoa_ref = $dbh->selectall_arrayref(
    q{SELECT name FROM names WHERE name_id="nm0512537"}
);
say "\nUTF-8 text stored in, and retrieved from, sqlite DB:";
show($aoa_ref->[0][0]);

sub show($str) {
    say "Binary:      ", join ' ', (unpack "H*", $str) =~ m/../g ;
    say "Text>STDOUT: $str";
}

[download]

In reply to My UTF-8 text isn't surviving I/O as expected by ibm1620

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.