Perl 5.8.2. I have trouble getting substr to recognize utf8 input. This works:

$ perl -e 'print substr("a\x{2322}bcd", 0, 3), "\n";' | hexdump
00000000  61 e2 8c a2 62 0a
[download]

This does not:

$ perl -e 'print "a\x{2322}bcd\n"' > uni-file
$ perl -ne 'print substr($_,0,3), "\n"' uni-file | hexdump
00000000  61 e2 8c 0a
[download]

Neither does this:

$ perl -ne 'utf8::upgrade($_);
> print substr($_,0,3), "\n"' uni-file |
> hexdump
00000000  61 e2 8c 0a
[download]

But this does:

$ perl -ne 'BEGIN{binmode(STDIN,":utf8");
> print substr($_,0,3), "\n"' uni-file |
> hexdump
00000000  61 e2 8c a2 62 0a
[download]

However, I can't use binmode in my program. In my program I use IO::File and Text::CSV_XS to read a file with cp1252. The line:

$io->open($in_file, "<:raw:encoding(cp1252)")  ||  die(...);
[download]

does indeed read the input, and converts the octet string into utf8, as expected. A few lines later, I need to substr() a column:

while (!$io->eof) {
   $cols = $csvin->getline($io);
   ...
   $str = $col[0];
   print substr($str,0,3), "\n";   # XXX cuts through utf8 multibyte!
}
[download]

but here the string is not recognized as utf8. The substr function cuts right in the middle of a utf8 multibyte. Even adding "utf8::upgrade($str)" doesn't help. There must be something obvious I'm missing here...

---

---

Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!

Titles consisting of a single word are discouraged, and in most cases are disallowed outright.

Read Where should I post X? if you're not absolutely sure you're posting in the right place.

Please read these before you post! —

• How do I compose an effective node title?
• How do I post a question effectively?
• Markup in the Monastery

Posts may use any of the Perl Monks Approved HTML tags:

a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr

You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)

	For:		Use:
	&		&
	<		&lt;
	>		&gt;
	[		&#91;
	]		&#93;

Link using PerlMonks shortcuts! What shortcuts can I use for linking?

See Writeup Formatting Tips and other pages linked from there for more info.