The exiting (production) code uses the open pragma. 

use open qw/:std :encoding(utf8)/;

From what I tell, that forces all $fh to UTF-8. After that, I found all hex-type substitution fail. As I've been labouring with the intention of substituting, I've been testing with '<:raw' instead.

But it looks like the encode step is what's needed. I can at least encode_json now.

use v5.22;
use warnings;

use Devel::Dwarn;
use Cpanel::JSON::XS;
use Encode qw/ decode encode /;
use Text::CSV_XS;

my $csv = Text::CSV_XS->new({ binary => 1, auto_diag => 1 });

while (my $line = <DATA>) {
    say "Before=".$line;
    my $string = encode('UTF8', $line);
    my $x = $csv->parse($string);
    warn $x if !$x;
    my @data = $csv->fields;
    Dwarn \@data;
    my $structure = encode_json(\@data);
}

__DATA__
IR_Choix_0092.tif,C,psi,,"Wild pansy (Viola tricolor), 19th century il
+lustration","19th-century hand painted illustration of wild pansy, he
+art<92>s ease, or love in idleness (Viola tricolor) flower by Pierre-
+Joseph Redoute (1759-1840). Published in Choix Des Plus Belles Fleurs
+, Paris (1827).",N/A,"Pansy, pansies, wild, Viola tricolor, 19th cent
+ury, painted, Engraving, illustration, nobody, no-one, flower, artwor
+k, Pierre Joseph redoute, bloom, blossom, botanical, botanist, bud, f
+lora, floral, history, historic, horticulture, leaves, petal, petals,
+ plant, vintage, watercolor, flower head, painting, stem, victorian s
+tyle, botanic, flowers, plants, Botany",,C,Fl,N/A,,,,^M

[download]

A big thank you. It's been a very frustrating day.

Dermot

In reply to Re^2: Substitute and converting to UTF8 by tomred
in thread Substitute and converting to UTF8 by tomred

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.