Esteem Monks,

LWP::ConsoleLogger::Easy installs some hooks on LWP::UserAgent for printing headers, content etc. during an LWP transaction.

For a particular website the console is messed up after LWP::ConsoleLogger::Easy prints received JSON response containing turkish characters. I suspect it contains broken unicode or I am sending the wrong headers. For other websites with greek or turkish characters in html it works fine. My question is whether I can do something about it.

use strict;
use warnings;

use LWP::UserAgent;
use LWP::ConsoleLogger::Easy qw/debug_ua/;

# with this uncommented, it still breaks but differently
#binmode STDERR, ':utf8';

my $url = 'https://geocode.arcgis.com/arcgis/rest/services/World/Geoco
+deServer/suggest?f=json&maxSuggestions=6&outSR=%7B%22latestWkid%22:43
+26%7D&text=kayseri';
#my $url = 'https://www.sigmalive.com';
#my $url = 'https://www.havadiskibris.com/';

# this also breaks:
# use utf8;
# my $url = 'https://geocode.arcgis.com/arcgis/rest/services/World/Geo
+codeServer/suggest?f=json&maxSuggestions=6&outSR=%7B%22latestWkid%22:
+4326%7D&text=αθήνα';

my $req = HTTP::Request->new(
    'GET' => $url,
    [
        'Connection' => 'keep-alive',
        'Accept' => '*/*',
        'Accept-Encoding' => 'gzip, deflate, br',
        'Accept-Language' => 'en-GB,en;q=0.5',
        'Referer' => 'https://livingatlas.arcgis.com/',
        'Content-Type' => 'application/json',
#        'Content-Type' => 'text/html; charset=utf-8',
        'User-Agent' => 'Mozilla/5.0 (Macintosh; Intel Mac OS X 13.5; 
+rv:116.0) Gecko/20100101 Firefox/116.0',
        'Origin' => 'https://livingatlas.arcgis.com',
        'Sec-Fetch-Dest' => 'empty',
        'Sec-Fetch-Mode' => 'cors',
        'Sec-Fetch-Site' => 'same-site'
    ],
);
my $ua = LWP::UserAgent->new();
my $logger = debug_ua($ua);
my $response = $ua->request($req);

#print $response->decoded_content;

# the console is messed, this is not printed:
print "\n\ndone\n\n";
# echo $? prints 5, so this is reached:
exit(5);

[download]

many thanks, bliako

In reply to LWP::ConsoleLogger::Easy messes the console on (seemingly) unicode LWP response: is it a case of broken unicode? by bliako

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.