Re^2: UTF8 issue when getting website via LWP::UserAgent in Perl

Replies are listed 'Best First'.
Re^3: UTF8 issue when getting website via LWP::UserAgent in Perl by ultranerds (Hermit) on May 12, 2016 at 14:19 UTC
Hi, Ah, you star... ->decoded_content worked just fine. I agree that they don't always match up - but I'm not really sure how you can get around that, is they have buggered up the correct response headers / meta tags for it ? Cheers Andy	[reply]
Re^3: UTF8 issue when getting website via LWP::UserAgent in Perl by ultranerds (Hermit) on May 12, 2016 at 14:40 UTC
Oh actually... maybe it didn't :/ It works the first time you print it out (directly) ... but then when saved to the DB, it gets corrupt (even though the DB is in utf_bin) Any other ideas? Who would have thought this would be such a royal PITA! I wish we just all moved over to one charset ;) Here is what a dumper of the values looks like: $VAR1 = { 'title_flag' => 0, 'og_desc' => "\x{420}\x{443}\x{43c}\x{44b}\x{43d}\x{438}\x{4 +4f} \x{43d}\x{435} \x{441}\x{43c}\x{43e}\x{436}\x{435}\x{442} \x{43f} +\x{43e}\x{43a}\x{430}\x{437}\x{44b}\x{432}\x{430}\x{442}\x{44c} \x{44 +2}\x{435}\x{43b}\x{435}\x{432}\x{438}\x{437}\x{438}\x{43e}\x{43d}\x{4 +3d}\x{44b}\x{439} \x{43a}\x{43e}\x{43d}\x{43a}\x{443}\x{440}\x{441} \ +x{ab}\x{415}\x{432}\x{440}\x{43e}\x{432}\x{438}\x{434}\x{435}\x{43d}\ +x{438}\x{435}-2016\x{bb}, \x{43f}\x{435}\x{432}\x{435}\x{446} \x{41e} +\x{432}\x{438}\x{434}\x{438}\x{443} \x{410}\x{43d}\x{442}\x{43e}\x{43 +d} \x{43d}\x{435} \x{432}\x{44b}\x{441}\x{442}\x{443}\x{43f}\x{438}\x +{442} \x{432} \x{421}\x{442}\x{43e}\x{43a}\x{433}\x{43e}\x{43b}\x{44c +}\x{43c}\x{435}, \x{430} \x{440}\x{443}\x{43c}\x{44b}\x{43d}\x{441}\x +{43a}\x{438}\x{435} \x{442}\x{435}\x{43b}\x{435}\x{437}\x{440}\x{438} +\x{442}\x{435}\x{43b}\x{438} \x{43d}\x{435} \x{441}\x{43c}\x{43e}\x{4 +33}\x{443}\x{442} \x{43f}\x{440}\x{43e}\x{433}\x{43e}\x{43b}\x{43e}\x +{441}\x{43e}\x{432}\x{430}\x{442}\x{44c} \x{437}\x{430} \x{43f}\x{43e +}\x{43d}\x{440}\x{430}\x{432}\x{438}\x{432}\x{448}\x{438}\x{445}\x{44 +1}\x{44f} \x{43c}\x{443}\x{437}\x{44b}\x{43a}\x{430}\x{43d}\x{442}\x{ +43e}\x{432} \x{2014} \x{438}\x{437}-\x{437}\x{430} \x{434}\x{43e}\x{4 +3b}\x{433}\x{430} \x{432} 16 \x{43c}\x{43b}\x{43d} \x{448}\x{432}\x{4 +35}\x{439}\x{446}\x{430}\x{440}\x{441}\x{43a}\x{438}\x{445} \x{444}\x +{440}\x{430}\x{43d}\x{43a}\x{43e}\x{432}.", 'title' => " \x{420}\x{443}\x{43c}\x{44b}\x{43d}\x{438}\x{44 +f} \x{43d}\x{435} \x{431}\x{443}\x{434}\x{435}\x{442} \x{443}\x{447}\ +x{430}\x{441}\x{442}\x{432}\x{43e}\x{432}\x{430}\x{442}\x{44c} \x{432 +} \x{ab}\x{415}\x{432}\x{440}\x{43e}\x{432}\x{438}\x{434}\x{435}\x{43 +d}\x{438}\x{438}-2016\x{bb} \x{438}\x{437}-\x{437}\x{430} \x{434}\x{4 +35}\x{43d}\x{435}\x{433} - \x{413}\x{430}\x{437}\x{435}\x{442}\x{430} +.Ru ", 'charset' => 'windows-1251', 'og_image' => ' http://img.gazeta.ru/files3/123/8192123/rumi +n-pic905-895x505-99564.jpg', 'description' => "\x{420}\x{443}\x{43c}\x{44b}\x{43d}\x{438} +\x{44f} \x{43d}\x{435} \x{441}\x{43c}\x{43e}\x{436}\x{435}\x{442} \x{ +43f}\x{43e}\x{43a}\x{430}\x{437}\x{44b}\x{432}\x{430}\x{442}\x{44c} \ +x{442}\x{435}\x{43b}\x{435}\x{432}\x{438}\x{437}\x{438}\x{43e}\x{43d} +\x{43d}\x{44b}\x{439} \x{43a}\x{43e}\x{43d}\x{43a}\x{443}\x{440}\x{44 +1} \x{ab}\x{415}\x{432}\x{440}\x{43e}\x{432}\x{438}\x{434}\x{435}\x{4 +3d}\x{438}\x{435}-2016\x{bb}, \x{43f}\x{435}\x{432}\x{435}\x{446} \x{ +41e}\x{432}\x{438}\x{434}\x{438}\x{443} \x{410}\x{43d}\x{442}\x{43e}\ +x{43d} \x{43d}\x{435} \x{432}\x{44b}\x{441}\x{442}\x{443}\x{43f}\x{43 +8}\x{442} \x{432} \x{421}\x{442}\x{43e}\x{43a}\x{433}\x{43e}\x{43b}\x +{44c}\x{43c}\x{435}, \x{430} \x{440}\x{443}\x{43c}\x{44b}\x{43d}\x{44 +1}\x{43a}\x{438}\x{435} \x{442}\x{435}\x{43b}\x{435}\x{437}\x{440}\x{ +438}\x{442}\x{435}\x{43b}\x{438} \x{43d}\x{435} \x{441}\x{43c}\x{43e} +\x{433}\x{443}\x{442} \x{43f}\x{440}\x{43e}\x{433}\x{43e}\x{43b}\x{43 +e}\x{441}\x{43e}\x{432}\x{430}\x{442}\x{44c} \x{437}\x{430} \x{43f}\x +{43e}\x{43d}\x{440}\x{430}\x{432}\x{438}\x{432}\x{448}\x{438}\x{445}\ +x{441}\x{44f} \x{43c}\x{443}\x{437}\x{44b}\x{43a}\x{430}\x{43d}\x{442 +}\x{43e}\x{432} \x{2014} \x{438}\x{437}-\x{437}\x{430} \x{434}\x{43e} +\x{43b}\x{433}\x{430} \x{432} 16 \x{43c}\x{43b}\x{43d} \x{448}\x{432} +\x{435}\x{439}\x{446}\x{430}\x{440}\x{441}\x{43a}\x{438}\x{445} \x{44 +4}\x{440}\x{430}\x{43d}\x{43a}\x{43e}\x{432}." }; [download] ...and here is the outputted JSON: {"page_title":" Румыния не будет участвовать в ŤЕвровидении-2016ť из-за денег - Газета.Ru ","description":"Румыния не сможет показывать телевизионный конкурс ŤЕвровидение-2016ť, певец Овидиу Антон не выступит в Стокгольме, а румынские телезрители не смогут проголосовать за понравившихся музыкантов из-за долга в 16 млн швейцарских франков."} ..yet here is what comes back out: $VAR1 = { 'images' => '', 'all_images' => '{"image_loop":["http://static.gazeta.ru/nm2 +012/i/quotes/finam_head.png","/nm2015//gzt/img/logo.png"," http://img +.gazeta.ru/files3/123/8192123/rumin-pic905-895x505-99564.jpg"," http: +//img.gazeta.ru/files3/885/8195885/igra-pic265-265x150-77294.jpg"," h +ttp://img.gazeta.ru/files3/725/7953725/RIAN_02710972.HR.ru-pic410-410 +x230-99945.jpg"," http://img.gazeta.ru/files3/331/8116331/2016-02-22T +104304Z_1519170817_D1AESOMBZKAD_RTRMADP_3_UKRAINE-TATARS-EUROVISION-p +ic410-410x230-5670.jpg","http://static.smi2.net/srcimg/2780020.png"," +/nm2015/gzt/img/logo_footer.png","http://static.gazeta.ru/nm2012/i/re +uters_a2.png","http://static.gazeta.ru/nm2012/i/prime_a2.png","http:/ +/static.gazeta.ru/nm2012/i/interfax_a2.png","http://static.gazeta.ru/ +nm2012/i/ria_a2.png","http://static.gazeta.ru/nm2012/i/it_a3.png","ht +tp://static.gazeta.ru/nm2012/i/lj_a2.png"," http://img.gazeta.ru/file +s3/123/8192123/rumin-pic905-895x505-99564.jpg"]}', 'url' => 'www.gazeta.ru/culture/2016/04/22/a_8191769.shtml', 'title' => ' Đ ŃĐźŃĐ˝Đ¸Ń Đ˝Đľ ĐąŃĐ´ĐľŃ&#1 +30; ŃŃĐ°ŃŃĐ˛ĐžĐ˛Đ°ŃŃ Đ˛ ÂŤĐĐ˛Ń&# +128;ĐžĐ˛Đ¸Đ´ĐľĐ˝Đ¸Đ¸-2016Âť Đ¸Đˇ-ĐˇĐ° Đ´ĐľĐ˝ĐľĐł - ĐĐ°ĐˇĐľŃ&#13 +0;Đ°.Ru ', 'description' => 'Đ ŃĐźŃĐ˝Đ¸Ń Đ˝Đľ ŃĐźĐžĐśĐľŃ& +#130; ĐżĐžĐşĐ°ĐˇŃĐ˛Đ°ŃŃ ŃĐľĐťĐľĐ˛Đ¸ĐˇĐ¸ĐžĐ˝Đ˝ +ŃĐš ĐşĐžĐ˝ĐşŃŃŃ ÂŤĐĐ˛ŃĐžĐ˛Đ¸Đ´ĐľĐ˝Đ¸Đľ +-2016Âť, ĐżĐľĐ˛ĐľŃ ĐĐ˛Đ¸Đ´Đ¸Ń ĐĐ˝ŃĐžĐ˝ Đ˝Đľ +Đ˛ŃŃŃŃĐżĐ¸Ń Đ˛ ĐĄŃĐžĐşĐłĐžĐťŃĐźĐ +ľ, Đ° ŃŃĐźŃĐ˝ŃĐşĐ¸Đľ ŃĐľĐťĐľĐˇŃĐ¸Ń&#13 +0;ĐľĐťĐ¸ Đ˝Đľ ŃĐźĐžĐłŃŃ ĐżŃĐžĐłĐžĐťĐžŃĐžĐ˛Đ°Ń&#13 +0;Ń ĐˇĐ° ĐżĐžĐ˝ŃĐ°Đ˛Đ¸Đ˛ŃĐ¸ŃŃŃ ĐźŃĐˇŃ +ĐşĐ°Đ˝ŃĐžĐ˛ â Đ¸Đˇ-ĐˇĐ° Đ´ĐžĐťĐłĐ° Đ˛ 16 ĐźĐť +Đ˝ ŃĐ˛ĐľĐšŃĐ°ŃŃĐşĐ¸Ń ŃŃĐ°Đ˝ĐşĐžĐ +˛.', 'grab_id' => '133' }; {"page_title":" Đ ŃĐźŃĐ˝Đ¸Ń Đ˝Đľ ĐąŃĐ´ĐľŃ Ń&# +131;ŃĐ°ŃŃĐ˛ĐžĐ˛Đ°ŃŃ Đ˛ ÂŤĐĐ˛ŃĐžĐ +˛Đ¸Đ´ĐľĐ˝Đ¸Đ¸-2016Âť Đ¸Đˇ-ĐˇĐ° Đ´ĐľĐ˝ĐľĐł - ĐĐ°ĐˇĐľŃĐ°.Ru + ","description":"Đ ŃĐźŃĐ˝Đ¸Ń Đ˝Đľ ŃĐźĐžĐśĐľŃ ĐżĐ +žĐşĐ°ĐˇŃĐ˛Đ°ŃŃ ŃĐľĐťĐľĐ˛Đ¸ĐˇĐ¸ĐžĐ˝Đ˝ŃĐš + ĐşĐžĐ˝ĐşŃŃŃ ÂŤĐĐ˛ŃĐžĐ˛Đ¸Đ´ĐľĐ˝Đ¸Đľ-2016Âť, +ĐżĐľĐ˛ĐľŃ ĐĐ˛Đ¸Đ´Đ¸Ń ĐĐ˝ŃĐžĐ˝ Đ˝Đľ Đ˛Ń +ŃŃŃĐżĐ¸Ń Đ˛ ĐĄŃĐžĐşĐłĐžĐťŃĐźĐľ, Đ° Ń&# +128;ŃĐźŃĐ˝ŃĐşĐ¸Đľ ŃĐľĐťĐľĐˇŃĐ¸ŃĐľĐťĐ¸ +Đ˝Đľ ŃĐźĐžĐłŃŃ ĐżŃĐžĐłĐžĐťĐžŃĐžĐ˛Đ°ŃŃ + ĐˇĐ° ĐżĐžĐ˝ŃĐ°Đ˛Đ¸Đ˛ŃĐ¸ŃŃŃ ĐźŃĐˇŃĐşĐ +°Đ˝ŃĐžĐ˛ â Đ¸Đˇ-ĐˇĐ° Đ´ĐžĐťĐłĐ° Đ˛ 16 ĐźĐťĐ˝ Ń&#136 +;Đ˛ĐľĐšŃĐ°ŃŃĐşĐ¸Ń ŃŃĐ°Đ˝ĐşĐžĐ˛.","cach +ed":1} [download] Cheers Andy	[reply] [d/l] [select]
Re^4: UTF8 issue when getting website via LWP::UserAgent in Perl by Your Mother (Archbishop) on May 12, 2016 at 15:06 UTC
Your DB handle must understand what it's getting (or else you'll have to do your own encode/decode in and out of it) and your DB column charset must be set to what you think it is. Search utf-8 in the Pod for whatever DBI DBD driver you are using and make sure the column definition in your table is what you expect/need.	[reply]
Re^5: UTF8 issue when getting website via LWP::UserAgent in Perl by ultranerds (Hermit) on May 12, 2016 at 15:13 UTC
Hi, Mmm.. its almost like its being double encoded... Here is an example: `print "GOT TITLE BEFORE: $title \n<Br><Br>"; $title = encode("utf-8",$title); print "GOT TITLE: $title \n<Br><Br>"; $DB->table("ReadingGrabCache")->add( { title => $title, url => +"Foo" }); my $test = $DB->table("ReadingGrabCache")->select ( { url => "F +oo" })->fetchrow_hashref; print "BLA 1: $test->{title} \n<br>";` [download] ...which gives me: GOT TITLE BEFORE: Румыния не будет участвовать в ŤЕвровидении-2016ť из-за денег - Газета.Ru GOT TITLE: Đ ŃƒĐźŃ‹Đ˝Đ¸Ń Đ˝Đľ ĐąŃƒĐ´ĐľŃ‚ ŃƒŃ‡Đ°ŃŃ‚Đ˛ĐžĐ˛Đ°Ń‚ŃŒ Đ˛ ÂŤĐ•Đ˛Ń€ĐžĐ˛Đ¸Đ´ĐľĐ˝Đ¸Đ¸-2016Âť Đ¸Đˇ-ĐˇĐ° Đ´ĐľĐ˝ĐľĐł - Đ“Đ°ĐˇĐľŃ‚Đ°.Ru BLA 1: Đ ŃƒĐźŃ‹Đ˝Đ¸Ń Đ˝Đľ ĐąŃƒĐ´ĐľŃ‚ ŃƒŃ‡Đ°ŃŃ‚Đ˛ĐžĐ˛Đ°Ń‚ŃŒ Đ˛ ÂŤĐ•Đ˛Ń€ĐžĐ˛Đ¸Đ´ĐľĐ˝Đ¸Đ¸-2016Âť Đ¸Đˇ-ĐˇĐ° Đ´ĐľĐ˝ĐľĐł - Đ“Đ°ĐˇĐľŃ‚Đ°.Ru After grabbing it back from the DB, if I do this: `$test->{title} = decode("utf-8",$title);` ...then it works! But it seems bonkers to have to do that, when it should be in utf8 already. Is there a way to "check" if a string has the correct utf8 markers? Thanks! Andy	[reply] [d/l] [select]
Re^6: UTF8 issue when getting website via LWP::UserAgent in Perl by Your Mother (Archbishop) on May 12, 2016 at 15:26 UTC
Re^7: UTF8 issue when getting website via LWP::UserAgent in Perl by afoken (Chancellor) on May 12, 2016 at 20:50 UTC
Re^7: UTF8 issue when getting website via LWP::UserAgent in Perl by ultranerds (Hermit) on May 12, 2016 at 15:46 UTC
Some notes below your chosen depth have not been shown here
Re^6: UTF8 issue when getting website via LWP::UserAgent in Perl by Corion (Patriarch) on May 12, 2016 at 15:15 UTC
Re^7: UTF8 issue when getting website via LWP::UserAgent in Perl by ultranerds (Hermit) on May 12, 2016 at 15:19 UTC
Some notes below your chosen depth have not been shown here
Re^4: UTF8 issue when getting website via LWP::UserAgent in Perl by ultranerds (Hermit) on May 12, 2016 at 14:58 UTC
Here is an example code, of where the issue is coming from: use LWP::UserAgent; use HTTP::Request::Common qw(GET); my $ua = LWP::UserAgent->new; # Define user agent type $ua->agent('Mozilla/8.0'); # Request object my $req = GET 'http://www.gazeta.ru/culture/2016/04/22/a_8191769.s +html'; # Make the request my $res = $ua->request($req); binmode STDOUT, ":utf8"; print "Content-Type: text/html; charset=utf-8 \n\n"; use Encode; if ($res->is_success) { my $title; $res->decoded_content =~ /<title>(.+?)<\/title>/ and $title = +$1; # prints correctly here! print "GOT TITLE: $title \n"; $DB->table("ReadingGrabCache")->add( { title => $title, url => + "Foo" }); my $test = $DB->table("ReadingGrabCache")->select ( { url => " +Foo" })->fetchrow_hashref; # buggered content here print "BLA: $test->{title} \n<br>"; } else { print $res->status_line . "\n"; } [download] The DB module is encoding insensative (i.e its not doing any kind of conversion) ... so I'm confused how it can be fine here, and then broken when grabbed back :( Checking it in phpmyAdmin also shows the issue: Đ ŃĐźŃĐ˝Đ¸Ń Đ˝Đľ ĐąŃĐ´ĐľŃ ŃŃĐ°ŃŃĐ˛ĐžĐ˛Đ°ŃŃ Đ˛ ÂŤĐĐ˛ŃĐžĐ˛Đ¸Đ´ĐľĐ˝Đ¸Đ¸-2016Âť Đ¸Đˇ-ĐˇĐ° Đ´ĐľĐ˝ĐľĐł - ĐĐ°ĐˇĐľŃĐ°.Ru The table is quite simple... but maybe I've missed something: CREATE TABLE IF NOT EXISTS `ReadingGrabCache` ( `grab_id` int(11) NOT NULL AUTO_INCREMENT, `url` varchar(255) CHARACTER SET latin1 NOT NULL, `images` text CHARACTER SET latin1 NOT NULL, `title` text COLLATE utf8_bin NOT NULL, `description` text COLLATE utf8_bin NOT NULL, `all_images` longtext CHARACTER SET latin1, PRIMARY KEY (`grab_id`) ) ENGINE=MyISAM DEFAULT CHARSET=utf8 COLLATE=utf8_bin AUTO_INCREMENT= +141 ; [download]	[reply] [d/l] [select]


Do you know where your variables are?
	PerlMonks