Re^2: Storing UTF-8 data into database from scraped web page

Yeah, I tried that. It just makes things even uglier. Output from mysql looks like this: Whatā s up with the water ??

Output when I dump the scraped content looks like this: Whatā~@~Ys up with the water ??

$PM = "Perl Monk's";
$MCF = "Most Clueless ~~Friar~~ ~~Abbot~~ ~~Bishop~~ ~~Pontiff~~ ~~Deacon~~ ~~Curate~~ Priest";
$nysus = $PM . ' ' . $MCF;
Click here if you love Perl Monks

Comment on Re^2: Storing UTF-8 data into database from scraped web page Select or Download Code

Replies are listed 'Best First'.
Re^3: Storing UTF-8 data into database from scraped web page by thanos1983 (Parson) on Jun 14, 2018 at 20:58 UTC
Hello again nysus, I just tried that on my local DB and it works. I found the module Text::Unidecode it should do what you need. `#!/usr/bin/perl use utf8; use strict; use warnings; use Text::Unidecode; my $encode = unidecode("What\x{2019}s up with the water ??"); print $encode . "\n"; __END__ $ perl test.pl What's up with the water ??` [download] Update: Not to forget you need to define also the column in your table as: `Column` VARCHAR(150) CHARACTER SET utf8 NOT NULL UNIQUE, [download] You do not the parameter `NULL UNIQUE` I just usually add them on my columns so I will avoid duplications etc. Update2: Sample of the whole code that I tested: Read more... (5 kB) The `conf.ini` file: Read more... (300 Bytes) Hope this helps, BR. Seeking for Perl wisdom...on the process of learning...not there...yet!	[reply] [d/l] [select]

Replies are listed 'Best First'.

Re^3: Storing UTF-8 data into database from scraped web page
by thanos1983 (Parson) on Jun 14, 2018 at 20:58 UTC

Hello again nysus,

I just tried that on my local DB and it works. I found the module Text::Unidecode it should do what you need.

#!/usr/bin/perl
use utf8;
use strict;
use warnings;
use Text::Unidecode;

my $encode = unidecode("What\x{2019}s up with the water ??");
print $encode . "\n";

__END__
$ perl test.pl
What's up with the water ??
[download]

Update: Not to forget you need to define also the column in your table as:

`Column` VARCHAR(150) CHARACTER SET utf8 NOT NULL UNIQUE,
[download]

You do not the parameter NULL UNIQUE I just usually add them on my columns so I will avoid duplications etc.

Update2: Sample of the whole code that I tested: