in reply to PgPP invalid byte seqnence for encoding UTF8

Hi,

what does the psql driver expect as input? A UTF-8 encoded string?

When you do the following: 

#!/usr/bin/perl
use warnings;
use strict;
use Data::Dumper;
use JSON;

my $json = q({"my": "En d\u00e9mocratie, on a le droit d'avoir tort"})
+;

my $perl  = decode_json $json;
my $flag_utf8 = utf8::is_utf8($perl->{'my'}) ? 1 : 0;
print "is unicode: $flag_utf8\n";
print Dumper($perl), "\n";

[download]
you see, that after decoding JSON you get probably strings which are marked as unicode. As soon as you are using "output" of these strings (a database is one of the many output channels) you have to asked which encoding is expected. That means: If psql expects UTF-8 encoding you have to encode the string appropriately with Encode::encode('UTF-8', $string);

McA

Replies are listed 'Best First'.
Re^2: PgPP invalid byte seqnence for encoding UTF8
by cormanaz (Deacon) on Mar 24, 2013 at 16:22 UTC
    That did take care of the encoding error tho I don't understand why because I thought Perl was natively UTF-8. Anyway it works.

    But now when I try to use that value to update a record like so:

    my $sth = $dbh->prepare("update userinfo set description=? where uid=$
+uid");
$sth->execute(encode('UTF-8',$userdata->[$i]->{description})) || die $
+sth->errstr;

    [download]
    I get DBD::PgPP::st execute failed: ERROR:  array value must start with "{" or dimension information at character 33

    This is confusing as the string is En démocratie, on a le droit d'avoir tort and character 33 is either an "a" or a "v" depending on whether it's counting form zero.

[reply]
[d/l]
[select]

      Hi,

      I'm pretty sure it has nothing to do with encoding but with the way the related columns are defined. Please, show use your table definition of userinfo.

      McA

[reply]
[d/l]
        Here it is

                                        Table "public.user
info"
      Column      |             Type              
| Modifiers | Storage  | Description 
------------------+-------------------------------
+-----------+----------+-------------
 uid              | integer                       
| not null  | plain    | 
 screen_name      | character varying             
| not null  | extended | 
 user_name        | character varying             
| not null  | extended | 
 location         | character varying             
|           | extended | 
 user_language    | character varying             
|           | extended | 
 favourites_count | integer                       |           | plain
    | 
 followers_count  | integer                       |           | plain
    | 
 friends_count    | integer                       |           | plain
    | 
 status_count     | integer                       |           | plain
    | 
 listed_count     | integer                       |           | plain
    | 
 created_at       | character varying             |           | exten
ded | 
 time_zone        | character varying             |           | exten
ded | 
 user_url         | character varying             |           | exten
ded | 
 description      | text[]                        |           | exten
ded | 
 added_on         | timestamp without time zone[] |           | exten
ded | 
Indexes:
    "pk_user" PRIMARY KEY, btree (uid)
Rules:
    my_table_on_duplicate_ignore AS
    ON INSERT TO userinfo
   WHERE (EXISTS ( SELECT 1
           FROM userinfo
          WHERE userinfo.uid = new.uid)) DO INSTEAD NOTHING
Has OIDs: no

        [download]
        Maybe the problem is the brackets after the text type? Those were inserted when I created the column with pgadminIII.
[reply]
[d/l]

In Section Seekers of Perl Wisdom