Re^4: How to Encode/Decode double encoded string.

Hi Haj,

Thank you for the reply.

Yes i did set pg_enable_utf8 = 0, after that only i can see the same above raw strings on the web applications.

Please note whatever i see in the database, i see as it is in the application too.

postgres server encoding and client encoding is 'UTF-8', i tried to change the client_encoding to SQL_ASCII but it didn't help.

I am still not quite sure what exactly needs to be done in order to get around this issue.

Thank you

Comment on Re^4: How to Encode/Decode double encoded string.

Replies are listed 'Best First'.
Re^5: How to Encode/Decode double encoded string. by haj (Vicar) on Sep 22, 2020 at 11:45 UTC
If your database actually contains `��`, then it will no longer be able to provide the invalid UTF-8 string which was converted to this sequence of replacement characters. So either the legacy version of the application didn't store correct data in the first place, or something went wrong when converting the server encoding to UTF-8. In the latter case, you might be able to grab an old database backup and restart from there. We don't have access to your database nor to your application, so it is up to you to find out which part of software did the bogus UTF-8-decoding (bogus because it failed to check for errors). In Perl, you can (and should) trap this type of error by catching errors like this: `Use Encode; eval { decode('UTF-8',$string,Encode::FB_CROAK) }; if ($@) { # Well, that $string wasn't a valid UTF-8-string. # Go now and die in what way seems best to you. -- Denethor }` [download]	[reply] [d/l]

Replies are listed 'Best First'.

Re^5: How to Encode/Decode double encoded string.
by haj (Vicar) on Sep 22, 2020 at 11:45 UTC

If your database actually contains ��, then it will no longer be able to provide the invalid UTF-8 string which was converted to this sequence of replacement characters. So either the legacy version of the application didn't store correct data in the first place, or something went wrong when converting the server encoding to UTF-8. In the latter case, you might be able to grab an old database backup and restart from there.

We don't have access to your database nor to your application, so it is up to you to find out which part of software did the bogus UTF-8-decoding (bogus because it failed to check for errors). In Perl, you can (and should) trap this type of error by catching errors like this:

Use Encode;
eval { decode('UTF-8',$string,Encode::FB_CROAK) };
if ($@) {
   # Well, that $string wasn't a valid UTF-8-string.
   # Go now and die in what way seems best to you. -- Denethor
}
[download]

[reply]
[d/l]