UTF8 to Latin-1 and back using Perl-6

kavita has asked for the wisdom of the Perl Monks concerning the following question:

I'm using Perl 5.6 on Solaris

Is this a valid sub to check if an UTF8 strings contains chars that can be validly translated to Latin-1?

sub validUTF8{
  ($_)=@_;   
  use utf8;
  my @a=split "",$_;  
  foreach (@a) { 
#    if (/[\0-\x{ff}]/  or /[\x{C0}-\x{C3}][\x{80}-\x{FF}]/){         
+    
    if (/[\0-\x{ff}]/  or /[\x{C2}-\x{C3}][\x{80}-\x{FF}]/){     
    }
    else {
      print STDERR "BAD:$_\n";
      no utf8;
      return 0;
    }
  }
  no utf8;
  return 1;
}

#converts from utf8 to latin1
[download]

I don't think the following 2 subs take into account all the latin-1 chars which doing tr one way or the other It doesn't include Latin-1 chars that can be represented like /[\x{C0}-\x{C3}][\x{80}-\x{FF}]. How do we write these better to correctly do the translations?

sub utf8tolatin1 {
  ($_)=@_; 
  use utf8;
  tr/\0-\x{ff}//UC; #change utf8 to  Latin-1
  no utf8;
  return $_;
}

#converts from latin1 to utf8
sub latin1toutf8 {
  ($_)=@_;
  use utf8;
  tr/\0-\0xff//CU; #change  Latin-1 to utf8
  no utf8;
  return $_;
}
[download]

Comment on UTF8 to Latin-1 and back using Perl-6 Select or Download Code

Replies are listed 'Best First'.
Re: UTF8 to Latin-1 and back using Perl-6 by mirod (Canon) on Feb 23, 2001 at 12:51 UTC
I am not enough of an expert in this but the UC/CU modifiers are (already!) deprecated and will be removed from future versions. I think they are not even in 5.7. The 2 preferred ways of converting are using Unicode::String or using Text::Iconv if your system supports the iconv function (try `iconv --list` to check out if it's on your system and for a list of supported character sets, I believe UTF8 or UTF-8 and LATIN1 or ISO_8859-1) or ISO_8859-1:1987 are the codes you want to use. I also found an other way to do utf8->latin1 transcoding, which does not require any additional module. But first a warning: Warning: Calgo-Cult programming follows. Here is the utf8 to latin1 conversion used in XML::TiePYX. I don't pretend I really know what's going on there. `sub encode { my ($text)=@_; $text=~s/\x0a/\\n/g; $text=~s{([\xc0-\xc3])(.)}{ my $hi = ord($1); my $lo = ord($2); chr((($hi & 0x03) <<6) \| ($lo & 0x3F)) }ge; return $text; }` [download]	[reply] [d/l]

Replies are listed 'Best First'.

Re: UTF8 to Latin-1 and back using Perl-6
by mirod (Canon) on Feb 23, 2001 at 12:51 UTC

I am not enough of an expert in this but the UC/CU modifiers are (already!) deprecated and will be removed from future versions. I think they are not even in 5.7. The 2 preferred ways of converting are using Unicode::String or using Text::Iconv if your system supports the iconv function (try iconv --list to check out if it's on your system and for a list of supported character sets, I believe UTF8 or UTF-8 and LATIN1 or ISO_8859-1) or ISO_8859-1:1987 are the codes you want to use.

I also found an other way to do utf8->latin1 transcoding, which does not require any additional module. But first a warning:

Warning: Calgo-Cult programming follows.

Here is the utf8 to latin1 conversion used in XML::TiePYX. I don't pretend I really know what's going on there.

sub encode {
  my ($text)=@_;
  $text=~s/\x0a/\\n/g;
  $text=~s{([\xc0-\xc3])(.)}{
    my $hi = ord($1);
    my $lo = ord($2);
    chr((($hi & 0x03) <<6) | ($lo & 0x3F))
   }ge;

  return $text;
}
[download]

[reply]
[d/l]