Yaerox has asked for the wisdom of the Perl Monks concerning the following question:

I need to convert all kinds of files to ANSI (8859-1) except ANSI (8859-1) files, skip those.

When I change my code and try to convert files into utf8 it doesn't work anymore. Seems like the decode line only works like this using decode on utf8. It doesn't distinguish ANSI on ISO.
Or am I doing something wrong?

utf8 to ansi

#!/usr/bin/perl -w

use strict;
use warnings;

use Encode qw(encode decode);

my $sFile = "";
my $sLine = "";
my $sCodepoints = "";

if ( $#ARGV == 0 )
{
   $sFile = $ARGV[0] ;
   if ( ! -e $sFile )
   {
        die "File '$sFile' doesn't exist!";
   }
}

open( FILE, "<", "$sFile") or die "Couldn't open file '$sFile'!";
{ # slurp file into string
    local $/;
    $sLine = <FILE>;
    close( FILE );
}

eval { $sCodepoints = decode( "utf8", $sLine, Encode::FB_CROAK ) };
if ( $@ )
{ # input was not utf8
    print "> No UTF-8, maybe ISO-8859-1 ?\n";
    $sCodepoints = $sLine;
}

open( FILENEW, ">:encoding(iso-8859-1)", "$sFile.new" ) or die "Couldn
+'t open file '$sFile.new'!";
print FILENEW $sCodepoints;
close( FILENEW );

[download]


ansi to utf8

#!/usr/bin/perl -w

use strict;
use warnings;

use Encode qw(encode decode);

my $sFile = "";
my $sLine = "";
my $sCodepoints = "";

if ( $#ARGV == 0 )
{
   $sFile = $ARGV[0] ;
   if ( ! -e $sFile )
   {
        die "File '$sFile' doesn't exist!";
   }
}

open( FILE, "<", "$sFile") or die "Couldn't open file '$sFile'!";
{ # slurp file into string
    local $/;
    $sLine = <FILE>;
    close( FILE );
}

eval { $sCodepoints = decode( "iso-8859-1", $sLine, Encode::FB_CROAK )
+ };
if ( $@ )
{ # input was not iso-8859-1
    print "> No ISO-8859-1, maybe UTF8 ?\n";
    $sCodepoints = $sLine;
}

open( FILENEW, ">:encoding(utf8)", "$sFile.new" ) or die "Couldn't ope
+n file '$sFile.new'!";
print FILENEW $sCodepoints;
close( FILENEW );

[download]


I created two test-files ansi.txt and utf8.txt for testing. Content of the files:

1TestöäüÖÄÜß
2TestöäüÖÄÜß
3TestöäüÖÄÜß

[download]

Replies are listed 'Best First'.
Re: convert files to ansi (8859-1)
by Corion (Patriarch) on Mar 29, 2017 at 07:54 UTC

    At least in your second example, you are not properly decoding your input.

    eval { $sCodepoints = decode( "iso-8859-1", $sLine, Encode::FB_CROAK )
+ };
if ( $@ )
{ # input was not iso-8859-1
    print "> No ISO-8859-1, maybe UTF8 ?\n";
    $sCodepoints = $sLine;
}

    [download]

    Here, $sCodepoints does not contain properly decoded UTF-8.

    I would try a loop over the possible encodings:

    for my $encoding_candidate (qw(iso-8859-1 UTF-8)) {
eval { $sCodepoints = decode( $encoding_candidate, $sLine, Encode::FB_
+CROAK ) };
if ( $@ )
{ # input was not $encoding_candidate
    print "> Not $encoding_candidate\n";
    #$sCodepoints = $sLine;
}
}

    [download]
[reply]
[d/l]
[select]
      At least in your second example, you are not properly decoding your input.
      I don't see what should be wrong? I did the same like in my first example, just turned the utf8 into iso-8859-1 on decode and file-open.

      Your idea using the loop would be my second step. Just now I'm worried because ANSI to UTF8 doesn't work properly.
[reply]

        You cannot read octets from a file and then hope that Perl will know that you meant UTF-8. You always have to decode your input and encode your output.

[reply]
[d/l]
[select]
Re: convert files to ansi (8859-1)
by vrk (Chaplain) on Mar 29, 2017 at 08:39 UTC

    This may be heretical, but couldn't you just use iconv?

[reply]
      We used to use iconv, but we have the situation that the files can be utf8 or ansi. If you use ivonc on utf8 files the outcome is corrupt. That's why we're looking for an own perl-solution.
[reply]
Re: convert files to ansi (8859-1)
by Anonymous Monk on Mar 29, 2017 at 08:02 UTC
[reply]
[reply]
[reply]
      binmode(FILE); didn't changed anything. I never used file-open with :raw, I can give it a try...

      Update: both didn't change anything the outcome.
[reply]
[d/l]
[select]

Back to Seekers of Perl Wisdom