in reply to how to find the length of any line of file

length() by default counts bytes, even if you have multibyte characters (UTF8/unicode) in your file.

For unicode lines to be counted correctly you need to open the file you are about to read with "encoding(UTF-8)" and if you want to print it out, you need to use "use utf8;" pragma.

#!/usr/bin/perl
use utf8;
use Encode;
use strict;
use warnings;

open(FF, "<:encoding(UTF-8)",'unicode.txt');
while(<FF>){
        chomp;
        my $l = length($_);
        my $str = "linenr=$. length=$l $_ \n";
        if(utf8::is_utf8($str)){
                # get rid of the wide character warning
                print encode('utf-8', $str); 
        }else{
                print $str;
        }
}
close(FF);

[download]

Now it could be your file is encoded in something else than UTF8 (like UTF16), so this might not be your solution yet.

See also how-do-i-find-the-length-of-a-unicode-string-in-perl

edit note: Edited the response to be more precise, as Choroba suggested.

Replies are listed 'Best First'.
Re^2: how to find the length of any line of file
by choroba (Cardinal) on Nov 24, 2017 at 20:03 UTC
    That's very imprecise.

    #! /usr/bin/perl
use warnings;
use strict;
use feature qw{ say };

my $string = "\N{LATIN SMALL LETTER A WITH GRAVE}";
say length $string;     # 1

{
    use bytes;
    say length $string; # 2
}

open my $OUT, '>:utf8', '1' or die $!;
print {$OUT} $string;
close $OUT;

no utf8;
open my $IN, '<:utf8', '1' or die $!;
my $char = <$IN>;
say length $char;       # 1
say $char eq $string;   # 1

use utf8;
open $IN, '<', '1' or die $!;
my $bytes = <$IN>;
say length $bytes;      # 2

    [download]

    utf8 tells Perl that the source code uses UTF-8. length is sensitive to bytes and to the UTF8 flag of its argument.

    ($q=q:Sq=~/;[c](.)(.)/;chr(-||-|5+lengthSq)`"S|oS2"`map{chr |+ord
}map{substrSq`S_+|`|}3E|-|`7**2-3:)=~y+S|`+$1,++print+eval$q,q,a,

    [download]
[reply]
[d/l]
[select]

In Section Seekers of Perl Wisdom