in reply to Re^2: help with cyrillic characters in odd places
in thread help with cyrillic characters in odd places

A good strategy for you would be either to start splitting your code into smaller and smaller parts until some of them start working - and seeing which change made a difference - or combining small self-contained reproducible examples we provide back into a whole that resembles your current code - and seeing when it stops working.

Right now, your encoding handling is doing the wrong thing. Let's start with a small file and get it to output UTF-8 from Perl wide characters:

use warnings;
binmode STDOUT, ":utf8";
print "\x{44b}\n";

[download]

No warnings, the string literal is definitely wide, and the output is evidently UTF-8. This was achieved by adding a perl IO layer to STDOUT that encodes wide characters to UTF-8 bytes. We can verify that:

use Data::Dumper;
binmode STDOUT, ":utf8";
use PerlIO;
print Dumper [ PerlIO::get_layers \*STDOUT, output => 1 ];
__END__
$VAR1 = [
          'unix',
          'perlio',
          'utf8'
        ];

[download]

Your code, 

use open qw/:std :utf8/;
use open OUT => ':encoding(UTF-8)', ':std';

[download]
adds the UTF-8 encoding layer multiple times: 
$VAR1 = [
          'unix',
          'perlio',
          'utf8',
          'encoding(utf-8-strict)',
          'utf8'
        ];

[download]

That would be one of the reasons why you are getting Mojibake instead of Cyrillic characters. It may be helpful to use more simple and explicit code for now, until you understand better the machinery that makes it all tick. Start with binmode STDOUT, ":utf8" and get your code to output correctly-encoded UTF-8 to STDOUT (after reading your code, I think you are almost there: everywhere you get UTF-8 bytes, you decode them correctly before printing). Once that works, start adding pragmas like open that save you typing.

I am not sure why would your code (appear to) entirely skip non-ASCII files and directories, but perhaps we could shed some light on it once we get Unicode display problem resoled.

Replies are listed 'Best First'.
Re^4: help with cyrillic characters in odd places
by Anonymous Monk on Feb 17, 2019 at 07:28 UTC
    Hmm, no, that was wrong, use open qw/:std :utf8/; use open OUT => ':encoding(UTF-8)', ':std'; alone doesn't cause Mojibake on my system (instead, I get proper UTF-8 output). Something else is going on.
[reply]
[d/l]
      Something else is going on.

      I've just tried to avoid that something else as I kept cutting out parts that weren't working, especially with caller, and moved as much of the html stuff out of it as I could.

      I went all the way to hard-coding the paths. There's only two for right now.

      # crossword params
my %vars = (

  cw          => path( $path2, 'crosswords' ),
  изображение => path( $path2, 'crosswords', 'изображение', "1.атаман.jpg" ),
  подписи     => path( $path2, 'crosswords', 'подписи', "1.кгосс.txt" ),

);

if ( $vars{изображение}->is_file ) {
  say "We have a path to the image";
}

my $rvars           = \%vars;
my $ref_html_values = init_values( $title, $path2, $abs );
my %html_vars       = %$ref_html_values;

# append returned hash from init
@vars{ keys %html_vars } = values %html_vars;

      If the following is init_values(), do I combine the hashes in a robust way?

      Read more... (2 kB)

      I'm able to get the captions inputed, and the output makes sense with a few different views to it:

      Read more... (3 kB)

      This is with the following sub:

      sub make_russian_crossword {
  use 5.011;
  use warnings;
  use POSIX qw(strftime);
  use Path::Tiny;
  use Encode;
  use open OUT => ':encoding(UTF-8)', ':std';
  use Data::Dumper;
  use utf8;

  my $rvars = shift;
  my %vars  = %$rvars;

  my $munge = strftime( "p%d-%m-%Y-%H-%M-%S\.txt", localtime );
  my $in_path = path( $vars{rus_captions}, $munge )->touchpath;

  say "in make russian xword------";

  ##Let mother know that you created a file, *verb* a reference:
  $vars{log_file} = $in_path;

  
  ## input use Path::Tiny methods
  my @lines = $vars{подписи}
+->lines_utf8;
  say "lines are @lines";
  my $ref_lines = \@lines;
  vars{data} = $ref_lines;
  say "with Dumper------";
  say Dumper $ref_lines;
  say "with Dump------";
  use Data::Dump; 
  dd $ref_lines;
  #print_aoa_utf8($ref_lines);
  say "using slurp method--------";
  my $guts = $vars{подписи}-
+>slurp_utf8;
  say "guts are $guts";

  my $width = 10;

  ## trim off white space beyond ten for every line
  # trim leading or trailing whitespace

  return $rvars;
}

      [download]

      Where I'm at is looking for ways to get these data rectangular. In particular, I want to trim the lines to length ten and trim leading and trailing whitespace.

      It seems like something that perl would do elegantly.

[reply]
[d/l]
[select]

In Section Seekers of Perl Wisdom