Also, I used XML::LibXML to parse the structure.

#!/usr/bin/perl
use warnings;
use strict;
use utf8;

use XML::LibXML;

my $file = shift;

my %to_cyrilic = (
    # Insert the hash definition here, see below.
);
my $regex = join '|', sort { length $b <=> length $a } keys %to_cyrili
+c;

my $dom = 'XML::LibXML'->load_html( location => $file );
for my $text ($dom->findnodes('//text()')) {
    my $etext = $text;
    $text->setData($etext) if $etext =~ s/($regex)/$to_cyrilic{$1}/g;
}
print $dom;
[download]

Note that PRE tags preserve non-latin1 characters.

    F => 'Ф',
    H => 'Х',
    N => ':',
    Nj => 'Њ',
    b => 'б',
    c => 'ц',
    d => 'д',
    e => 'e',
    h => 'х',
    i => 'и',
    l => 'л',
    m => 'м',
    n => 'н',
    nj => 'њ',
    p => 'п',
    r => 'р',
    s => 'с',
    t => 'т',
    u => 'у',
    v => 'в',
    z => 'з',
    ć => 'ћ',
    Č => 'Ч',
    č => 'ч',
    Đ => 'Ђ',
    đ => 'ђ',
    Š => 'Ш',
    š => 'ш',
    Ž => 'Ж',
    ž => 'ж',
    # etc., this is enough to run the example.

($q=q:Sq=~/;[c](.)(.)/;chr(-||-|5+lengthSq)`"S|oS2"`map{chr |+ord
}map{substrSq`S_+|`|}3E|-|`7**2-3:)=~y+S|`+$1,++print+eval$q,q,a,
[download]

choroba, thank you very much for the fast and good help. Yes, I see now I could use regex for two character combinations. I have some experience with regex, not too much, but I can do that myself.

I will try your code soon, just have to install that module. The code is really short, can't be shorter. I don't understand all yet but hope will be clear while working with the code.

Sorry, I am not sure that I understand well what You wrote:

"Note that PRE tags preserve non-latin1 characters"

Does it mean that I have to put in hash all Cirillyc letters, including these that are the same as in Latin (a, o, e, k…)?

The comment about PRE tags is about this site: as you've seen, you can't include some characters into CODE tags, as the site changes them into entities, but you can include them into PRE tags.

But your question is a good ont - yes, cyrilic alphabet exists as a whole in the UTF-8, even the letters that are the same as the latin ones. Cf:

 ~ $ perl -CS -lwe 'print chr for 65, 1040'
A
А
 ~ $ perl -CS -we 'print chr for 65, 1040' | xxd
0000000: 41d0 90                                  A..

See Cyrillic capital letter A.

($q=q:Sq=~/;[c](.)(.)/;chr(-||-|5+lengthSq)`"S|oS2"`map{chr |+ord
}map{substrSq`S_+|`|}3E|-|`7**2-3:)=~y+S|`+$1,++print+eval$q,q,a,
[download]

Hi predrag,

choroba has already shown you a more "perlish" approach to the problem, using a hash table and a regular expression. However, note that just because you're writing in Perl doesn't mean you have to do it that way, since in Perl, TIMTOWTDI - There Is More Than One Way To Do It. (My comments that it's better to parse HTML with a module still apply, though.)

I had a look at your code, and even though I haven't tested it myself since you said that it works, it does look like the logic is fairly sound. I'm not entirely clear yet on the order of operations in the foreach $char loop, but as I said before the best way to go about checking it is with enough sample input that exercises all the logic branches.

The one thing that I'm a little confused about is the placement of the if (substr($txtstring, 0, 1) eq "&" ){ statement. It seems to me like this is only handling & characters at certain points in the input string, instead of anywhere in the input string. This might be a place where either index or a regular expression might be more appropriate (or, of course, a full HTML parser :-)).

Anyway, I thought I might give some general comments about your code:

• Instead of binmode(STDOUT, ":utf8"); use open ':encoding(utf8)';, I believe you can just say use open qw/:std :utf8/; (this also affects STDIN and STDERR).
• open INPUT, "<index_latin.html"; - I'd recommend the three-argument form of open, with error handling, as well as lexical filehandles (my $infh instead of INPUT): open my $infh, '<', 'index_latin.html' or die "open: $!";
• undef $/; - the effect of a change to the $/ variable will be global, throughout the whole program. A common way to do this is to use local inside of a do block; the effect of local will then cause the variable to be reset to its original value when the block is exited. You'll see this often in Perl to read an entire file at once ("slurp"): my $infile = do { local $/; <INPUT> };
• You have quite a few variable declarations (my ...) before the code starts. Note that it's usually better to wait with declaring variables until the scope where they're needed, as otherwise there might be conflicts if you accidentally re-use a variable or forget which scope you're working in. For example, instead of my $char; foreach $char ... it's usually better to say foreach my $char ... (unless of course you specifically need $char after the loop ends).
• my $k; - I'd recommend to use textual representations instead of magic numbers here. For example, you can use constant: use constant { INSIDE_TAG=>1, OUTSIDE_TAG=>2 }; and then use the two values INSIDE_TAG and OUTSIDE_TAG instead of the numbers.
• my $Nj; ... $Nj = "&#1034;"; $out = $out.$Nj; can also be written much shorter as $out = $out."&#1034;"; (since each of those variables like $Nj is used only once).

As for your question here about &nbsp;, you're right, my code didn't handle that. The solution is to change the 'text' to 'dtext' (decoded text) in $p->handler(text => sub { ... }, 'text');. Also, I didn't have full UTF-8 handling in that code, I should have said open my $out, '>:utf8', $outfile or die ... to open the output file, and for parsing the input file I should have done this: open my $infh, '<:utf8', $infile or die "open $infile: $!"; $p->parse_file($infh); (this is mentioned in the HTML::Parser documentation).

As you've noticed, PerlMonks isn't perfect in regards to Unicode. Even though Perl itself handles it fine, I just wanted to point out that there are other ways to represent Unicode characters in Perl where the source file can be left in ASCII (and that won't cause trouble when posting to PerlMonks). For example, instead of "č"=>"ч", you can write "\x{010D}"=>"\x{0447}" or "\N{LATIN SMALL LETTER C WITH CARON}"=>"\N{CYRILLIC SMALL LETTER CHE}" (depending on the Perl version, for the latter you may have to add use charnames ':full'; at the top of your code). These forms certainly don't look as nice, so you don't have to use them if Unicode works for you, but it's also noteworthy that this will make the difference between "A" and "А" more obvious (one of them is actually "\N{CYRILLIC CAPITAL LETTER A}").

Hope this helps,
-- Hauke D

Hauke D, thanks so much for such detailed answer, comments and suggestions.

I accepted your suggestion regarding the order of operations in the foreach $char loop, but didn't have time to rewrite this yet, and will try the way you've suggested, that I also see is the most logical. Also, I understand your suggestions about using modules for parsing HTML, instead of doing the way I did. I've already installed XML::LibXML module and tried choroba's code but I didn't get good result yet. Never mind, I tried one other, simplest example with that module, without any conversion, just to have some experience. It worked well and I've noticed it works for &nbsp; The example I've sent doesn't have that test, but I've tried on other example.

You are absolutely right in your third paragraph (handling & characters) when noticed that my block of the code is a too limited solution. I knew that and I've wrote that for &nbsp; just for a test, and was very happy to see that way I could maybe even handle some more complicated and mixed HTML pages. That doesn't mean that I will use that way, it was just a phase in practicing

But I think that I should wait for new design of my site and see what HTML code will have and then, it will be much easier to finish a complete converter. It is because I could have some CSS, or something other that have to be additionaly handled. But anyway, even if I would have some pages with something very special, it will not be so big problem if I work on these conversion "manually".

I've already successfully tried some examples with creating PDF files (a choice for printing some articles), working with directories etc. and that way I am preparing myself for final work on my website design. I often prefer to learn in phases, "in circles", first time just to touch, then deeper etc. instead of going straight to the essence in one step

So, I have to convert all Latin letters, not just "different ones", but other characters (such interpunction etc. will stay untouched? Hopefully, it is not a programmers' site where I would have all possible characters in the text area. :)

All your comments, general and other are really very useful for me and I am learning from these better then just to passively read somewhere.

Regarding my code

 binmode(STDOUT, ":utf8"); use open ':encoding(utf8)';

I will try your suggestion too. I had to put that code (found on the web) because I practiced to print some output on STDOUT and without that Cyrillic letters were not visible.

Somehow, I love TIMTOWTDI, maybe because I love this principle in general life. Regarding this my project, the most important for me is that code works perfectly, so the output will be good too. I do not need too fast code or something too fancy, but of course, I understand that code must be correct and clean.

One separate task for my site in Cyrillic will be IDN encoding, it is something completely new for me and I've recently learned just a little about that from our national domain service. It is because I will have the url that is on Cyrillic too (national domain) and maybe other pages will have urls in Cyrillic too (more complicated option).

I am maybe a bit slow in work and learning, but I don't have always free time and as I've wrote in a previous post, I try to build a good foundation for my future learning, not just to strive for fast solutions. Also, I have many other interests but is is really amazing for me to see that when I've found Perl, I even don't need to go any further in programming and that Perl could be useful for me in some other fields too

Hi predrag,

One separate task for my site in Cyrillic will be IDN encoding

I just wanted to point out the power of CPAN. Perl and CPAN have been around for a long time and two of several areas where Perl excels is text processing and web development. I've already linked you to several HTML and XML processing modules, and a quick search on CPAN for "translit" is what gave me, among other things, Lingua::Translit, and a quick search for "IDN" shows me Net::IDN::Encode, again just one module among several.

use warnings;
use strict;
use open qw/:std :utf8/;

use Lingua::Translit;
my $tr = new Lingua::Translit("ISO/R 9");
my $txt = "\x{0441}\x{0440}\x{043F}\x{0441}\x{043A}\x{0438}";
my $latin = $tr->translit($txt);
my $cyrillic = $tr->translit_reverse($latin);
die "text mismatch" unless $txt eq $cyrillic;
print "$latin <-> $cyrillic\n";

use Net::IDN::Encode qw/domain_to_ascii domain_to_unicode/;
my $idn = "\x{0442}\x{0435}\x{0441}\x{0442}.\x{0441}\x{0440}\x{0431}";
my $asc = domain_to_ascii($idn);
my $dom = domain_to_unicode("xn--e1aybc.xn--90a3ac");
die "domain mismatch" unless $idn eq $dom;
print "$asc <-> $dom\n";
[download]

Output:

srpski <-> српски
xn--e1aybc.xn--90a3ac <-> тест.срб

Note: I did not verify that the "ISO/R 9" transliteration table is identical to the Serbian / Cyrillic transliteration table you're using, but at least Wikipedia says it's suitable.

Regards,
-- Hauke D