Beefy Boxes and Bandwidth Generously Provided by pair Networks
Welcome to the Monastery
 
PerlMonks  

Re: Using pragma utf8::all in processing non-utf data.

by farang (Chaplain)
on Sep 05, 2013 at 05:39 UTC ( [id://1052489]=note: print w/replies, xml ) Need Help??


in reply to Using pragma utf8::all in processing non-utf data.

If you have a file with a known encoding, for instance koi8-r, just use

open( $fh, '< :encoding(koi8-r)', "in_file");
and it should work fine under utf8::all.
And i want the scripts to: a) warn me when i open a file that contains non-utf characters rather than die at it;
I don't think it's possible to do it "when the file is opened" because the error arises when some non-Unicode utf8 sequence is read into Perl's internals. One way to do it is to use eval while reading the file line-by-line and trap the error. Here is some code which does that, trying first in utf8 and if that fails to be valid, warns and retries with koi8-r.
use strict; use warnings; use utf8::all; open(my $fh, '<', "in_file") or die "cannot open in_file: $!"; eval { process_file_by_line() }; if ( $@ =~ /does not map to Unicode/ ) { warn $@; print "...trying encoding koi8-r instead of utf8\n\n"; close $fh; open( $fh, '< :encoding(koi8-r)', "in_file") or die "cannot open i +n_file: $!"; process_file_by_line(); } elsif ( $@ ne '' ) { die $@; # bail out on other eval errors } sub process_file_by_line { while ( <$fh> ) { print; # whatever else... } }

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://1052489]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others about the Monastery: (2)
As of 2024-04-25 20:25 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found