Checking for Valid Characters

hozefa has asked for the wisdom of the Perl Monks concerning the following question:

Replies are listed 'Best First'.
Re: Checking for Valid Characters by rsriram (Hermit) on Sep 11, 2006 at 06:12 UTC
Hi, What you exactly want to check is that whether all the characters in the file/string are within the ASCII character range. There are several ways to check this. First, using a `:ascii:`. Store the file/string you want to check in a variable, say `$str` `if($str !~ /[^[:ascii:]]/) { print "Non ASCII character found in $."; }` [download] `:ascii:` is built-in perl function to check whether there are any non-ascii characters in the content. Secondly, give a range of ASCII characters and find whether your `$str` is within that range. `if ($str =~ /[^!-~\s]/g){print "Non-ASCII character found"}` This is a kind of question which is very frequently asked in this forum. Please try Super search. You will get a lot other ways to do this.	[reply] [d/l] [select]
Re: Checking for Valid Characters by GrandFather (Saint) on Sep 11, 2006 at 04:32 UTC
A little more context may help here. Do you mean a GUI with a little animated finger the jumps around the screen pointing to invalid characters? Perhaps you mean a coloured printout with invalid characters in red? Maybe you mean a list of character positions in the file where there are bad characters? Are the characters expected to be 7 bit ASCII characters, or are they perhaps UTF8, or maybe they are something else? Are control characters ok? How about the DEL character? Perhaps you need to write a little code, then ask for some help when you have sorted out what you really want to achieve? DWIM is Perl's answer to Gödel	[reply]
Re: Checking for Valid Characters by graff (Chancellor) on Sep 11, 2006 at 13:33 UTC
Looking at your example, you might find it better to replace the "bad" character with a "good" equivalent. Maybe the thing between Y and S there should really be an apostrophe ("\x27") -- it's more and more common these days to find text data with "special" non-ASCII characters being used for quotes, apostrophes, hyphens and other basic ASCII punctuation marks. Just deleting them will tend to "damage" the text data. Once you figure out exactly what these "bad" characters are and what they are supposed to mean, replacing them is easy. The first step is to get a proper look at them: `my %badchars; while (<>) { my @bad = ( /([^[:ascii:]])/g ); # find all bad characters if ( @bad ) { for my $badch ( @bad ) { $badchars{$badch}++; } my $badlist = join ' ', map { sprintf("%02x", ord{$_}) } @bad warn "line contains bad char(s) ($badlist): $_"; } } for my $badch ( sort keys %badchars ) { printf "%6d %02x %s\n", $badchars{$badch}, ord($badch), $badch; }` [download] That will tell you (on STDOUT) which non-ASCII characters are in a given set of data, and how many of each. It also lists (on STDERR), the lines containing bad characters, and the hex codes for the bad characters on each line. (If you prefer, you can add some code there to open two distinct output files yourself, write the "warn" messages to one of them and the "printf" output to the other.) That will get you started, but if your input data is utf8 or some other multi-byte encoding, you need to know what encoding it is, and use perl to interpret it correctly as characters. I won't pursue this further, because you haven't given enough detail yet. If you have more questions, come back with some specific details (the perl code you used, some actual input data, and the actual output you got).	[reply] [d/l]