Hi there

An interesting question, but I think you will have to provide some more information:

• What do the special characters look like in the original data?>
• Are they encoded with &#xxx; or in some encoding scheme?
• Is there a header indicating the encoding?
• What do the characters look like when viewed as hex?
• What is the bit of code that is removing the characters?
• What version of perl are you using?

I won't be online till tomorrow, but hopefully someone else will be able to help you if you provide that information.

Cheers, F.V.S.

I don't know if it alerts you when a post you've commented on is updated, so in case it doesn't, I've updated the post with the info you asked for. Note that the second update has the correct URL and you should ignore the URL in the first update.

Hmm

I can't see the obvious source of the problem. I think you need to dump out the result of the request before any processing and be sure exactly where the special characters are being lost. i.e. is it coming correctly out of LWP, is it the regex, could it be the MARC:: module, etc.

As graff said it shouldn't be losing these characters, but there are a number of places where things can go wrong.

It's all a bit complicated and I can't think of a good guide to it at the moment. On the other hand, I've never heard of Perl completely stripping special characters because of an encoding problem - normally, you would get a multi-byte utf-8 character treated as 2 or 3 characters if the encoding is not set correctly. So I suspect an error in some code somewhere - could it be that something is validating input and stripping out characters it doesn't think are "safe"...?

Sorry I can't be of more help. Try to narrow it down to where they disappear and it will be solved eventually.

It matters what sort of character encoding the web site is using (some sort of latin-1? utf-8? something else?), and it also matters what your script is doing when opening file handles for input or output, making database connections, and using LWP methods. Oh, and it also matters what character encoding is being used in the database. (Is it the same or different compared to what is being used at the web site?)

Lacking all those details, I don't think there's much we can say about your problem -- except that it sounds a bit implausible: if the web site content includes accented characters, I wouldn't expect a quiet conversion to "basic ASCII", unless your script is explicitly applying this sort of behavior somehow. I might expect warnings or errors or some sort of character-entity-reference stuff, if the data is ending up different from its original form.

in order to decode these you can use

HTML::Entities::decode_entities($string)
[download]

That said, more info would be useful as this solution only deals with the default case.

Since I got different content from what you were getting, your regex didn't really apply for my data (and I guess your regex isn't related to the problem anyway, since it has nothing to do with accented letters). Anyway, here's some code that demonstrates how the non-ascii content works:

#!/usr/bin/perl

use strict;
use LWP::UserAgent;
use HTTP::Request;

use Encode;  # you need this module

binmode STDOUT, ":utf8";

my $ua = LWP::UserAgent->new;

my $url = "some_url_that_works_for_you";
my $req = HTTP::Request->new( GET => $url );

my $res = $ua->request( $req );
$txt = decode( 'utf-8', $res->content );  # decode "external" utf8 to 
+"internal"

my @accented = ( $txt =~ /(\w*?[^[:ascii:]]\w*)/g );
if ( @accented ) {
    printf( "found %d words with non-ascii characters.\n", scalar @acc
+ented );

    my @alphanumerics = grep /^\w+$/, @accented;
    printf( "of those, %d words match ^\\w+\$:\n ", scalar @alphanumer
+ics );
    print join( "\n ", @alphanumerics ),"\n";

    my @diacritic_marks = grep /\p{NonspacingMark}/, @accented;
    printf( "and %d used separate diacritic marks:\n ", scalar @diacri
+tic_marks );
    print join( "\n ", @diacritic_marks ), "\n";
}
[download]

You might want to check out this little command-line tool I posted a while back -- it can really help with getting a handle on what kinds of unicode data you are really dealing with: tlu -- TransLiterate Unicode; check my home page for a few other unicode tools.

(UPDATE: Forgot to mention -- I also noticed that the source data from the web site tended to use both the single-character "accented_letter" and the two-character "letter accent_mark" for the same thing -- that is, their unicode usage is inconsistent, and somewhat non-standard.)