Thanks for the update. I had no luck with the urls you posted, but I was able to go to the web page, put in a request that yielded accented characters in the output, and use the resulting url to push that request through LWP.

Since I got different content from what you were getting, your regex didn't really apply for my data (and I guess your regex isn't related to the problem anyway, since it has nothing to do with accented letters). Anyway, here's some code that demonstrates how the non-ascii content works: 

#!/usr/bin/perl

use strict;
use LWP::UserAgent;
use HTTP::Request;

use Encode;  # you need this module

binmode STDOUT, ":utf8";

my $ua = LWP::UserAgent->new;

my $url = "some_url_that_works_for_you";
my $req = HTTP::Request->new( GET => $url );

my $res = $ua->request( $req );
$txt = decode( 'utf-8', $res->content );  # decode "external" utf8 to 
+"internal"

my @accented = ( $txt =~ /(\w*?[^[:ascii:]]\w*)/g );
if ( @accented ) {
    printf( "found %d words with non-ascii characters.\n", scalar @acc
+ented );

    my @alphanumerics = grep /^\w+$/, @accented;
    printf( "of those, %d words match ^\\w+\$:\n ", scalar @alphanumer
+ics );
    print join( "\n ", @alphanumerics ),"\n";

    my @diacritic_marks = grep /\p{NonspacingMark}/, @accented;
    printf( "and %d used separate diacritic marks:\n ", scalar @diacri
+tic_marks );
    print join( "\n ", @diacritic_marks ), "\n";
}

[download]
Having tried it myself, I learned that non-spacing diacritic marks (presented as separate characters, rather than being an intrinsic part of a letter -- e.g. the second character in "U+0061 U+02CA" for á, rather than U+00E1) all fall into the category of things that match "\w".

You might want to check out this little command-line tool I posted a while back -- it can really help with getting a handle on what kinds of unicode data you are really dealing with: tlu -- TransLiterate Unicode; check my home page for a few other unicode tools.

(UPDATE: Forgot to mention -- I also noticed that the source data from the web site tended to use both the single-character "accented_letter" and the two-character "letter accent_mark" for the same thing -- that is, their unicode usage is inconsistent, and somewhat non-standard.)

In reply to Re: keeping diacritical marks in a string by graff
in thread keeping diacritical marks in a string by Foxpond Hollow

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.