This is my second german character question of the day, first was here.

Anyway, I isolated another weird behavior when I do an html post request. Specifically, I'm trying to retrieve google keyword suggestions for words with german characters in them. (People that search for x are also likely to search for y.)

The problem is, unless I run a special utility function I rolled myself

germanchars_to_strange_html_chars($query)
[download]

before fetching the html, google returns no keyword suggestions. That is, the html fetch succeeds, but no keywords are fetched, as though I had searched for suggestions on "bqwersjoseifjsslei asflse8asd" which of course retrieves nothing.

Here's the program, with the problem isolated as best I could. The html grabs are saved as "works.html" and "doesntwork.html" so you can see the results for yourself.

Oh -- winxp and active state perl 5.8, though I don't think this could be a platform problem... or could it?

use strict;

use HTTP::Request::Common; # HTTP handling
use LWP::UserAgent; # HTTP handling
use crypt::ssleay;

my $query = 'b�rse';
my $www;

#doesn't work.
#sends $query = 'b�rse';
$www = google_keyword_suggestions_html_debug('de', 'de', $query);
open F, "> doesntwork.html" or die "Cannot open.";
print F $www->content,"\n";
close F;    


#works -- keyword suggestions are retrieved
#sends $query = 'börse';
#keyword suggestions are retrieved, although the html is kind of warpe
+d looking.
$query = germanchars_to_strange_html_chars($query);
$www = google_keyword_suggestions_html_debug('de', 'de', $query);
open F, "> works.html" or die "Cannot open.";
print F $www->content,"\n";
close F;        

# returns $www object containing html for a successful code, or an err
+or code
sub google_keyword_suggestions_html_debug {

    my $language = shift;
    my $country = shift;
    my $query = shift; #this could be a list, but leaving it as a sing
+le word.  maybe change later. 
        
    my $action = POST
        'https://adwords.google.com/select/KeywordSandbox',
        [
            'save'  => "save",
            'wizard_name' => "keywordsandbox_wizard",
            'language' => $language,
            'country' => $country,
            'keywords' => $query,
        ];
    
    my $ua = LWP::UserAgent->new;
    $ua->agent('Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)');
    $ua->timeout(30);
    my $www = $ua->request( $action );
    return $www;
}

# Takes a variable and spits it back out with the proper german charac
+ters
sub germanchars_to_strange_html_chars {
    my $var = shift;
    my %table = ( '�' => 'ß', '�' => 'ä', '�' => 'ö',
              '�' => 'ä', '�' => 'ö', '�' => 'ü',
              '�' => 'ü');


    while (my ($k,$v) = each %table) {
        $var =~ s/$k/$v/g;
    }

    return $var;
}
[download]

I'm wondering if there is some setting or "mode", or locale that I could be in, that would enable my code to run without the hand-rolled post filter. If that's not the case, it seems to me that all post requests with german characters in them are at risk in LWP. In which case I should submit my findings to cpan or something or something, no?

Wise monks, I hope once again you have light to shed into my gloomy cubicle!

thomas. UPDATE: edited above code to use::ssleay, per holli's recommendations below.