This is my second german character question of the day, first was here.

Anyway, I isolated another weird behavior when I do an html post request. Specifically, I'm trying to retrieve google keyword suggestions for words with german characters in them. (People that search for x are also likely to search for y.)

The problem is, unless I run a special utility function I rolled myself 

germanchars_to_strange_html_chars($query)

[download]
before fetching the html, google returns no keyword suggestions. That is, the html fetch succeeds, but no keywords are fetched, as though I had searched for suggestions on "bqwersjoseifjsslei asflse8asd" which of course retrieves nothing.

Here's the program, with the problem isolated as best I could. The html grabs are saved as "works.html" and "doesntwork.html" so you can see the results for yourself.

Oh -- winxp and active state perl 5.8, though I don't think this could be a platform problem... or could it? 

use strict;

use HTTP::Request::Common; # HTTP handling
use LWP::UserAgent; # HTTP handling
use crypt::ssleay;

my $query = 'börse';
my $www;

#doesn't work.
#sends $query = 'börse';
$www = google_keyword_suggestions_html_debug('de', 'de', $query);
open F, "> doesntwork.html" or die "Cannot open.";
print F $www->content,"\n";
close F;    


#works -- keyword suggestions are retrieved
#sends $query = 'börse';
#keyword suggestions are retrieved, although the html is kind of warpe
+d looking.
$query = germanchars_to_strange_html_chars($query);
$www = google_keyword_suggestions_html_debug('de', 'de', $query);
open F, "> works.html" or die "Cannot open.";
print F $www->content,"\n";
close F;        

# returns $www object containing html for a successful code, or an err
+or code
sub google_keyword_suggestions_html_debug {

    my $language = shift;
    my $country = shift;
    my $query = shift; #this could be a list, but leaving it as a sing
+le word.  maybe change later. 
        
    my $action = POST
        'https://adwords.google.com/select/KeywordSandbox',
        [
            'save'  => "save",
            'wizard_name' => "keywordsandbox_wizard",
            'language' => $language,
            'country' => $country,
            'keywords' => $query,
        ];
    
    my $ua = LWP::UserAgent->new;
    $ua->agent('Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)');
    $ua->timeout(30);
    my $www = $ua->request( $action );
    return $www;
}

# Takes a variable and spits it back out with the proper german charac
+ters
sub germanchars_to_strange_html_chars {
    my $var = shift;
    my %table = ( 'ß' => 'ß', 'ä' => 'ä', 'ö' => 'ö',
              'Ä' => 'ä', 'Ö' => 'ö', 'Ü' => 'ü',
              'ü' => 'ü');


    while (my ($k,$v) = each %table) {
        $var =~ s/$k/$v/g;
    }

    return $var;
}

[download]

I'm wondering if there is some setting or "mode", or locale that I could be in, that would enable my code to run without the hand-rolled post filter. If that's not the case, it seems to me that all post requests with german characters in them are at risk in LWP. In which case I should submit my findings to cpan or something or something, no?

Wise monks, I hope once again you have light to shed into my gloomy cubicle!

thomas. UPDATE: edited above code to use::ssleay, per holli's recommendations below.

In reply to problem with german chars in html post fetch by tphyahoo

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.