Re^2: WWW::Mechanize always use utf8

Yes, here is it, this problem appear in a linux 64 and don't appear in windows server, i think becouse windows use iso-8859-1 as charset.

#/usr/bin/perl -wT

use strict;
use CGI qw(param);

binmode STDOUT, ':encoding(iso-8859-1)';

print "Content-type: text/html; charset=iso-8859-1\n\n";

my $q = param('status') || 'nothing';

print <<HTML;
<html>
<body>
<h1>You has selected $q</h1>
<form name="form" action="http://localhost/">
<select name="status">
<option value="ACCIÓN">ACCIÓN</option>
<option value="PINGÜINO">PINGÜINO</option>
</select>
<input type="submit" name="send" value="send">
</form>
</body>
</html>
HTML
[download]

#!/usr/bin/perl

use strict;
use warnings;
use WWW::Mechanize;

binmode STDOUT, ':encoding(iso-8859-1)';

my $mech = WWW::Mechanize->new;
$mech->agent_alias( 'Windows IE 6' );
$mech->add_header('Accept-Charset' => 'iso-8859-1');
$mech->get('http://localhost/');
$mech->submit();
my ($status) = $mech->find_all_inputs( name => 'status', type => 'opti
+on');
if (ref $status && ref $status->{menu} eq 'ARRAY') {
    for my $option (@{$status->{menu}}) {
        $mech->form_name( 'form' );
        $mech->select( 'status', $option->{value}); # Here's the probl
+em is send some utf8 string instead iso-8859-1
        sleep(2);
        $mech->click('send'); 
        if ($mech->success()) {
            print $mech->content(); # the response is in utf8 too
        }
        $mech->back();
    }
}
[download]

Comment on Re^2: WWW::Mechanize always use utf8 Select or Download Code

Replies are listed 'Best First'.
Re^3: WWW::Mechanize always use utf8 by ikegami (Patriarch) on Mar 22, 2009 at 06:58 UTC
I can replicate the problem with WWW::Mechanize 1.54, but not with WWW::Mechanize 1.34. (LWP 5.825 in both cases.) My goal was to rule out a badly encoded file. I used the following equivalent cgi script for testing: `#!/usr/bin/perl -wT use strict; use CGI qw(param); binmode STDOUT, ':encoding(iso-8859-1)'; print "Content-type: text/html; charset=iso-8859-1\n\n"; my $q = param('status') \|\| 'nothing'; print <<HTML; <html> <body> <h1>You has selected $q</h1> <form name="form" action="http://localhost/"> <select name="status"> <option value="ACCI\x{D3}N">ACCI\x{D3}N</option> <option value="PING\x{DC}INO">PING\x{DC}INO</option> </select> <input type="submit" name="send" value="send"> </form> </body> </html> HTML` [download] The relevant difference between your systems is not the OS, it's the version of the module.	[reply] [d/l]
Re^4: WWW::Mechanize always use utf8 by Anonymous Monk on Mar 23, 2009 at 04:42 UTC
Yes ikegami, downgrading the version of the module solve the problem exactly as you said, do you think that it's a bug, to report? Thank you a lot	[reply]
Re^5: WWW::Mechanize always use utf8 by ikegami (Patriarch) on Mar 23, 2009 at 05:01 UTC
I'm pretty sure it's a bug (but I'm not sure). It's definitely inconsistent with the behaviour of browsers.	[reply]
Re^6: WWW::Mechanize always use utf8 by way (Sexton) on Mar 23, 2009 at 20:04 UTC
Re^7: WWW::Mechanize always use utf8 by ikegami (Patriarch) on Mar 23, 2009 at 20:56 UTC
Some notes below your chosen depth have not been shown here
Re^3: WWW::Mechanize always use utf8 by dolmen (Beadle) on Mar 23, 2009 at 12:34 UTC
First, the CGI code is not portable as you do not specify the encoding of the source code. Either use pure ASCII () or add an encoding statement ("use utf8;" if your source code is encoded in UTF-8). Secondly, URL encoding is a historical problem. Originally URL were defined as ASCII only. But some people started to encode non ASCII (8 bits) characters. Some using iso-8859-1. Some with UTF-8. Some with other encodings. Then the IETF normalized the URL encoding for HTTP as UTF-8. For backward compatibilty, the User-Agent are using the encoding of the document of the form source to decide which encoding to use in GET URLs. You can change this behavior in MSIE in the advanced settings. So WWW::Mechanize is working as expected. Change your CGI output to UTF-8 and WWW::Mechanize will probably send URL encoded as UTF-8.	[reply]
Re^4: WWW::Mechanize always use utf8 by ikegami (Patriarch) on Mar 23, 2009 at 13:20 UTC
First, the CGI code is not portable as you do not specify the encoding of the source code. How is it not portable? The script will be portable no matter what encoding he specifies as long as the encoding in the following two lines match: `binmode STDOUT, ':encoding(iso-8859-1)'; ... print "Content-type: text/html; charset=iso-8859-1\n\n";` [download] The only question is whether the browser can encode using iso-8859-1 or not. I'd be very surprise to meet one that couldn't.	[reply] [d/l]
Re^3: WWW::Mechanize always use utf8 by Anonymous Monk on Mar 22, 2009 at 08:08 UTC
How can you tell from that code?	[reply]
Re^4: WWW::Mechanize always use utf8 by ikegami (Patriarch) on Mar 23, 2009 at 05:01 UTC
I checked the server's access log.	[reply]
Re^5: WWW::Mechanize always use utf8 by Anonymous Monk on Mar 23, 2009 at 06:29 UTC
I would `print CGI->query_string;` :)	[reply] [d/l]
Re^6: WWW::Mechanize always use utf8 by way (Sexton) on Mar 23, 2009 at 19:34 UTC
Re^4: WWW::Mechanize always use utf8 by Anonymous Monk on Mar 23, 2009 at 04:43 UTC
Yes, downgrading the version as ikegami said, solve the problem	[reply]