BernieC has asked for the wisdom of the Perl Monks concerning the following question:

This is odd: I'm trying to post a form that has fields with underscores in them. It appears that LWP converts the underscores to hyphens. Can that be? {my problem is that the post doesn't work and I am trying to figure out why not} When I GET the page with the form on it and scrape it for INPUT fields, I find 
[...]
$VAR53 = 'slagk%40xx.com_nomail';
$VAR54 = 'on';
$VAR55 = 'slagk%40xx.com_nodupes';
$VAR56 = 'on';
[...]

[download]
but when I use Dumper to look at the _request object, it tells me
[...]
                     'slagk%40xx.com-plain' => 'on',
                      'slagk%40xx.com-nomail' => 'on',
                      'qoz%40puz.org-language' => 'en',
[...]

[download]
I looked at the code in HTTP::Request::Common and didn't see anything obvious that might be causing this {if this is actually happening, of course}

two questions: is LWP really converting underscores to hyphens? And if so, how can I pass an underscore through to the web server.

Replies are listed 'Best First'.
Re: POST fieldnames with underscores
by Perlbotics (Archbishop) on Aug 11, 2018 at 18:05 UTC

    For HTTP-headers there seems to be a conversion. From HTTP::Headers (used by HTTP::Request::Common via HTTP::Message):

    ON-CANONICALIZED FIELD NAMES
    The header field name spelling is normally canonicalized including the '_' to '-' translation. There are some application where this is not appropriate. Prefixing field names with ':' allow you to force a specific spelling. For example if you really want a header field name to show up as "foo_bar" instead of "Foo-Bar", you might set it like this:

    $h->header(":foo_bar" => 1);

    Assuming, you want to post a list of variables, do transfer them as content (application/x-www-form-urlencoded), not as HTTP-headers.
    Please check POST method invocation parameters (order/names) or post the specific lines, where you're creating the POST request, for further analysis.

[reply]
      No wonder I couldn't find it...:o). For example 
      <FORM action="../../../admin.cgi/testlist-fantasyfarm.com/privacy/send
+er" method="POST" >
<input type="hidden" name="csrf_token" value="280200000069096e6f5b7347
+000000746573746c6973742d66616e746173796661726d2e636f6d2b61646d696e3a6
+638333163393562386564343865383939386464623764346337663661356161333937
+6135316132">

      [download]
      and so I modified my program to stick a ':" in front of every field and Just as you said - -I tried it and it program *worked*. 
      'user-agent' => 'Mozilla/5.0',
':member_verbosity_threshold' => 0,
':csrf_token' => '28020000006916706f5b7347000000746573746c6973742d6661
+6e746173796661726d2e636f6d2b61646d696e3a63356234316237373565643432393
+036383931306131616635643239653666366463643239373065',
'content-type' => 'application/x-www-form-urlencoded',

      [download]
      Thanks! I'd have never found that info {since I had no idea what I should be looking for}.
[reply]
[d/l]
[select]

        Good, it works for you, but I am not convinced, your program works as intended. Consider:

        Token in HTTP-header (your current fix): 

        my $res = $ua->request(POST $url,
                       'user-agent'                  => 'Mozilla/5.0',
                       ':member_verbosity_threshold' => 0,
                       ':csrf_token'                 => '2802000000...
+',
#-- not required:      'content-type'                => 'application/x
+-www-form-urlencoded',
                      );

        [download]

        Result: 

        POST /xxx HTTP/1.1
TE: deflate,gzip;q=0.3
Connection: TE, close
Host: localhost:5000
User-Agent: Mozilla/5.0
Content-Length: 0
Content-Type: application/x-www-form-urlencoded
csrf_token: 28020000006.....
member_verbosity_threshold: 0

        [download]

        Token in HTTP-body (content) - this is more like what your <FORM ...> example would do: 

         my $res2 = $ua->request(POST $url,
                       'user-agent'                  => 'Mozilla/5.0',
                       ':member_verbosity_threshold' => 0,
#-- not in header:     ':csrf_token'                 => '28020000006..
+...',
                       Content                       => [ csrf_token =
+> '28020000006.....', ]  #-- but in body
                      );

        [download]

        Result: 

        POST /xxx HTTP/1.1
TE: deflate,gzip;q=0.3
Connection: TE, close
Host: localhost:5000
User-Agent: Mozilla/5.0
Content-Length: 27
Content-Type: application/x-www-form-urlencoded
member_verbosity_threshold: 0

csrf_token=28020000006.....

        [download]

        Hope, that shed some light. I was also going to nitpick, that you should prepend your non-standard headers with 'X-' (like X-Csrf-Token), but that practise seems discouraged since a couple of years: SO: Custom HTTP headers : naming conventions. Thanks, I learned something new ;-)

[reply]
[d/l]
[select]

Back to Seekers of Perl Wisdom