Rest::Client Headers not getting through

Random_Walk has asked for the wisdom of the Perl Monks concerning the following question:

Oh wise and splendiferous monks of the Perly persuasion

I am trying to use REST::Client to update a wiki from some text files. Although I can connect and read data, when I try to POST a page edit, it complains I have not sent a 'token'. The token is meant to go in the headers, the API docco https://www.mediawiki.org/wiki/API:Edit shows it as part of the URL such as

/mediawiki/api.php?action=edit&title=Tst&summary=Tsummary&text=article
+%20content&token=81750dcca64dc
[download]

Note:Token above cut short to stop line wrapping on my laptop screen

Now I have added the token in my URI such as above, and got the error:

The 'token' parameter was found in the query string, but must be in th
+e POST body
[download]

I have tried adding the token as a custom header in the code: $client->addHeader('token', $token); then I get this error:

The token parameter must be set
[download]

Here is my code with an attempt to add token as a custom header

use strict;
use warnings;
use v5.10.0;
use Data::Dumper;
use MIME::Base64;
use JSON;
use REST::Client;
use Getopt::Long;

my $debug = 0;
GetOptions ("debug"  => \$debug) or die "Error in command line argumen
+ts\n";

my $wiki = 'http://HeartOfGold.improbable.tea';
my $api  = '/mediawiki/api.php';
my $request;

my $JSON = JSON->new->allow_nonref;

# Create rest client and set some options
my $client = REST::Client->new();
$client->setHost($wiki);
$client->addHeader('format', 'json');

# First we get a login token
$request   = 'action=login&lgname=Marvin&lgpassword=leftdiode';
my $result = post($request);
my $token  = $result->{login}{token};
print "Got Token: $token\n";

$client->addHeader('token', $token);
$request = "action=edit&title=LookupAutomationTest&summary=test%20summ
+ary&text=article%20content";
$result = post($request);

sub post {
    my $query = shift @_;
    $query .= '&format=json';
    my $req = "$api?$query";
    print "Fetching data with: POST $req\n" if $debug;
    print Dumper $client;
    $client->POST($req);

    if ( $client->responseCode() == 200 ) {
        print "Got Data OK, decoding ... " if $debug;
        my $data = $JSON->decode( $client->responseContent() );
        print Dumper $data;
        return $data;
    }
    else {
        print "Failed with code $client->responseCode()\n";
        print Dumper $client->responseContent();
    }
}
[download]

The first call to log in and get the token works fine.
The second call to edit the page gives the missing token type errors.
As you see the code dumps the client just before posting, so here is the output. One can see the headers are added in the _headers key of the client object but Wiki is ignoring them. I had a similar problem with Service Now a short time back, but I just &stuffed=them in the URI and it was happy then.

rwalk@HeartOfGold~> ./LookupsToWiki.pl -d
Fetching data with: POST /mediawiki/api.php?action=login&lgname=Marvin
+&lgpassword=leftdiodes&format=json
$VAR1 = bless( {
                 '_config' => {
                                'host' => 'http://HeartOfGold.improbab
+le.tea',
                                'useragent' => bless( {
                                                        'protocols_all
+owed' => undef,
                                                        'no_proxy' => 
+[],
                                                        'ssl_opts' => 
+{
                                                                      
+  'verify_hostname' => 1
                                                                      
+},
                                                        'max_redirect'
+ => 7,
                                                        'requests_redi
+rectable' => [
                                                                      
+               'GET',
                                                                      
+               'HEAD'
                                                                      
+             ],
                                                        'handlers' => 
+{
                                                                      
+  'response_header' => bless( [
                                                                      
+                                {
                                                                      
+                                  'owner' => 'LWP::UserAgent::parse_h
+ead',
                                                                      
+                                  'line' => '/usr/lib/perl5/vendor_pe
+rl/5.18.2/LWP/UserAgent.pm:683',
                                                                      
+                                  'm_media_type' => 'html',
                                                                      
+                                  'callback' => sub { "DUMMY" }
                                                                      
+                                }
                                                                      
+                              ], 'HTTP::Config' )
                                                                      
+},
                                                        'local_address
+' => undef,
                                                        'proxy' => {},
                                                        'show_progress
+' => undef,
                                                        'use_eval' => 
+1,
                                                        'timeout' => 1
+80,
                                                        'max_size' => 
+undef,
                                                        'def_headers' 
+=> bless( {
                                                                      
+            'user-agent' => 'REST::Client/273'
                                                                      
+          }, 'HTTP::Headers' ),
                                                        'protocols_for
+bidden' => undef
                                                      }, 'LWP::UserAge
+nt' )
                              },
                 '_headers' => {
                                 'format' => 'json'
                               }
               }, 'REST::Client' );
Got Data OK, decoding ... $VAR1 = {
          'login' => {
                       'cookieprefix' => 'bitnami_mediawiki',
                       'sessionid' => 'ltmfbpjfgj5il28d9k5105h8kc2a74d
+b',
                       'result' => 'NeedToken',
                       'token' => 'bfbd1a1bdf1d1e0167e07ccdd6bdd4675af
+df46d+\\'
                     },
          'warnings' => {
                          'login' => {
                                       '*' => 'Fetching a token via ac
+tion=login is deprecated. Use action=query&meta=tokens&type=login ins
+tead.'
                                     }
                        }
        };
Got Token: bfbd1a1bdf1d1e0167e07ccdd6bdd4675afdf46d+\
Fetching data with: POST /mediawiki/api.php?action=edit&title=LookupAu
+tomationTest&summary=test%20summary&text=article%20content&format=jso
+n
$VAR1 = bless( {
                 '_config' => {
                                'host' => 'http://HeartOfGold.improbab
+le.tea',
                                'useragent' => bless( {
                                                        'protocols_all
+owed' => undef,
                                                        'no_proxy' => 
+[],
                                                        'ssl_opts' => 
+{
                                                                      
+  'verify_hostname' => 1
                                                                      
+},
                                                        'max_redirect'
+ => 7,
                                                        'requests_redi
+rectable' => [
                                                                      
+               'GET',
                                                                      
+               'HEAD'
                                                                      
+             ],
                                                        'handlers' => 
+{
                                                                      
+  'response_header' => bless( [
                                                                      
+                                {
                                                                      
+                                  'owner' => 'LWP::UserAgent::parse_h
+ead',
                                                                      
+                                  'line' => '/usr/lib/perl5/vendor_pe
+rl/5.18.2/LWP/UserAgent.pm:683',
                                                                      
+                                  'm_media_type' => 'html',
                                                                      
+                                  'callback' => sub { "DUMMY" }
                                                                      
+                                }
                                                                      
+                              ], 'HTTP::Config' )
                                                                      
+},
                                                        'local_address
+' => undef,
                                                        'proxy' => {},
                                                        'show_progress
+' => undef,
                                                        'use_eval' => 
+1,
                                                        'timeout' => 3
+00,
                                                        'max_size' => 
+undef,
                                                        'def_headers' 
+=> bless( {
                                                                      
+            'user-agent' => 'REST::Client/273'
                                                                      
+          }, 'HTTP::Headers' ),
                                                        'protocols_for
+bidden' => undef
                                                      }, 'LWP::UserAge
+nt' )
                              },
                 '_headers' => {
                                 'token' => 'bfbd1a1bdf1d1e0167e07ccdd
+6bdd4675afdf46d+\\',
                                 'format' => 'json'
                               },
                 '_res' => bless( {
                                    '_content' => '{"warnings":{"login
+":{"*":"Fetching a token via action=login is deprecated. Use action=q
+uery&meta=tokens&type=login instead."}},"login":{"result":"NeedToken"
+,"token":"bfbd1a1bdf1d1e0167e07ccdd6bdd4675afdf46d+\\\\","cookieprefi
+x":"bitnami_mediawiki","sessionid":"ltmfbpjfgj5il28d9k5105h8kc2a74db"
+}}',
                                    '_protocol' => 'HTTP/1.1',
                                    '_rc' => '200',
                                    '_msg' => 'OK',
                                    '_headers' => bless( {
                                                           'content-le
+ngth' => '295',
                                                           'cache-cont
+rol' => 'private, must-revalidate, max-age=0',
                                                           'x-frame-op
+tions' => 'DENY',
                                                           '::std_case
+' => {
                                                                      
+       'x-frame-options' => 'X-Frame-Options',
                                                                      
+       'client-response-num' => 'Client-Response-Num',
                                                                      
+       'client-date' => 'Client-Date',
                                                                      
+       'client-peer' => 'Client-Peer',
                                                                      
+       'x-powered-by' => 'X-Powered-By',
                                                                      
+       'x-content-type-options' => 'X-Content-Type-Options',
                                                                      
+       'set-cookie' => 'Set-Cookie'
                                                                      
+     },
                                                           'date' => '
+Thu, 17 May 2018 21:30:21 GMT',
                                                           'client-pee
+r' => '10.255.64.83:80',
                                                           'x-powered-
+by' => 'PHP/5.6.21',
                                                           'set-cookie
+' => 'bitnami_mediawiki_session=ltmfbpjfgj5il28d9k5105h8kc2a74db; pat
+h=/; httponly',
                                                           'connection
+' => 'close',
                                                           'x-content-
+type-options' => 'nosniff',
                                                           'server' =>
+ 'Apache/2.4.18 (Unix) OpenSSL/1.0.2h PHP/5.6.21 mod_perl/2.0.8-dev P
+erl/v5.16.3',
                                                           'content-ty
+pe' => 'application/json; charset=utf-8',
                                                           'client-dat
+e' => 'Thu, 17 May 2018 21:30:21 GMT',
                                                           'client-res
+ponse-num' => 1
                                                         }, 'HTTP::Hea
+ders' ),
                                    '_request' => bless( {
                                                           '_uri_canon
+ical' => bless( do{\(my $o = 'http://HeartOfGold.improbable.tea/media
+wiki/api.php?action=login&lgname=Marvin&lgpassword=leftdiodes&format=
+json')}, 'URI::http' ),
                                                           '_headers' 
+=> bless( {
                                                                      
+            '::std_case' => {
                                                                      
+                              'format' => 'Format'
                                                                      
+                            },
                                                                      
+            'user-agent' => 'REST::Client/273',
                                                                      
+            'content-length' => 0,
                                                                      
+            'format' => 'json'
                                                                      
+          }, 'HTTP::Headers' ),
                                                           '_content' 
+=> '',
                                                           '_method' =
+> 'POST',
                                                           '_uri' => $
+VAR1->{'_res'}{'_request'}{'_uri_canonical'}
                                                         }, 'HTTP::Req
+uest' )
                                  }, 'HTTP::Response' )
               }, 'REST::Client' );
Got Data OK, decoding ... $VAR1 = {
          'error' => {
                       'code' => 'notoken',
                       'info' => 'The token parameter must be set',
                       '*' => 'See http://HeartOfGold.improbable.tea/m
+ediawiki/api.php for API usage'
                     }
        };
[download]

Cheers,
R.

Pereant, qui ante nos nostra dixerunt!

Comment on Rest::Client Headers not getting through Select or Download Code

Replies are listed 'Best First'.
Re: Rest::Client Headers not getting through by bliako (Abbot) on May 18, 2018 at 09:28 UTC
Hi Random Walk, try to separate urlpath from POST params in the POST() call. The manpage says: `POST ( $url, [$body_content, %$headers] )` [download] [] meaning optional I suggest you do something like this and see how it goes (untested): `my $client = REST::Client->new(); $client->setHost($wiki); ... # got token # preparing a request: $request = "action=edit&title=LookupAutomationTest&summary=test%20summ +ary&text=article%20content"; $request .= '&format=json'; $request .= "&token=$token"; $client->POST($api, $request); # << separate path (e.g. "/mediawiki/ap +i.php") from the request-params e.g. the ? part which is now in strin +g $request.` [download] 'token' must be the last item in the params list according to the MediaWiki. there is a pass in your code!	[reply] [d/l] [select]
Re^2: Rest::Client Headers not getting through by Random_Walk (Prior) on May 18, 2018 at 22:14 UTC
Hi bliako, Thanks for taking the time to answer me. I tried your suggestion splitting the host and API path from the request and I also put the '&format=json' into each request string before calling post, so the "&token=$token" is then put last in the edit request. I'm afraid the result is a step back. Now I get an API help page in HTML from the first post. This I get at the first log attempt to get the token, which worked fine before. I updated the post sub like this sub post { my $query = shift @_; # my $req = "$api?$query"; # This removed print "Fetching data with: POST $api $query\n" if $debug; print Dumper $client; $client->POST($api, $query); if ( $client->responseCode() == 200 ) { print "Got Data OK, decoding ... " if $debug; print Dumper $client->responseContent() ; my $data = $JSON->decode( $client->responseContent() ); print Dumper $data; return $data; } else { print "Failed with code $client->responseCode()\n"; print Dumper $client->responseContent(); } } [download] If Is switch this line: `$client->POST($api, $query);` To This `$client->POST("$api?$query");` Then I get back to where I was before. I get the Token, but can't get wiki to accept it for the edit Cheers, R. Pereant, qui ante nos nostra dixerunt!	[reply] [d/l] [select]
Re^3: Rest::Client Headers not getting through by bliako (Abbot) on May 19, 2018 at 14:42 UTC
Hi Random Walk, The API you are using is pretty clear about this: The token parameter was found in the query string, but must be in the POST body . So, the token must go to the body. Which is by using POST($api, $body_content); I would add also the header "Content-type", "application/x-www-form-urlencoded" to become this: `POST($api, "...token=the_token". {"Content-type" => "application/x-www +-form-urlencoded"});` [download] Now the above is for whenever you are making a request involving a token. The 1st param is the api path, the 2nd param is the post-body as a query string (you can use `$client->buildQuery([a=>$a,b=>$b...])` for this) and the 3rd param is a hashref of headers. If the login phase worked somehow differently, then keep using that method and ... end up having two different post() - what can I say? Let me know if there is a public url to test this from my side.	[reply] [d/l] [select]
Re^4: Rest::Client Headers not getting through by Random_Walk (Prior) on May 21, 2018 at 15:32 UTC