Runs from Command Line but NOT from Perl

Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

I have this code that prints out a command and also runs it from Perl. When I run the curl command from Perl the remote server, queried from the curl request, complains about an HTTP 400 error.

When I run the same command, copied from the Perl print statement, it runs fine and returns a JSON string.

This behavior happens on Linux and Windows with Strawberry Perl. Any ideas?

Here is the code:

#!/usr/bin/perl

use strict;
use warnings;

use Data::Dumper;

my $command = <<'END';
curl 'https://tax-calculator-api.taxfoundation.org/taxcalc/tcja_submit
+' -H 'Origin: https://taxfoundation.org' -H 'Accept-Encoding: gzip, d
+eflate, br' -H 'Accept-Language: en-US,en;q=0.9' -H 'User-Agent: Mozi
+lla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like
+ Gecko) Chrome/69.0.3497.100 Safari/537.36' -H 'content-type: applica
+tion/json' -H 'Accept: */*' -H 'Referer: https://taxfoundation.org/20
+18-tax-reform-calculator/' -H 'Connection: keep-alive' --data-binary 
+$'{"filing_status": 1, "child_dep": 2, "nonchild_dep": 0, "ordinary_i
+ncome1": 150000, "ordinary_income2": 0, "business_income": 0, "ss_inc
+ome": 0, "qualified_income": 0, "401k_contributions": 0, "medical_exp
+enses": 0, "sl_income_tax": 0, "sl_property_tax": 0, "interest_paid":
+ 0, "charity_contributions": 0, "other_itemized": 0}' --compressed
END

print $command;

my $data = `$command` or die "$!\n";
print Dumper $data;
[download]

Comment on Runs from Command Line but NOT from Perl Download Code

Replies are listed 'Best First'.
Re: Runs from Command Line but NOT from Perl by LanX (Saint) on Oct 18, 2018 at 15:44 UTC
why don't you use LWP instead of shelling out to curl? Cheers Rolf _{(addicted to the Perl Programming Language :) Wikisyntax for the Monastery FootballPerl is like chess, only without the dice}	[reply]
Re^2: Runs from Command Line but NOT from Perl by Your Mother (Archbishop) on Oct 18, 2018 at 17:31 UTC
Tangent: Has anyone ever written a `curl2lwp` kind of thing in the vein of `s2p` and `a2p`? It would obviously be more verbose, especially in output, but I just realized that I'd like to have it so I can copy `curl` commands from the dev panel and such and use them to stub out a new LPW/Mech script. Update: `curl2perl` really rolls off the tongue…	[reply] [d/l] [select]
Re^3: Runs from Command Line but NOT from Perl by Corion (Patriarch) on Nov 14, 2018 at 18:09 UTC
There now is HTTP::Request::FromCurl (curl2lwp - convert Curl command line arguments to LWP / Mechanize Perl code), which includes `curl2lwp.pl`, that converts curl command lines to Perl code. There also is an online Curl to Perl converter running that module as its backend.	[reply] [d/l]
Re^3: Runs from Command Line but NOT from Perl by Anonymous Monk on Oct 19, 2018 at 01:10 UTC
You can write curl2perl	[reply]
Re^4: Runs from Command Line but NOT from Perl by Your Mother (Archbishop) on Oct 19, 2018 at 01:40 UTC
Re^2: Runs from Command Line but NOT from Perl by Anonymous Monk on Oct 19, 2018 at 02:23 UTC
I did try with LWP but I get the same perplexing problem!!! I'm trying to go to https://taxfoundation.org/2018-tax-reform-calculator/ and use their tax calculator from Perl. I am TOTALLY bamboozakled on why this is failing! #!/usr/bin/perl use strict; use warnings; use HTTP::Request::Common qw( POST ); use LWP::UserAgent; my $ua = LWP::UserAgent->new; my $req = POST 'https://tax-calculator-api.taxfoundation.org/taxcalc/t +cja_submit', [ Host => 'tax-calculator-api.taxfoundation.org', Connection => 'keep-alive', Origin => 'https://taxfoundation.org', 'User-Agent' => 'Mozilla/5.0 (Windows NT 10.0; Win64; x64 +) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/ +537.36', 'content-type' => 'application/json', Accept => '/', Referer => 'https://taxfoundation.org/2018-tax-refor +m-calculator/', 'Accept-Encoding' => 'gzip, deflate, br', 'Accept-Language' => 'en-US,en;q=0.9', Content => [ '401k_contributions' => 0, business_income => 0, charity_contributions => 0, child_dep => 2, filing_status => 2, interest_paid => 0, medical_expenses => 0, nonchild_dep => 0, ordinary_income1 => 150000, ordinary_income2 => 0, other_itemized => 0, qualified_income => 0, sl_income_tax => 0, sl_property_tax => 0, ss_income => 0, ], ]; print $ua->request($req)->as_string; [download]	[reply] [d/l]
Re^3: Runs from Command Line but NOT from Perl by poj (Abbot) on Oct 19, 2018 at 06:24 UTC
why this is failing Because `content-type' => 'application/json'` you have to json encode the content #!/usr/bin/perl use strict; use warnings; use HTTP::Request::Common qw( POST ); use LWP::UserAgent; use Encode qw(encode_utf8); use JSON::MaybeXS qw(encode_json); my $ua = LWP::UserAgent->new; my $data = { '401k_contributions' => 0, business_income => 0, charity_contributions => 0, child_dep => 2, filing_status => 2, interest_paid => 0, medical_expenses => 0, nonchild_dep => 0, ordinary_income1 => 150000, ordinary_income2 => 0, other_itemized => 0, qualified_income => 0, sl_income_tax => 0, sl_property_tax => 0, ss_income => 0, }; my $content = encode_utf8(encode_json($data)); my $req = POST 'https://tax-calculator-api.taxfoundation.org/taxcalc/t +cja_submit', Host => 'tax-calculator-api.taxfoundation.org', Connection => 'keep-alive', Origin => 'https://taxfoundation.org', 'User-Agent' => 'Mozilla/5.0 (Windows NT 10.0; Win64; x64 +) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/ +537.36', 'content-type' => 'application/json', Accept => '/', Referer => 'https://taxfoundation.org/2018-tax-refor +m-calculator/', 'Accept-Encoding' => 'gzip, deflate, br', 'Accept-Language' => 'en-US,en;q=0.9', Content => $content ; #print $req->as_string; print $ua->request($req)->as_string; [download] poj	[reply] [d/l] [select]
Re^4: Runs from Command Line but NOT from Perl by Anonymous Monk on Oct 19, 2018 at 08:15 UTC
Re: Runs from Command Line but NOT from Perl by Anonymous Monk on Oct 18, 2018 at 15:39 UTC
`--data-binary $'{"filing_status": ...'` `$'...'` is a bashism: a feature of Bash_(Unix_shell) that POSIX does not require other shells to understand. Perl runs `/bin/sh` (the actual rules may be more complicated), which may not correspond to Bash (or may be Bash running in strict POSIX compliance mode), so the command fails. Solution: use the ordinary single-quoted literal `'{"filing_status": ...er_itemized": 0}'` for the JSON string argument. But that would only help you to run the command from a POSIX-compliant shell; making your command compatible with both `/bin/sh` and `CMD.EXE` is more of an arcane art I'm not proficient at.	[reply] [d/l] [select]
Re^2: Runs from Command Line but NOT from Perl by ikegami (Patriarch) on Oct 20, 2018 at 01:53 UTC
(the actual rules may be more complicated) It uses `perl -V:sh`, which someone could change to something other than `/bin/sh`, but that would be a very weird (i.e. inadvisable) thing to do on a system with `/bin/sh`.	[reply] [d/l] [select]
Re: Runs from Command Line but NOT from Perl by tybalt89 (Monsignor) on Oct 18, 2018 at 18:58 UTC
If it needs to run under bash then run it under bash: #!/usr/bin/perl # https://perlmonks.org/?node_id=1224244 use strict; use warnings; use Data::Dumper; my $command = 'date'; # simplified command for test purposes my $data = `bash -c $command` or die"$!\n"; print Dumper $data; [download]	[reply] [d/l]