[SOLVED] Basic HTTP problems

Bolemo has asked for the wisdom of the Perl Monks concerning the following question:

Greetings fellow monks,

I am unable to reproduce a very simple Python script into Perl. Basically, it just do a few HTTP requests. The Python script is successful and gets the data needed. The Perl script fails (no errors, but the HTTP sent does not convince the server; I do get the sid fine.). I compared the tcpdump produced by both scripts, and they are different (8 packets total for Python and 18 for Perl).

Python script:

 #!/usr/bin/env python3
import requests

# User vars
ont_host = "192.168.18.1"
ont_username = USERNAME
ont_password = BASE64PASS

# Build URL prefix
ont_urlprefix = "http://" + ont_host

# Start http session
ont_session = requests.Session()
# Get sid
req = ont_session.post(ont_urlprefix + "/asp/GetRandCount.asp")
sid = req.text[-32:]
# Session authentication
auth_data = {"UserName": ont_username, "PassWord": ont_password, "x.X_
+HW_Token": sid}
req = ont_session.post(ont_urlprefix + "/login.cgi", data=auth_data)
# Retrieve info
req = ont_session.get(ont_urlprefix + "/html/amp/opticinfo/opticinfo.a
+sp")
print(req.text)
# Logout
ont_session.post(ont_urlprefix + "/logout.cgi?RequestFile=html/logout.
+html")

exit(0)
[download]

My Perl script attempt:

 #!/usr/bin/perl
use strict;
use warnings;
use LWP::UserAgent;
use HTTP::Request::Common;

# User vars
my $ont_host = "192.168.18.1";
my $ont_username = USERNAME;
my $ont_password = BASE64PASS;

# Build URL prefix
my $url = "http://" . $ont_host;

# Start http session
my $ua = LWP::UserAgent->new(keep_alive => 1);
# Get sid
my $res = $ua->post($url . '/asp/GetRandCount.asp');
my $sid = $res->content;
# Session authentication
$res = $ua->post($url . '/login.cgi', [UserName => "$ont_username", Pa
+ssWord => "$ont_password", 'x.X_HW_Token' => "$sid"]);
# Retrieve info
$res = $ua->get($url . '/html/amp/opticinfo/opticinfo.asp');
print($res->content);
$res = $ua->get($url . '/logout.cgi?RequestFile=html/logout.html');
[download]

Python script tcpdump:

 root@HERMES:~$ tcpdump -ttnnvvS 'tcp and net 192.168.18.0/24'
tcpdump: listening on ethwan, link-type EN10MB (Ethernet), capture siz
+e 262144 bytes
1612530239.807502 IP (tos 0x0, ttl 63, id 6338, offset 0, flags [DF], 
+proto TCP (6), length 52)
    192.168.18.2.54798 > 192.168.18.1.80: Flags [S], cksum 0xb418 (cor
+rect), seq 4024652913, win 64240, options [mss 1460,nop,nop,sackOK,no
+p,wscale 7], length 0
1612530239.808064 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], pro
+to TCP (6), length 52)
    192.168.18.1.80 > 192.168.18.2.54798: Flags [S.], cksum 0x90f7 (co
+rrect), seq 1577355000, ack 4024652914, win 14600, options [mss 1460,
+nop,nop,sackOK,nop,wscale 3], length 0
1612530239.808814 IP (tos 0x0, ttl 63, id 6339, offset 0, flags [DF], 
+proto TCP (6), length 40)
    192.168.18.2.54798 > 192.168.18.1.80: Flags [.], cksum 0x08d8 (cor
+rect), seq 4024652914, ack 1577355001, win 502, length 0
1612530239.808876 IP (tos 0x0, ttl 63, id 6340, offset 0, flags [DF], 
+proto TCP (6), length 223)
    192.168.18.2.54798 > 192.168.18.1.80: Flags [P.], cksum 0x7f17 (co
+rrect), seq 4024652914:4024653097, ack 1577355001, win 502, length 18
+3: HTTP, length: 183
        POST /asp/GetRandCount.asp HTTP/1.1
        Host: 192.168.18.1
        User-Agent: python-requests/2.25.1
        Accept-Encoding: gzip, deflate
        Accept: */*
        Connection: keep-alive
        Content-Length: 0
        
1612530240.359236 IP (tos 0x0, ttl 64, id 26662, offset 0, flags [DF],
+ proto TCP (6), length 522)
    192.168.18.1.80 > 192.168.18.2.54798: Flags [FP.], cksum 0xa1b0 (c
+orrect), seq 1577390430:1577390912, ack 4024653989, win 2361, length 
+482: HTTP
1612530240.361735 IP (tos 0x0, ttl 63, id 6369, offset 0, flags [DF], 
+proto TCP (6), length 40)
    192.168.18.2.54798 > 192.168.18.1.80: Flags [.], cksum 0x785d (cor
+rect), seq 4024653989, ack 1577390913, win 501, length 0
1612530240.390570 IP (tos 0x0, ttl 63, id 6370, offset 0, flags [DF], 
+proto TCP (6), length 40)
    192.168.18.2.54798 > 192.168.18.1.80: Flags [F.], cksum 0x785c (co
+rrect), seq 4024653989, ack 1577390913, win 501, length 0
1612530240.391007 IP (tos 0x0, ttl 64, id 21626, offset 0, flags [DF],
+ proto TCP (6), length 40)
    192.168.18.1.80 > 192.168.18.2.54798: Flags [.], cksum 0x7118 (cor
+rect), seq 1577390913, ack 4024653990, win 2361, length 0
^C
8 packets captured
8 packets received by filter
0 packets dropped by kernel
[download]

Perl script tcpdump;

 root@HERMES:~$ tcpdump -ttnnvvS 'tcp and net 192.168.18.0/24'
tcpdump: listening on ethwan, link-type EN10MB (Ethernet), capture siz
+e 262144 bytes
1612532228.397661 IP (tos 0x0, ttl 63, id 13929, offset 0, flags [DF],
+ proto TCP (6), length 52)
    192.168.18.2.54818 > 192.168.18.1.80: Flags [S], cksum 0xe5dc (cor
+rect), seq 3899533838, win 64240, options [mss 1460,nop,nop,sackOK,no
+p,wscale 7], length 0
1612532228.398192 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], pro
+to TCP (6), length 52)
    192.168.18.1.80 > 192.168.18.2.54818: Flags [S.], cksum 0x6b28 (co
+rrect), seq 3067119039, ack 3899533839, win 14600, options [mss 1460,
+nop,nop,sackOK,nop,wscale 3], length 0
1612532228.398911 IP (tos 0x0, ttl 63, id 13930, offset 0, flags [DF],
+ proto TCP (6), length 40)
    192.168.18.2.54818 > 192.168.18.1.80: Flags [.], cksum 0xe308 (cor
+rect), seq 3899533839, ack 3067119040, win 502, length 0
1612532228.591851 IP (tos 0x0, ttl 64, id 27663, offset 0, flags [DF],
+ proto TCP (6), length 507)
    192.168.18.1.80 > 192.168.18.2.54818: Flags [FP.], cksum 0x52f1 (c
+orrect), seq 3067119452:3067119919, ack 3899534340, win 2093, length 
+467: HTTP
1612532228.596974 IP (tos 0x0, ttl 63, id 13936, offset 0, flags [DF],
+ proto TCP (6), length 40)
    192.168.18.2.54818 > 192.168.18.1.80: Flags [F.], cksum 0xdda3 (co
+rrect), seq 3899534340, ack 3067119920, win 501, length 0
1612532228.597411 IP (tos 0x0, ttl 64, id 27664, offset 0, flags [DF],
+ proto TCP (6), length 40)
    192.168.18.1.80 > 192.168.18.2.54818: Flags [.], cksum 0xd76b (cor
+rect), seq 3067119920, ack 3899534341, win 2093, length 0
1612532228.598786 IP (tos 0x0, ttl 63, id 64322, offset 0, flags [DF],
+ proto TCP (6), length 52)
    192.168.18.2.54820 > 192.168.18.1.80: Flags [S], cksum 0x7b93 (cor
+rect), seq 4176643025, win 64240, options [mss 1460,nop,nop,sackOK,no
+p,wscale 7], length 0
1612532228.599223 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], pro
+to TCP (6), length 52)
    192.168.18.1.80 > 192.168.18.2.54820: Flags [S.], cksum 0x7611 (co
+rrect), seq 274904827, ack 4176643026, win 14600, options [mss 1460,n
+op,nop,sackOK,nop,wscale 3], length 0
1612532228.599942 IP (tos 0x0, ttl 63, id 64323, offset 0, flags [DF],
+ proto TCP (6), length 40)
    192.168.18.2.54820 > 192.168.18.1.80: Flags [.], cksum 0xedf1 (cor
+rect), seq 4176643026, ack 274904828, win 502, length 0
1612532228.604128 IP (tos 0x0, ttl 64, id 24356, offset 0, flags [DF],
+ proto TCP (6), length 507)
    192.168.18.1.80 > 192.168.18.2.54820: Flags [FP.], cksum 0x6117 (c
+orrect), seq 274905011:274905478, ack 4176643195, win 1825, length 46
+7: HTTP
1612532228.604628 IP (tos 0x0, ttl 64, id 24357, offset 0, flags [DF],
+ proto TCP (6), length 40)
    192.168.18.1.80 > 192.168.18.2.54820: Flags [R.], cksum 0xe58e (co
+rrect), seq 274905479, ack 4176643195, win 1825, length 0
1612532228.611532 IP (tos 0x0, ttl 63, id 44109, offset 0, flags [DF],
+ proto TCP (6), length 52)
    192.168.18.2.54822 > 192.168.18.1.80: Flags [S], cksum 0x3645 (cor
+rect), seq 3432379770, win 64240, options [mss 1460,nop,nop,sackOK,no
+p,wscale 7], length 0
1612532228.612001 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], pro
+to TCP (6), length 52)
    192.168.18.1.80 > 192.168.18.2.54822: Flags [S.], cksum 0xd064 (co
+rrect), seq 2196415697, ack 3432379771, win 14600, options [mss 1460,
+nop,nop,sackOK,nop,wscale 3], length 0
1612532228.612563 IP (tos 0x0, ttl 63, id 44110, offset 0, flags [DF],
+ proto TCP (6), length 40)
    192.168.18.2.54822 > 192.168.18.1.80: Flags [.], cksum 0x4845 (cor
+rect), seq 3432379771, ack 2196415698, win 502, length 0
1612532228.618717 IP (tos 0x0, ttl 64, id 19728, offset 0, flags [DF],
+ proto TCP (6), length 40)
    192.168.18.1.80 > 192.168.18.2.54822: Flags [F.], cksum 0x3f59 (co
+rrect), seq 2196416348, ack 3432379947, win 1959, length 0
1612532228.618842 IP (tos 0x0, ttl 63, id 44113, offset 0, flags [DF],
+ proto TCP (6), length 40)
    192.168.18.2.54822 > 192.168.18.1.80: Flags [.], cksum 0x450c (cor
+rect), seq 3432379947, ack 2196416348, win 501, length 0
1612532228.618873 IP (tos 0x0, ttl 64, id 19729, offset 0, flags [DF],
+ proto TCP (6), length 40)
    192.168.18.1.80 > 192.168.18.2.54822: Flags [R.], cksum 0x3f55 (co
+rrect), seq 2196416349, ack 3432379947, win 1959, length 0
1612532228.619373 IP (tos 0x0, ttl 64, id 54882, offset 0, flags [DF],
+ proto TCP (6), length 40)
    192.168.18.1.80 > 192.168.18.2.54822: Flags [R], cksum 0x09cf (cor
+rect), seq 2196416348, win 0, length 0
^C
18 packets captured
18 packets received by filter
0 packets dropped by kernel
[download]

Any ideas on what I am doing wrong?

Comment on [SOLVED] Basic HTTP problems Select or Download Code

Replies are listed 'Best First'.
Re: Basic HTTP problems by marto (Cardinal) on Feb 05, 2021 at 15:32 UTC
`my $sid = $res->content;` [download] Probably isn't the same as `sid = req.text[-32:]` [download] Do some basic debugging to ensure you're sending the same thing. References: https://github.com/CutJiuCai/hg8347r/blob/master/hg8347r.go#L37 https://www.slideshare.net/rfc1036/exploring-the-huawei-hg8010h-gpon-ont	[reply] [d/l] [select]
Re^2: Basic HTTP problems by Bolemo (Acolyte) on Feb 05, 2021 at 17:03 UTC
Actually, you are right: there are 3 invisible control characters before the 32 sid string. So substring 32 is indeed different, However, that does not solve the issue...	[reply]
Re: Basic HTTP problems by hippo (Archbishop) on Feb 05, 2021 at 14:06 UTC
`sid = req.text[-32:]` I don't speak python so don't know what this is doing but as there's no reference to "32" anywhere in your Perl code perhaps it is something which needs to be addressed. See the Basic Debugging Checklist for more info. Print data sent before every request (particularly that auth - seems an obvious place for something to go wrong) and `$res->code` for every response until you nail it down. 🦛	[reply] [d/l] [select]
Re^2: Basic HTTP problems by Fletch (Bishop) on Feb 05, 2021 at 14:16 UTC
I believe that's a python-y array slice pulling off the last 32 characters of the (in this case) string so OP'd need something like `$sid = substr( $req->content, -32 )` to get the equivalent value (although blindly pulling out the last 32 characters rather than actually parsing the returned value may be living dangerously regardless of the implementation language . . .). The cake is a lie. The cake is a lie. The cake is a lie.	[reply] [d/l]
Re^2: Basic HTTP problems by choroba (Cardinal) on Feb 05, 2021 at 14:16 UTC
It corresponds to `my $sid = substr $res->content, -32;` [download] `map{substr$_->[0],$_->[1]\|\|0,1}[\\|\|{},3],[[]],[ref qr-1,-,-1],[{}],[sub{}^ARGV,3]`	[reply] [d/l] [select]
Re: Basic HTTP problems by Bolemo (Acolyte) on Feb 05, 2021 at 14:42 UTC
The sid returned is a 32 characters long string. I believe that Python might add a prefix so the programmer made sure only the returned 32 chars are taken. Anyway, with or without substring in Perl, the sid string is exactly the same (len of 32), and the result is the same. The issue is not there. $res->code is 200 for each request. There is no HTTP error. With the Python script, the server is sending the right information. With the Perl script, the server is acting like it is not authenticated. The problem seems to rely more on what an how $ua->post is actually sending the data. The tcpdump shows it is different, where it should be the same. Also, maybe each request is not considered in the same HTTP session in Perl?	[reply]
Re^2: Basic HTTP problems by Corion (Patriarch) on Feb 05, 2021 at 14:54 UTC
Consider inspecting the actual data that goes over the wire. The `tcpdump` output does not show the data. You want to print the request from both programs and compare that. Either create the POST request separately and dump it (using HTTP::Request::Common maybe): `print $req->as_string;` [download] ... or add dumping of sent and received data: `$ua->add_handler("request_send", sub { shift->dump; return }); $ua->add_handler("response_done", sub { shift->dump; return });` [download] I'm sure that Python also has ways to show the data that goes out and comes back in.	[reply] [d/l] [select]
Re^3: Basic HTTP problems by Bolemo (Acolyte) on Feb 05, 2021 at 17:10 UTC
The handlers are so precious advice, thank you! It allowed me to discover that the sid has hidden control chars before the 32 chars, so I fixed that. BUT that does not fix the issue. However after sending the first sid for authentication, the server is sending a cookie with another sid, so I need to probably deal with that. Also, I need to do the same handler trick to the python code, and that will definitively help to solve the problem. EDIT: it was the cookies. Adding a cookie jar did the trick! Thank you!	[reply]
Re^3: Basic HTTP problems by Anonymous Monk on Feb 05, 2021 at 19:27 UTC
When troubleshooting issues like this, always look at server logs also. Looks like this was an authentication-related issue due to the absence of cookies, and the server will have logged it as such. You really can't tell much from TCPDUMP.	[reply]
Re^4: Basic HTTP problems by Bolemo (Acolyte) on Feb 05, 2021 at 19:45 UTC
Re^5: Basic HTTP problems by Fletch (Bishop) on Feb 05, 2021 at 20:32 UTC