Seeking help with HTTPS form upload.

FeistyLemur has asked for the wisdom of the Perl Monks concerning the following question:

>Greetings, I am attempting to reconstruct a post (using WWW::Mechanize and LWP::UserAgent so far) in order to upload a file and navigating some things I'm unfamiliar with. All my reading into examples has led me to the following code. I've truncated to the relevant section. I am having no issues with my first request to Authenticate and populate my $auth_key value with a previous request.

#!/usr/bin/perl -w
use strict;
use IO::Socket::SSL qw();
use LWP::UserAgent;

my $url_fw_upload='https://xxx.xxx.xxx.xxx/local/fwLoad?X-Progress-ID=
+xxxxxxxxxxxxx';

my $ua=LWP::UserAgent->new(ssl_opts => {
    SSL_verify_mode => IO::Socket::SSL::SSL_VERIFY_NONE,
    verify_hostname => 0,
}); 
$response=$ua->post($url_fw_upload,
    Authorization=>"Bearer $auth_key",
    Content_Type=>'form-data',
    Content => [
        "file" =>['firmware.img' => '/home/user/firmware.img'],
    ],
);
[download]

So the thing which is new to me is "fwLoad?X-Progress-ID", which is a Javascript generated UUID for file upload progress. I've used the same UUID that I grabbed from a browser debug session, within that browser debug I also have after the Request headers and before the Request Payload "Query String Parameters" with "X-Progress-ID: xxxxxxxxxxxxx". Since I have no Javascript capability here I'm trying to get around this and not sure if I even can without full Javascript functionality.

The results are always the same, 400 bad request. Any suggestions to point me in the right direction would be welcome as I've come up blank in my searches so far.

Comment on Seeking help with HTTPS form upload. Download Code

Replies are listed 'Best First'.
Re: Seeking help with HTTPS form upload. by hippo (Archbishop) on Oct 13, 2021 at 09:06 UTC
All my reading into examples has led me to the following code. `my $ua=LWP::UserAgent->new(ssl_opts => { SSL_verify_mode => IO::Socket::SSL::SSL_VERIFY_NONE, verify_hostname => 0, });` [download] No doubt there is a lot of publicly-visible code which encourages the disablement of such verification but that does not mean it is not really bad practice. Far better to pass no args to `new` at all or else explicitly switch on all the verification instead. 🦛	[reply] [d/l] [select]
Re^2: Seeking help with HTTPS form upload. by FeistyLemur (Acolyte) on Oct 13, 2021 at 16:05 UTC
The ssl is broken on the device. It's out of necessity. When using a browser you have to use the this website is not safe advanced box to gain access.	[reply]
Re^3: Seeking help with HTTPS form upload. by hippo (Archbishop) on Oct 13, 2021 at 23:05 UTC
9 times out of 10 all that is required to fix this is to use the right CA in the client. Disabling all verification should be the last resort, not the first. YMMV, of course. 🦛	[reply]
Re^4: Seeking help with HTTPS form upload. by FeistyLemur (Acolyte) on Oct 14, 2021 at 18:42 UTC
Re: Seeking help with HTTPS form upload. by Corion (Patriarch) on Oct 13, 2021 at 07:26 UTC
I would look for mentions of the UUID in the HTTP requests made before the progress request. Maybe it is client generated , then you can make up your own stuff, but maybe the server sends a UUID and expects it back. If all else fails, you can use something like my module WWW::Mechanize::Chrome to get a user agent that runs Javascript, if only to dump the incoming and outgoing requests to search for the origin of the UUID.	[reply]
Re: Seeking help with HTTPS form upload. by NERDVANA (Priest) on Oct 13, 2021 at 02:29 UTC
A common problem when uploading files is to show the user feedback about the progress of the upload. The browsers don’t have this capability, so people usually resort to running ajax requests in parallel to ask the server on a second connection how much data was received so far on the first connection. In order to ask about he first connection, there needs to be some kind of ID to reference it. Since you say javascript generated a UUID, they probably implemented this by just accepting whatever ID the client provides on the upload, then the other progress checks can refer to that. Their system might depend on the string being unique. It’s pretty easy to generate a random string in Perl, so just make your own UUID that looks like the one JavaScript builds.	[reply]
Re^2: Seeking help with HTTPS form upload. by FeistyLemur (Acolyte) on Oct 13, 2021 at 16:42 UTC
As it turns out you're correct, the string numbers are unimportant to the server. I got past this step today. As it turns out the content type was all I overlooked and needed to be set along with the file. It was defaulting to application/octet-stream and needed to be application/x-raw-disk-image.	[reply]
Re^2: Seeking help with HTTPS form upload. by Bod (Parson) on Oct 13, 2021 at 21:07 UTC
people usually resort to running ajax requests in parallel to ask the server on a second connection how much data was received so far on the first connection An alternative approach that I prefer is to use an AJAX request to perform the upload. Using `XMLHttpRequest` an event handler can be added to the request object to then show the progress something like this: `xhr.upload.addEventListener('progress', function(ev) { document.getElementById('progbar').value = ev.loaded * 100 / ev.to +tal; });` [download] Where `progbar` is an HTML `progress` element.	[reply] [d/l] [select]