in reply to HTTP-POST with IO::Socket -- Header problem

I already did simulate the POST-request into a file for debugging and forgot to tell you (yesterday I was very exhausted after 20 hours of coding ;-).

Instead of noticing this error with the missing "\r\n" I just noticed the different lengths:
Between the stated "Content-Length" and the real length of the content there were always some bytes discrepancy - the bigger the file, the bigger the difference in length.

So after I added the "\r\n" between header and body I used a static header with a manually calculated (correct) length. And it worked.

Then I got the idea, that the linebreaks in the script itself and/or the linebreaks in the file could be the problem.

So I converted all "\r\n" in the script into "\n" and printed everything to a file and the byte-discrepancies didn't change a bit.

Next I noticed that I missed to use "binmode" for my output file that I used for debugging.
Then I added "binmode" and SHAZZAM: The difference between the lengths still were some bytes, but for files with difference in size it always stayed a constant 4 bytes.


I'd still like to know if someone can figure out why there is this 4-byte-discrepancy.
But if noone knows/cares to share I can live with this dirty fix ;-)


At last, my working code:
#!/usr/bin/perl use warnings; use strict; use Socket; my $buffersize = 2 * 1024 * 1024; my $host = "127.0.0.1"; my $path = "/debug.php"; my $url = "http://".$host.$path; my $local_file = "testfile.rar"; my $local_path = "F:\\skript\\"; my $user = "cardman"; my $pass = "yvan eht nioj"; my $serverid = 666; my $upfile = shift || $local_path.$local_file; system("cls"); print STDOUT "Starting upload to ".$url."\r\n\r\n"; $| = 1; my ($iaddr, $paddr, $proto); $iaddr = inet_aton($host); $paddr = sockaddr_in(80, $iaddr); $proto = getprotobyname('tcp'); unless(socket(SOCK, PF_INET, SOCK_STREAM, $proto)) { die "Couldn't ini +t socket: $!"; } unless(connect(SOCK, $paddr)) { die "Couldn't connect: $!\r\n"; } my $boundary = "zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz"; my @data = ( "--".$boundary."", "Content-Disposition: form-data; name=\"username\"", "", "".$user."", "--".$boundary."", "Content-Disposition: form-data; name=\"password\"", "", "".$pass."", "--".$boundary."", "Content-Disposition: form-data; name=\"serverid\"", "", "".$serverid."", "--".$boundary."", "Content-Disposition: form-data; name=\"file\"; filename=\"".$loca +l_file."\"", "Content-Type: application/octet-stream", "", "", ); open (FILE,"< $upfile") or die "$!\n"; binmode FILE; my $data = join("\r\n", @data); my $length = 0; $length += length($data); # length of the data to be POST'ed $length += -s FILE; # filesize $length += length($boundary); # boundary is added once more at the +end of all the file-chunks $length += 4; # adding 4 bytes (no idea as to why, +but it works -- tested with 4 rng-files: 5 byte, 2mb, 15mb and 100mb) my @head = ( "POST ".$path." HTTP/1.1", "Host: ".$host."", "Content-Length: $length", "Connection: close", "Content-Type: multipart/form-data; boundary=".$boundary."", "", "", ); my $header = join("\r\n", @head).$data; # FOR DEBUGGING # open (FILE2,"< $upfile") or die "$!\n"; binmode FILE; # open(LOG, ">".$local_path."headers.txt"); binmode LOG; # print LOG $header; # while(sysread(FILE2, my $buf, 8)) { print LOG $buf; } # print LOG "\r\n--".$boundary."--"; # close LOG; close FILE2; select SOCK; $| = 1; binmode SOCK; print SOCK $header; while(sysread(FILE, my $buf, $buffersize)) { if(length($buf) < $buffersize) { $buf = $buf."\r\n--".$boundary."--"; syswrite SOCK, $buf, length($buf); } else { syswrite SOCK, $buf, $buffersize; } } close FILE; my @response = (<SOCK>); shutdown SOCK, 1; print STDOUT "Result:\n-------\n @response"; close SOCK;


Thanks to you both for helping me so quickly :-)
Kay

Replies are listed 'Best First'.
Re: SOLVED: HTTP-POST with IO::Socket -- Header problem
by Corion (Patriarch) on Jun 22, 2011 at 12:14 UTC
      No, because when I merge the header-array with the data via

      join("\r\n", @head).$data

      the $data variable has no "\r\n" at the beginning, so the header-array needs to have 2 line breaks so the paket as a whole looks like this:
      [...] Content-Type: multipart/form-data; boundary=".$boundary." --".$boundary." Content-Disposition: form-data; name=\"username\" [...]

      PS: If you had read my first post you would know that all these nice comfortable modules have the problem that they copy the file completely into RAM while uploading it.
      You can circumvent that by setting "$HTTP::Request::Common::DYNAMIC_FILE_UPLOAD = 1", but then the speed drops drastically - from 5mb/s to about 100kb/s. And you can't set the buffersize anywhere to increase that speed.

      Best regards,
      Kay

        I'm sorry that I didn't read your original post closely enough.

        It would seem to me that a very simple approach to testing where the bottleneck lies would be in patching/replacing the subroutine HTTP::Request::Common::form_data to read the data in chunks larger than 2048 bytes. Unfortunately, ->form_data is very large and monolithic and there is no easy way to change it other than copying it into your source code and replacing it:

        use HTTP::Request::Common; sub my_post_file { my $bufsize = 10_240_000; local *HTTP::Request::Common::form_data = sub { ... my $buflength = length $buf; my $n = read($fh, $buf, $bufsize, $buflength); if ($n) { $buflength += $n; unshift(@parts, ["", $fh]); } ... };

        If this change alone brings "enough" speedup, it might be worth to submit a patch back upstream that makes the POST buffer size configurable.