Humble greetings, Monks of the Perl.

After silently reading and learning for a couple of weeks I could use some help, where google and such couldn't help me. I hope that someone can give me a nudge in the right direction.


What I am trying to do:
---
My perl-script is supposed to upload some large files (around 100-500MiB) parallel to different services (the code is of course more compact for simplicity). This I have to do via HTTP-POST.
I noticed that the files uploaded via WWW:Mechanize or LWP::UserAgent are loaded into memory completely while they are being sent to the server.
So to circumvent this I tried setting "$HTTP::Request::Common::DYNAMIC_FILE_UPLOAD = 1" and it worked.
But with what speed loss: Before the changed variable 5+mb/s, after ~100kb/s. Then I tried to find how/where I could set the used buffer-/chunksize ... nowhere :-|

So finally I abandoned all these nice modules and sat down with IO::Socket (sidenote: the speed is ideal).


My problem:
---
On every request I try I get a "400 Bad Request" error. I assume the calculated Content-Length is wrong, but I am clueless as to why.
I have even tried a "brute force attack" with varying Content-Lengths to find a correct length to go from there (^^)... But no luck :-|
How can I do it right? Or better: What am I doing wrong? ;>


My code:
(you might need to remove line 19 as I have to operate under windows)
#!/usr/bin/perl
use warnings;
use strict;
use Socket;

my $buffersize = 2 * 1024 * 1024;

my $host = "127.0.0.1";
my $path = "/debug.php";
my $url  = "http://".$host.$path;
my $local_file = "smallishtestfile.tar.gz";

my $user = "cardman";
my $pass = "yvan eht nioj";
my $serverid = 666;

my $upfile = shift || 'F:\\skript\\'.$local_file;

system("cls");
print STDOUT "Starting upload to ".$url."\n\n";

$| = 1;

my ($iaddr, $paddr, $proto);
$iaddr = inet_aton($host);
$paddr = sockaddr_in(80, $iaddr);
$proto = getprotobyname('tcp');
unless(socket(SOCK, PF_INET, SOCK_STREAM, $proto)) { die "Couldn't ini
+t socket: $!"; }
unless(connect(SOCK, $paddr)) { die "Couldn't connect: $!\n"; }

my $boundary = "zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz";

my @data = (
    "--".$boundary."",
    "Content-Disposition: form-data; name=\"username\"",
    "",
    "".$user."",
    "--".$boundary."",
    "Content-Disposition: form-data; name=\"password\"",
    "",
    "".$pass."",
    "--".$boundary."",
    "Content-Disposition: form-data; name=\"serverid\"",
    "",
    "".$serverid."",
    "--".$boundary."",
    "Content-Disposition: form-data; name=\"file\"; filename=\"".$loca
+l_file."\"",
    "Content-Type: application/octet-stream",
    "",
);

open (FH,"< $upfile") or die "$!\n";
binmode FH;

my $data = join("\r\n", @data);
my $length = 0;
$length += length($data);        # length of the data to be POST'ed
$length += -s FH;                # filesize
$length += length($boundary);    # boundary is added once more at the 
+end of all the file-chunks

my @head = (
    "POST ".$path." HTTP/1.1",
    "Host: ".$host."",
    "Content-Length: $length",
    "Connection: keep-alive",
    "Content-Type: multipart/form-data; boundary=".$boundary."",
    "",
);
my $header = join("\r\n", @head).$data;

select SOCK;
$| = 1;
binmode SOCK;

print SOCK $header;

while(sysread(FH, my $buf, $buffersize))
{
    if(length($buf) < $buffersize) {
        $buf = $buf."\r\n--".$boundary."--";
        syswrite SOCK, $buf, length($buf);
     } else {
        syswrite SOCK, $buf, $buffersize;
    }
 }
close FH;

my @response = (<SOCK>);
shutdown SOCK, 1;

print STDOUT "Result:\n-------\n @response";
close SOCK;

[download]


Error message:
HTTP/1.1 400 Bad Request
Date: Tue, 21 Jun 2011 21:01:32 GMT
Server: Apache
Content-Length: 408
Connection: close
Content-Type: text/html; charset=iso-8859-1

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>400 Bad Request</title>
</head><body>
<h1>Bad Request</h1>
<p>Your browser sent a request that this server could not understand.<br />
Request header field is missing ':' separator.<br />
<pre>
--zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz</pre>
</p>
<hr>
<address>Apache Server at localhost Port 80</address>
</body></html>


Thanks in advance,
Kay

In reply to HTTP-POST with IO::Socket -- Header problem by Beavis

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.