Baz has asked for the wisdom of the Perl Monks concerning the following question:

Ive writen a scipt that allows a client upload mp3s to my server, but Ive seen some browser upload done in the following fashion 
while (my $bytesread = read($filesource, my $buffer,1024)) 
{
  print OUTFILE $buffer;
}

[download]
My code is similar to the above code except for the 1024. What is its purpose?. Are the blocks transfered in 1KB packets - and if so that would be the best block size for uploading mp3s
Also, at the moment when the client selects a file and hits the upload button, the browser hangs until the file is fully transferred and only then does the next page display notifying the client that the transfer was successful - is there anyway of giving a progress report will a file is uploading - the status bar at the bottom of an IE4 window gives no reflexion of how much long the file will take to transfer.

Replies are listed 'Best First'.
Re: optimization of browser upload
by vladb (Vicar) on May 15, 2002 at 23:14 UTC
    Your question:

    except for the 1024. What is its purpose?...

    Following the principle of RTFM, I've typed this in my shell: 
    > perldoc -f read

    [download]
    And received this, which is exactly the answer you were seeking there: 
         read FILEHANDLE,SCALAR,LENGTH
             Attempts to read LENGTH bytes of data into variable
             SCALAR from the specified FILEHANDLE.  Returns the
             number of bytes actually read, "0" at end of file,
             or undef if there was an error.  SCALAR will be
             grown or shrunk to the length actually read.  If
             SCALAR needs growing, the new bytes will be zero
             bytes.  An OFFSET may be specified to place the read
             data into some other place in SCALAR than the
             beginning.  The call is actually implemented in
             terms of stdio's fread(3) call.  To get a true
             read(2) system call, see "sysread".

    [download]
    Remember, let 'perldoc' be thy oracle at all times of trouble! ;)

    For your second question, I'm not aware of any Perl module that would allow you to implement a progress bar inside your web page. I guess this could be done by writing your current program progress to a file and displaying this file inside a continually refreshing page. The way you could track your program's progress, however, is discussed in the Progress Bar thread. I hope this helps ;).

    _____________________ 
    $"=q;grep;;$,=q"grep";for(`find . -name ".saves*~"`){s;$/;;;/(.*-(\d+)
+-.*)$/;$_=&#91"ps -e -o pid | "," $2 | "," -v "," "]`@$_`?{print"
++ $1"}:{print"- $1"}&&`rm $1`;print"\n";}

    [download]
[reply]
[d/l]
[select]
Re: optimization of browser upload
by samtregar (Abbot) on May 16, 2002 at 01:05 UTC
    Choosing a read() buffer size is largely arbitrary. Making it bigger might help but you'll have to test to know for sure.

    My preference would be to just use CGI.pm or Apache::Request (if running under mod_perl) to handle the upload. Chances are their method will be as fast, and far more reliable, than anything you or I would create independently. Just think about how much testing these modules have undergone! How much testing will you do with your code?

    -sam

    PS. I first misread your question as being about sending files fast. I dove into my copy of the Eagle book and dug out $r->send_fd($fh), which is a really fast way to send files to clients under Apache/mod_perl. No need to loop - just one call and Apache handles the rest with blazingly fast C code that might even use the sendfile() system call if you're lucky. Unfortunately I don't know of anything similar for handling file uploads.

[reply]
[d/l]
Re: optimization of browser upload
by fireartist (Chaplain) on May 16, 2002 at 08:22 UTC
    quote:
    at the moment when the client selects a file and hits the upload button, the browser hangs until the file is fully transferred and only then does the next page display notifying the client that the transfer was successful - is there anyway of giving a progress report will a file is uploading

    Unfortunately the Progress Bar thread mentioned above doesn't appear to relate to CGI at all.

    If the CGI's output (the new HTML page) doesn't change until the upload is complete, then you could use a server-side technology to display a notice saying "file upload in progress".
    One option would be to use JavaScript.
    (I know this might not be popular, it's just an option!)

    You would use the onSubmit() handler attached to the submit button.
    This could then, for example, turn on the visibilty of a layer with a notice saying "file upload in progress".
    note: this method would not be able to calculate the progress of the file upload.
    Here is a couple of links for more information,
    script examples
    tutorials
[reply]
Re: optimization of browser upload
by domm (Chaplain) on May 16, 2002 at 19:19 UTC
    except for the 1024. What is its purpose?

    As vladb pointed out, this limits the amount of data read. If you upload a 3 GB file without using some value for the buffer size, (I think) all the 3 GB would be loaded into memory, thus probably crashing your server. By using the technique you mentioned, you only load 1024 byte into memory, write them to the filehandel, read the next 1024, and so on.

    is there anyway of giving a progress report will a file is uploading - the status bar at the bottom of an IE4 window gives no reflexion of how much long the file will take to transfer.

    There is Apache::UploadMeter which does this, but I never used it. I think it requires mod_perl. 

    --
#!/usr/bin/perl -w just another perl hacker
print+seek(DATA,$=*.3,@-)?~~<DATA>:$:__DATA__
[reply]
      As vladb pointed out, this limits the amount of data read. If you upload a 3 GB file without using some value for the buffer size, (I think) all the 3 GB would be loaded into memory, thus probably crashing your server.

      Incorrect. The third argument to read is indeed the buffer size, but not setting it will not result in all data being read at once. Instead, it will result in this error:

      Not enough arguments for read

      -sam

[reply]
[d/l]

Back to Seekers of Perl Wisdom