in reply to Perl threads to open 200 http connections

Change use threads; to use threads ( stack_size => 4096 ); See Use more threads. for the explanation.

You might also consider using LWP::Simple::getstore( $url, $file ).

Replies are listed 'Best First'.
Re^2: Perl threads to open 200 http connections
by robrt (Novice) on Aug 03, 2010 at 10:59 UTC

    Thanks! I changed the stack size to 4096, and now around 185 connections are getting established. I tried to lower the stack size to 1024 hoping to see the connections rise but it didn't happen.

    I tried to implement LWP::Simple but I noticed that it copies only the website matter(visible content on site), not the actual 10MB files.

    So, when I tried copy the files using the below code, this script is failing after 30 threads itself.
    use strict; use LWP::Simple; my $path = "http://10.2.1.23/http-path/payload/10MB/"; for (my $i=0;$i<200;$i++) { my $filename = "File-10M-".$i.".txt"; my $url = $path.$filename; my $file = "D:\\kshare\\payload\\$filename"; $thread = threads->new(\&httpcon, $url, $file); } $thread->join; sub httpcon { my $url = shift; my $file = shift; is_success(getstore($url, $file)) or die "$!\n"; }
    My aim is to fill the network pipe almost to 400 MB, but opening around 180 files is filling only around 70MB. Can anyone suggest me better way to fill the network bandwidth please.

      The first thing I notice is that there are things wrong with the code you've posted.

      You have use strict; but $thread is never declared?

      Also, you are attempting to start 200 threads, but only waiting for one to complete.

      Try this:

      #! perl -slw use strict; use threads ( stack_size => 4096 ); use threads::shared; use LWP::Simple; use Time::HiRes qw[ time sleep ]; our $T ||= 200; my $url = ### your url (of the actual file!) here ###; my $running :shared = 0; my $start = time; for( 1 .. $T ) { async( sub{ { lock $running; ++$running }; sleep 0.001 while $running < $T; my $id = shift; getstore( $url, qq[c:/test/dl.t.$id] ); --$running; }, $_ )->detach; } sleep 1 while $running; printf "Took %.3f seconds\n", time() - $start;

      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.

        Thanks! I executed the prg that you gave, but it failed throwing the below error: Free to wrong pool 234eb0 not 11475c80 at C:/Perl/lib/Errno.pm line 16.

        Also, in my original prg, I was trying to download 200 different files changing the URL each time, but in the prg that you gave, looks like it downloads a single file 200 times.. is that right ?

        If you dont mind, could you please explain your program, (short desc for each line) as its confusing for me the whole concept of async, also the way you have coded to prg, I could not understand it. Thanks again!