Hi monks,

After two days of frustrating experimenting, I'm hoping that one of you can help me with this. Ahead: I'm not looking for suggestions. Please only post if you have a solution you know works, as any suggestion you may give I've likely tried already.

As such, here's the question: How would i go about downloading multiple files in parallel
---

Solutions so far:

Combined from the input of ikegami and Corion, a solution that uses IPC::Open2 and an external wget executable. Runs very fast and does not require much RAM. 
    use IPC::Open2;

    for my $id  (@ids) {
        $wgets++;
        
        push @pids, open2(undef, undef, 'wget', $url.$id, '-q', '-O', 
+$dir.$id);
        
        while ( @pids >= 10 ) {
            waitpid( shift @pids, 0 );
        }
    }

    while ( @pids ) {
        waitpid( shift @pids, 0 );
    }

[download]

From BrowserUk a solution that uses threads and Thread::Queue, thus eleminating the need for an external executable. It does however use more RAM when running at speeds comparable to the previous solution. 
sub fetch_xml_data {
    my ($ids) = @_;

    my $dir = 'quicklook/';
    my $url = 'http://api.eve-central.com/api/quicklook?typeid=';
    my $thread_count = 20;
    my $Q = new Thread::Queue;
    my @threads;
    
    for my $id (@{ $ids }) {
        $Q->enqueue( $id );
    }
    
    for ( 1 .. $thread_count ) {
        push @threads, threads->create(
            sub {
                require LWP::Simple;
                while( my $id = $Q->dequeue ) {
                    say "Downloading XML file for id $id.<br>";
                    LWP::Simple::getstore( $url.$id, $dir.$id );
                }
            }
        );
        $Q->enqueue( undef );
    }
    
    $_->join for @threads;
}

[download]
</readmore

In reply to Parallel downloading under Win32? by Xenofur

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.