You could designate a (single!) thread whose job is “to do fetches.”   It would service a thread-safe queue:   sleep until a filename appears on the queue, pop it off, fetch the file, and post a completion-message onto some other queue.   The number of occurrences of this thread would determine how many fetches could be going on simultaneously.

If you are not settled on LWP::UserAgent for getting the files, you could also take a look at AnyEvent::HTTP, Mojo::Client and other goodies that CPAN has to offer.

My current solution times out because the file takes to long to fetch.

As for doing it "asynchronously", there are three classical solutions: use an event loop, fork a new process, spawn a new thread. I don't see how any of them will help with a timeout.

Hmm. The main problem is that the script is supposed to run via Apache, and is supposed to print "1", which will signal that it's finished. After this it will fetch the file and do some other stuff, but this won't be printed to the user.

Maybe it's possible to tell Apache that the script is finished immediately after the "1" is printed, so that the web page stops loading, but the script continues to work in the "background"?
There isn't a problem that the script takes time, the problem is that the system that calls the URL can't wait for the webpage to load.

So you want to Watch Long Processes Through CGI, actually?

It's curious that you don't want any error checking (in which case you could use Watch Long Processes Through CGI).

I can see two approaches.

• Spawn a task to do the work.

  use POSIX qw( setsid );
  
  print "Content-Type: text/plain\n";
  print "\n";
  print "1";
  
  my $pid = fork();
  exit(0) if !defined($pid) || $pid;
  
  open STDIN,  '</dev/null' or die;
  open STDOUT, '>/dev/null' or die;
  open STDERR, '>&STDOUT'   or die;
  setsid();
  
  ... handle request ...
  [download]

• Assign the work to an existing task or to a cron job.

  ... queue request in a file or database ...
  
  print "Content-Type: text/plain\n";
  print "\n";
  print "1";
  [download]