fork subroutine

aquitaine has asked for the wisdom of the Perl Monks concerning the following question:

i've been thinkin over this and cant seem to get the right way into things. I have a batch of URL's. Lets say

http://www.cnn.com/
http://www.bbc.com/
http://www.news.com/

i would like to use LWP::Simple;

and get() them on my machine, without waiting for them to finish the download - something like a wget url.com &

How could i do this with a fork() ?

my plan was something like

#!/usr/bin/perl -w

use LWP::Simple;
use strict;
my @urls = qw!http://www.cnn.com/
http://www.bbc.com/
http://www.news.com/!;

for (@urls) { 
&get_em($_);
}

sub get_em { 
my $url = shift;
$url =~ m!http://www.(.*?)/!;
if ($pid = fork) {  } 
elsif ($pid == 0 ) { 
getstore($url, "$1.txt");
}
}
[download]

I dont know the real code i want for this, could someone help me out please ?

Thanks

Aquitaine

Comment on fork subroutine Download Code

Replies are listed 'Best First'.
Re: fork subroutine by saintmike (Vicar) on Apr 12, 2004 at 20:48 UTC
If you're looking for different ways to pull in web documents in parallel with or without forking, check out LWP::Parallel::UserAgent, POE::Component::HTTP::Client and ForkedUA.pm.	[reply]
Re: fork subroutine by jweed (Chaplain) on Apr 12, 2004 at 20:39 UTC
I would write it like this. Basically, the change that I made is to have it so that the parent continues to fork off children to do its bidding, while the child downloads then exits. You were very close: `#!/usr/bin/perl -w use LWP::Simple; use strict; my @urls = qw!http://www.cnn.com/ http://www.bbc.com/ http://www.news.com/!; for my $url (@urls) { get_em($url); } sub get_em { my $url = shift; unless (my $pid = fork) { die "Couldn't fork on $url" unless defined $pid; die "Malformed url $url" unless $url =~ m!http://www.(.*?)/!; my $response = getstore($url, "$1.txt"); die "$url get failed" if is_error($response); exit; } }` [download] Update: Moved the malformed url check Update: See tilly's reply Code is (almost) always untested. http://www.justicepoetic.net/	[reply] [d/l]
Re: Re: fork subroutine by tilly (Archbishop) on Apr 13, 2004 at 01:43 UTC
For future reference, if you code up an example robot, please show how to fetch robots.txt and use http://WWW::RobotRules. It is unlikely that the person asking will know that this is important, and therefore it is our responsibility to pass along best practices.	[reply]
Re: fork subroutine by gmpassos (Priest) on Apr 13, 2004 at 06:59 UTC
Take a look in the module forks, since you can create something like a thread system using fork. Graciliano M. P. "Creativity is the expression of the liberty".	[reply]
Re: fork subroutine by DigitalKitty (Parson) on Apr 13, 2004 at 10:11 UTC
Hi Aquitaine. In addition to the other suggestions, I would recommend: Parallel::ForkManager It's available on both cpan as well as the various ppm repositories. Hope this helps, -Katie.	[reply]