cormanaz has asked for the wisdom of the Perl Monks concerning the following question:

Good day monks. I am trying to get the hang of using multiple processes with WWW::Mechanize. I think I've just about got it figured out except for one thing. Can different processes share the same mech object? In other words can I
use Proc::Queue size => 3;
my $mech = WWW::Mechanize->new(timeout => 90);
while (@urllist) {
    $pid = fork;
    if(defined ($pid) and $pid==0) {
        $mech->get($url);
        print "$mech->content()->as_HTML();
        exit(0);
    }
    1 while waitpid(-1, WNOHANG)>0; # reaps childs
}

[download]
or do I have to
use Proc::Queue size => 3;
while (@urllist) {
    $pid = fork;
    if(defined ($pid) and $pid==0) {
    my $mech = WWW::Mechanize->new(timeout => 90);
        $mech->get($url);
        print "$mech->content()->as_HTML();
        exit(0);
    }
    1 while waitpid(-1, WNOHANG)>0; # reaps childs
}

[download]
?

Many thanks...

Steve

Replies are listed 'Best First'.
Re: Fork and WWW::Mechanize: Can agent be shared?
by andyford (Curate) on Feb 18, 2007 at 17:16 UTC

    You can do it the first way. But after the fork it's no longer the same object, it's a copy, so you can't share info.

    non-Perl: Andy Ford

[reply]
Re: Fork and WWW::Mechanize: Can agent be shared?
by shmem (Chancellor) on Feb 18, 2007 at 21:06 UTC
    As said before, the first way should work - but I really wouldn't place the waitpid inside the loop, but move that into a $SIG{CHLD} handler.

    --shmem 

    _($_=" "x(1<<5)."?\n".q·/)Oo.  G°\        /
                              /\_¯/(q    /
----------------------------  \__(m.====·.(_("always off the crowd"))."·
");sub _{s./.($e="'Itrs `mnsgdq Gdbj O`qkdq")=~y/"-y/#-z/;$e.e && print}
[reply]
[d/l]
[select]
Re: Fork and WWW::Mechanize: Can agent be shared?
by perrin (Chancellor) on Feb 18, 2007 at 17:17 UTC
    It can be shared, because it won't actually be shared, if you see what I mean. The only thing you have to worry about with shared objects and forking is forking after opening a socket or file, which I don't think will happen here.
[reply]
Re: Fork and WWW::Mechanize: Can agent be shared?
by ikegami (Patriarch) on Feb 19, 2007 at 18:18 UTC
    You'll probably find Parallel::ForkManager useful. 
    use Parallel::ForkManager qw( );

use constant MAX_CHILDREN => 3;

{
   my $mech = WWW::Mechanize->new(timeout => 90);

   my @urllist = ...;

   my $pm = Parallel::ForkManager->new(MAX_CHILDREN);
   foreach my $url (@urllist) {
      # Forks and returns the pid for the child.
      my $pid = $pm->start() and next; 

      $mech->get($url);
      print $mech->content()->as_HTML();

      # Exit child.
      $pm->finish();
   }
}

    [download]
[reply]
[d/l]
Re: Fork and WWW::Mechanize: Can agent be shared?
by TOD (Friar) on Feb 19, 2007 at 04:15 UTC
    if you take a shared memory segment you can store a serialized form of the object you want to be common to all processes on it. it must be serialized, because you can't store references on a shmem segment.

    but basically i would prefer threads instead of forking.
[reply]
Re: Fork and WWW::Mechanize: Can agent be shared?
by jdrago_999 (Hermit) on Feb 19, 2007 at 18:22 UTC
    If you need to share data between multiple processes, take a look at the forks and forks::shared modules (as well as the threads and threads::shared modules).

    The semantics are different than what you're doing now, and there is some extra overhead, but I've been using both threads and forks for a couple years now with no problem, in production environments no less.
[reply]
Re: Fork and WWW::Mechanize: Can agent be shared?
by tphyahoo (Vicar) on Feb 20, 2007 at 13:20 UTC
[reply]

Back to Seekers of Perl Wisdom