in reply to Sequential processing with fork.

It looks like you've got clients forking more clients, and that goes out of control fast.

ISTM that what you want is:

  1. A master routine to:
    1. Fork off ten clients
    2. Add to the queue of tasks until done
    3. Queue up ten special 'terminate' tasks
  2. A client routine (ten copies made by the master) to:
    1. Shift a task out of the queue
    2. Do the task
    3. Terminate if the task is a special terminate task
    4. Sleep if there are no tasks in the queue at the moment

Keeping your forked code sterile will help prevent you from accidentally making a fork bomb.

Replies are listed 'Best First'.
Re^2: Sequential processing with fork.
by stevieb (Canon) on Aug 04, 2015 at 19:45 UTC
    "It looks like you've got clients forking more clients, and that goes out of control fast."

    I laughed out loud quite loudly in my office as soon as I read this for some reason. Nice :)

[reply]
      Now if I could get all my clients to fork each other, the problem would take care of itself aye? Haha
      ~~~~~~~~~ I'm unique, just like everybody else! ~~~~~~~~~
[reply]
Re^2: Sequential processing with fork.
by Kelicula (Novice) on Aug 04, 2015 at 20:33 UTC

    Yes, I would definitely not want a fork bomb.

    The process that runs for each client is actually a 3000 line script that uses .NET to automate Internet Explorer in "private" mode, (so they can all run together without cookie_jar issues) So there's no room for more processes than needed.

    As said previously I've been just checking the count and sleeping for roughly an hour to let all 10 finish, then starting the next group, but I often see only 1 or 2 left running and realized I could significantly speed it up by ensuring there were ALWAYS 10 running..

    I'll incorporate all you guys advice and post the results to show solution.

    Thank you!!

    Oh yeah, I obviously don't want to start my process with system() anymore because that creates another fork right?

    Could I just use backticks to call the other script and pass the client ID as $ARGV[0]?

    Any thoughts?...
    ~~~~~~~~~ I'm unique, just like everybody else! ~~~~~~~~~
[reply]
      "exec" it
[reply]

In Section Seekers of Perl Wisdom