in reply to Re: Parallel::ForkManager (high cpu and a lot of memory)
in thread Parallel::ForkManager (high cpu and a lot of memory)

I'm checking around 200 ip's. And when I look at my task manager in about 15 secs the process already has 400mb and it keeps on getting more. The cpu jumps directly to 100% when I run the script.
  • Comment on Re^2: Parallel::ForkManager (high cpu and a lot of memory)

Replies are listed 'Best First'.
Re^3: Parallel::ForkManager (high cpu and a lot of memory)
by Joost (Canon) on Oct 07, 2008 at 21:29 UTC
    The fact that your cpu usages jumps up is good. You generally want to use your cpu as much as possible, as that means you're not waiting on network traffic.

    On the other hand:

    1. Unless you're on Windows, you do not get 1 process, you'd get a process for each fork(), meaning 10 processes in this case.

    2. Each child process will take a finite amount of memory. ForkMananger should keep the number of processes limited and memory should be reclaimed when the forked() child is exiting. Memory usage shouldn't increase indefinitely.

    My guess: you're running on windows, and perl's fork() emulation is giving you trouble. It's possible that using threads may work a little better in that case.

Re^3: Parallel::ForkManager (high cpu and a lot of memory)
by ikegami (Patriarch) on Oct 08, 2008 at 01:03 UTC

    The cpu jumps directly to 100% when I run the script.

    Maximizing resource utilisation is the point of running parallel tasks, so that's a good thing.