in reply to Parallel::ForkManager (high cpu and a lot of memory)

When you say the module 'seems to work fine', do you mean the module itself, or this script specifically? How many lines in lan.txt? Given the name 'lan.txt', are all the sites you are hitting accessible via fairly high bandwidth links? If the sites you are hitting are generally responding pretty quickly, and you have lots (more than say, 300) of sites in your file, it is not surprising that the CPU pegs. Could you be more specific about draining memory, ie, how much how fast?
  • Comment on Re: Parallel::ForkManager (high cpu and a lot of memory)

Replies are listed 'Best First'.
Re^2: Parallel::ForkManager (high cpu and a lot of memory)
by marto9 (Beadle) on Oct 07, 2008 at 20:42 UTC
    I'm checking around 200 ip's. And when I look at my task manager in about 15 secs the process already has 400mb and it keeps on getting more. The cpu jumps directly to 100% when I run the script.
[reply]
      The fact that your cpu usages jumps up is good. You generally want to use your cpu as much as possible, as that means you're not waiting on network traffic.

      On the other hand:

      1. Unless you're on Windows, you do not get 1 process, you'd get a process for each fork(), meaning 10 processes in this case.

      2. Each child process will take a finite amount of memory. ForkMananger should keep the number of processes limited and memory should be reclaimed when the forked() child is exiting. Memory usage shouldn't increase indefinitely.

      My guess: you're running on windows, and perl's fork() emulation is giving you trouble. It's possible that using threads may work a little better in that case.

      "What should it profit a man, if he should win a flame war, yet lose his cool?"
[reply]

      The cpu jumps directly to 100% when I run the script.

      Maximizing resource utilisation is the point of running parallel tasks, so that's a good thing.

[reply]

In Section Seekers of Perl Wisdom