Why dp you want to use less resources?
If it is to allow other processes to use more, then reduce the priority (increase the niceness on *nix?), to allow other processes a greater share of the cpu.
Or, use the register() method to add a callback and add a short sleep to slow things down a bit.
But remember, all the time that your cpu is not running at 100% utilisation, it is just wasting energy and time.
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.
| [reply] |
I'm glad I work for you, because I can keep you happy by just constantly going "is it done yet?" as fast as I can. Any time I spend not running around at full throttle is just wasting me as a resource / employee.
Yes, even CPUs (modern ones) use less energy when they aren't super busy. Then there is the small matter of resource allocation not being perfect so having one CPU hog usually means your system becomes sluggish, slowing more important tasks (even though their priority is set higher), wasting the human user's time, and causing frustration.
| [reply] |
Have you tried fetching the content of just 50 sites at a time? Perhaps 2000+ is too aggressive for your hardware. | [reply] |
How do I set how many sites I want to fetch at a time with LWP Parallel? I have both max_hosts and max_req set to 10.
| [reply] |