Most memory efficient way to threading

Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Hi PerlMonks,

I currently have a http server monitor script that I wawnt to improve its memory efficiency. The script spawns 20 threads (using the good old threads, Thread::Queue) each firing a couple of a LWP::UserAgent get (can't use LWP simple as I need to set useragent, among other customized headers). The script costs me around 200MB memory, and I cannot afford any upgrade to my VPS yet. So I really want to know whether there are any better way to thread/fork?

PS, if anyone can show me any articles on how to reduce memory cost for perl scripts, that would be really great. Thanks in advance.

Comment on Most memory efficient way to threading

Replies are listed 'Best First'.
Re: Most memory efficient way to threading by BrowserUk (Patriarch) on Jun 23, 2011 at 02:23 UTC
A quick check shows that each instance of an LWP::UserAgent costs around 6.8MB. Times 20 and that accounts for 136MB of your 200MB before you start adding in threads, stacks, and queues. Are you sure that using 20 threads is actually achieving gains over (say) just 10? If you are just fetching lots of pages in parallel, then you might consider LWP::Parallel::UserAgent. Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice.	[reply]
Re: Most memory efficient way to threading by chromatic (Archbishop) on Jun 23, 2011 at 04:25 UTC
I switched from LWP to WWW::Curl::Simple, which was sufficiently parallel for my purposes and allowed me to use an unthreaded Perl, which is faster and uses less memory. It may not work for you, but it worked nicely for me.	[reply]
Re: Most memory efficient way to threading by Corion (Patriarch) on Jun 23, 2011 at 07:58 UTC
As an alternative to using threads (for simple socket communications), consider looking at AnyEvent resp. AnyEvent::HTTP, which will do nonblocking parallel HTTP transfers from a single thread. It's also fairly lightweight, but I haven't looked at its actual memory usage.	[reply]