The most likely reason is that you are never joining your threads. Just loading the front page at CNN.com started 31 threads. These threads then terminate, but are never cleaned up. Each thread is somewhat over 1 MB in size, and the result is that having loaded cnn.com, the memory use has grown to close to 50 MB in total. A couple of refreshes and this will force swapping and resource exhausion.
Your also passing $req and $host, lexical scalars, to each thread. Whilst the scalar is being shared automatically for you, and you are using it read-only, so the lack of locking is probably ok. Each time you share a variable, it is shared with every thread. That means that every thread created (including those that are dormant but unjoined is getting a copy of every request object added to it memory space.
As a first pass at fixing this, you should detach your threads once you've spawned them so that the die a natural death and undef $req & $host before the thread terminates.
... } else { threads->create( \&process_one_req, $browser, $req, $host )->detach; } ... sub process_one_req { my ($browser, $req, $host) = @_; my $remote = new IO::Socket::INET( Proto => "tcp", PeerAddr => $host, PeerPort => 80 ); if ($remote) { print $remote $req; my $chunk; print $browser $chunk while (sysread($remote, $chunk, 10000)); close($remote); undef($remote); } else { print $browser RES_400; } close($browser); undef($req); undef($host); undef($browser); }
Making these changes, I can load and reload the cnn frontpage and whilst the memory use grows to around 8 MB at the peak, it rapidly falls back to aroud 5 MB as the requests complete and the threads die. This seems to cure the continuous memory growth completely and may effect a cure for your transient core dumps.
I also noticed that if the request is a POST rather than a GET, then your regex to extract the page name fails and results in
Use of uninitialized value in concatenation (.) or string at P:\test\p +roxy.pl8 line 42. Received request for [perlmonks.com, ]
HTH.
In reply to Re: another core dump
by BrowserUk
in thread another core dump
by pg
| For: | Use: | ||
| & | & | ||
| < | < | ||
| > | > | ||
| [ | [ | ||
| ] | ] |