threads hogging memory

saskaqueer has asked for the wisdom of the Perl Monks concerning the following question:

Before I start, I will just say I understand that threads are not very stable as it is, let alone on win32 systems. However, I've got this belief that maybe I am doing something completely wrong and that the main problem this time around is not to be blamed on threads. I am just throwing together a simple HTTP proxy, the barebone code follows below. The code works fine if you only allow one connection at a time, but that makes things far too slow: so naturally, I turned to threading. Each time a new thread is created, an extra few thousand K of memory is taken up and never released (the thread itself is destroyed; task manager shows the thread count drop). Am I supposed to be using one or more of threads' detach(), yield(), join() methods? I've tried tossing them in multiple places and it always just gives me even more problems!

#!c:/perl/bin/perl -w
$| = 1;

use strict;
use threads;
use IO::Socket;
use HTTP::Request;
use LWP::UserAgent;

my $proxy = IO::Socket::INET->new(
    Listen    => 1,
    LocalAddr => '127.0.0.1',
    LocalPort => 8090,
    ReuseAddr => 1
) or die( "Failed to start HTTP proxy: $!" );

while ( my $client = $proxy->accept() ) {
    threads->create( 'new_request', $client );
}

sub new_request {
    my $client = shift;

    my $page = <$client>;
    $page =~ s![\015\012]+\z!!;

    my ($method, $server, $url, $http_v) = $page
        =~ m!\A(GET|POST|HEAD)\s+http://([^/]+)(/.*)?\s+HTTP/(1.[01])\
+z!i;
    $url ||= '/';

    my %headers;
    while ( my $line = <$client> ) {
        goto( DISCONNECT ) unless ( defined($line) );

        $line =~ s![\015\012]+\z!!;
        last if ( $line eq "" );

        my ($k, $v) = $line =~ m!\A([^:]+):\s*(.+)\z!;
        $headers{ lc $k } = $v;
    }

    my $ua = LWP::UserAgent->new();
    my $req = HTTP::Request->new();
    $req->method( $method );
    $req->uri( "http://$server$url" );

    $req->header( $_ => $headers{$_} ) for ( keys %headers );

    if ( exists( $headers{'content-length'} ) ) {
        read( $client, my $buf, $headers{'content-length'} );
        $req->content( $buf );
    }

    my $res = $ua->request( $req );

    print $client "HTTP/$http_v ", $res->status_line(), "\015\012";
    $res->scan( sub { print $client "$_[0]: $_[1]\015\012"; } );
    print $client "\015\012", $res->content();

DISCONNECT:
    shutdown( $client, 2 );
    return();
}
[download]

Comment on threads hogging memory Download Code

Replies are listed 'Best First'.
Re: threads hogging memory by BrowserUk (Patriarch) on May 19, 2004 at 05:19 UTC
Try modifying the line in your while loop to read `threads->create( 'new_request', $client )->detach();` [download] It won't fix all the problems, but it does reduce the memory consuption of your proxy substantially. You will still see long term memory growth, but the rate of growth is less than 1% of the unmodified code (based on very unscientific and insubstantial tests!). Examine what is said, not who speaks. "Efficiency is intelligent laziness." -David Dunham "Think for yourself!" - Abigail	[reply] [d/l]
Re^2: threads hogging memory by saskaqueer (Friar) on May 19, 2004 at 05:27 UTC
I had already tried such an approach. Changing that specific line yields the following slew of errors: Attempt to free unreferenced scalar: SV 0x280babe0 at c:/Perl/lib/IO/S +ocket.pm l ine 46, <GEN2> line 14. Attempt to free unreferenced scalar: SV 0x280babe0 at c:/Perl/site/lib +/Net/HTTP/ Methods.pm line 37, <GEN2> line 14. Attempt to free unreferenced scalar: SV 0x280babe0 at c:/Perl/site/lib +/Net/HTTP/ Methods.pm line 42, <GEN2> line 14. Attempt to free unreferenced scalar: SV 0x280babe0 at c:/Perl/lib/IO/S +ocket.pm l ine 46. Attempt to free unreferenced scalar: SV 0x280babe0 at c:/Perl/lib/IO/S +ocket.pm l ine 46, <GEN3> line 14. Attempt to free unreferenced scalar: SV 0x280babe0 at c:/Perl/site/lib +/Net/HTTP/ Methods.pm line 37, <GEN3> line 14. < snip > [download] After a few more such errors, Windows throws up a fatal error and the perl executeable exits. "This is perl, v5.8.2 built for MSWin32-x86-multi-thread" -- build 808, IThreads enabled.	[reply] [d/l]
Re: Re^2: threads hogging memory by BrowserUk (Patriarch) on May 19, 2004 at 05:33 UTC
Upgrade to AS 809, there is a patch specifically applied to correct that. `Locally applied patches: ActivePerl Build 809 22218 Remove the caveat about detached threads crashing on Win +dows 22201 Avoid threads+win32 crash by freeing Perl interpreter sl +ightly later 22169 Display 'out of memeory' errors using low-level I/O 22159 Upgrade to Time::Hires 1.55 22120 Make 'Configure -Dcf_by=...' work 22051 Upgrade to Time::HiRes 1.54 21540 Fix backward-compatibility issues in if.pm Built under MSWin32 Compiled at Feb 3 2004 00:28:51` [download] I see no errors at all using your (modified) code. I'm posting through it now. In fact, having navigated all the main subhead pages at news.bbc.co.uk, the memory usage is completely stable. The use rises and falls as I navigate, but stays remarkably stable at ~ 13 MB over time. Examine what is said, not who speaks. "Efficiency is intelligent laziness." -David Dunham "Think for yourself!" - Abigail	[reply] [d/l] [select]