fysh has asked for the wisdom of the Perl Monks concerning the following question:

Morning all,

I've got a program here, which submits 'n' megabytes of data into a cgi, via POST, using the UserAgent mechanisms. The core memory usage of perl appears to be about eight times greater than that directly attributable to code outside of the UserAgent module, which seems to represent an excessive degree of string copying within the module.

I've included a minimum subset program for your perusal; I thought I'd get some comments and suggestions before bothering the author of the package.

Has anybody any ideas about why this much inefficiency appears to exist inside of the function call? Is this something about should be looked into, or am I missing something obvious here? 


#!/usr/bin/perl
# Includes.

use HTTP::Request::Common;
use LWP::UserAgent;
use HTTP::Cookies;

# Changable Parameters

$size=32;


$WebHost='1.2.3.4';
$FirstStage="http://$WebHost/wibble/flip/pants.cgi";

$cookie_jar=HTTP::Cookies->new( file => "./cookies");

# Code

print STDERR "Creating user agent...";
$ua=LWP::UserAgent->new;
print STDERR "done\n";
print STDERR "Creating cookie jar...";
$ua->cookie_jar($cookie_jar);
print STDERR "done\n";

%myhash = [ username   => "" ];

print STDERR "making string...\n";
open(zero, "/dev/zero") || die "/dev/zero failed to open: $!";
sysread(zero, $myhash{username}, $size*1024*1024); 

# using sysread to remove i/o buffers from our problem space.  Solaris
+ will honour our request in full. YMMV.

close(zero);
print STDERR "username now ";
print length($myhash{username}), " bytes\n";
print STDERR "About to post form data...";

$response=$ua->request(POST $FirstStage, \%myhash);

print STDERR "done\n";

[download]

With size=32, I get a 228M (RSS 193M) core image.

With size=64, I get a 452M (RSS 333M) core image.

Any comments or ideas would be much appreciated.

Cheers, Mike.

Replies are listed 'Best First'.
Re: Excessive UserAgent/Request memory usage.
by saucepan (Scribe) on Dec 09, 2000 at 00:29 UTC

    I don't know how much this might help, but one trick I've used to see who is gobbling memory is to add some lines like this in strategic places: warn "About to initialize Foo::Bar\n"; sleep(8);

    Then when you see this message you can quickly run ps in another window to see if the process has swollen up yet. By moving these lines around you can home in on the problem code.

    I'm pretty sure there's a more elegant way of doing this, but this brute force technique has done it for me every time (so far).

[reply]
[d/l]
      Ta, saucey.

      It's definitely happening inside the API call; there's already enough debugging output in there to be sure of that.

      I guess I was hoping for a "You fool! You're passing the <blah> not a reference to it!" type response.

      I suppose it's time to approach the package author. Thanks for letting me run it past you guys first, anyway. I'll let you know what happens.

      Cheers,

      Mike.

[reply]
Re: Excessive UserAgent/Request memory usage.
by Hot Pastrami (Monk) on Dec 08, 2000 at 20:25 UTC
    I'm not sure if it has anything to do with your problem, but to my knowledge, the following line in your code is not correct syntax: 
    %myhash = [ username   => "" ];

    [download]
    ...I think you want 
    %myhash = ( username   => "" ); # parenthases rather than square brack
+ets

    [download]

    Also, have you tried testing the return value of your sysread()? Perhaps something is amiss thre.

    Alan "Hot Pastrami" Bellows
[reply]
[d/l]
[select]
      It doesn't, unfortunately, since the original code doesn't use that (duff) bit of syntax. I hurredly produced the simple subset program - I am, after all, supposed to be working :) The line is completely redundant anyway so removing it should produce a better minimum subset - my apologies.

      The sysread() must(tm) be doing the right thing, since the length of the string is tested on the next line, and is correct. Again the sysread is used only to guarantee maximum "simpleness" for this example; it isn't actually what happens in the program itself.

      Take is as read that these two aspects don't impact on the problem.

      Cheers for responding,

      Mike.

[reply]

Back to Seekers of Perl Wisdom