Morning all,

I've got a program here, which submits 'n' megabytes of data into a cgi, via POST, using the UserAgent mechanisms. The core memory usage of perl appears to be about eight times greater than that directly attributable to code outside of the UserAgent module, which seems to represent an excessive degree of string copying within the module.

I've included a minimum subset program for your perusal; I thought I'd get some comments and suggestions before bothering the author of the package.

Has anybody any ideas about why this much inefficiency appears to exist inside of the function call? Is this something about should be looked into, or am I missing something obvious here?

#!/usr/bin/perl # Includes. use HTTP::Request::Common; use LWP::UserAgent; use HTTP::Cookies; # Changable Parameters $size=32; $WebHost='1.2.3.4'; $FirstStage="http://$WebHost/wibble/flip/pants.cgi"; $cookie_jar=HTTP::Cookies->new( file => "./cookies"); # Code print STDERR "Creating user agent..."; $ua=LWP::UserAgent->new; print STDERR "done\n"; print STDERR "Creating cookie jar..."; $ua->cookie_jar($cookie_jar); print STDERR "done\n"; %myhash = [ username => "" ]; print STDERR "making string...\n"; open(zero, "/dev/zero") || die "/dev/zero failed to open: $!"; sysread(zero, $myhash{username}, $size*1024*1024); # using sysread to remove i/o buffers from our problem space. Solaris + will honour our request in full. YMMV. close(zero); print STDERR "username now "; print length($myhash{username}), " bytes\n"; print STDERR "About to post form data..."; $response=$ua->request(POST $FirstStage, \%myhash); print STDERR "done\n";

With size=32, I get a 228M (RSS 193M) core image.

With size=64, I get a 452M (RSS 333M) core image.

Any comments or ideas would be much appreciated.

Cheers, Mike.


In reply to Excessive UserAgent/Request memory usage. by fysh

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":



  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.