Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Hi,

I am using SOAP::Lite (latest version) to make call to a webservice with large file (about 50MB). I am using following code. I see that when making soap call the memory usage is very high (approx 1.5GB). And if I go above 60MB, I am running out of memory and perl scripting is exiting with out making the SOAP call. Is this the error in the SOAP::Lite module or I am using it the wrong way. Also, what is a better way to make a SOAP call with large data. Please see below for the code I am currently using: 

#Read the file
my $value = read_file( $attachmentFilename , binmode => ':raw', scalar
+_ref => 1 );
    
my @params = ( $header,
               SOAP::Data->name(fileName => $attachFilename),
               SOAP::Data->name(taskDataContent => encode_base64($$val
+ue)), #base64 encode the file contents
               SOAP::Data->name(results => $results));

#Call the webservice with SOAP::Lite
$returnResult = $soap->call($ptrMethod => @params);

[download]

Any help is greatly appreciated.

Thanks, Ana

Replies are listed 'Best First'.
Re: Sending large file contents in SOAP call using SOAP::Lite
by keszler (Priest) on Oct 18, 2013 at 01:40 UTC
    It's been a while since I last used SOAP::Lite, but my old code from then looked like 
    my $soap = SOAP::Lite
-> uri('http://server.name/nas/')
-> outputxml(1)
;
$resp = $soap->call('nas:show_session_ids');
$resp = $soap->call('nas:add_device' =>
   SOAP::Data->name(sessionid     => $sid      )->prefix('nas'),
   SOAP::Data->name(ip            => $ip       )->prefix('nas'),
   SOAP::Data->name(comment       => $comment  )->prefix('nas'),
   SOAP::Data->name(hostname      => $hostname )->prefix('nas'),
   SOAP::Data->name(domain        => $domain   )->prefix('nas'),
   SOAP::Data->name(location      => $location )->prefix('nas'),
   SOAP::Data->name(unmanaged     => $unmanaged)->prefix('nas'),
   SOAP::Data->name(accessmethods => $accessm  )->prefix('nas'),
);

    [download]
    Depending on what is in $header, how you initialized $soap, and whether or not you need a namespace (e.g. 'nas'), your example code could be correct.

    It does not seem reasonable to me that a 50MB file could cause 1.5GB of memory usage. I'd start troubleshooting by looking into $value. Was your SOAP::Data->name(taskDataContent => encode_base64($$value)) a typo, or is it actually $$value because read_file is returning a reference vs a string?

[reply]
[d/l]
[select]
Re: Sending large file contents in SOAP call using SOAP::Lite
by Anonymous Monk on Oct 18, 2013 at 01:36 UTC

    Is this the error in the SOAP::Lite module or I am using it the wrong way.

    Yeah, 1.5GB seems like there might be a problem ... but not where, I doubt its within SOAP::Lite

    perl -Mojo -e " b( shift )->slurp->b64_encode->say; int <>; " 49_678_352_bytesfile > 65M-b64ed

    This takes about ~130mb on my machine, and around 20mb of that is perl -Mojo itself alone

    I can't imagine SOAP::Lite tacking eleven times that amount, maybe one or three temporarily, but not eleven

    Without mojo, with File::Slurp+SOAP::Lite, to slurp this 65M b64ed file takes about 74mb, 1500mb/74mb = 20 so still nowhere in the ballpark

    I don't feel like looking into it past this; You can try 

        $soap->serializer()->envelope(
        method => 'DoThis', 
        $param
    ),

    [download]
    to see how much memory it takes to compose that request you're using -- shouldn't be 1.5 GB

    I imagine it is the response that you're getting (your program is getting) to that request is an xml bomb ... or something else entirely

    Also, what is a better way to make a SOAP call with large data

    I imagine maybe https://metacpan.org/module/SOAP::Lite#ATTACHMENTS,

    or even XML::Compile::XOP

    but if your server doesn't accept that :)

    Good luck

[reply]
[d/l]
      I tried serializer code you mentioned on a 34mb file and it almost used up 500MB while it is serializing and I printed the length of the resultant serialized text 47mb. So, the issue is not the size of the SOAP message it is sending or receiving, it is that serialization itself using up too much memory. Do you or anyone see the same issue?
[reply]
[reply]

Back to Seekers of Perl Wisdom