sits has asked for the wisdom of the Perl Monks concerning the following question:

As a quick project, I was attempting to use WebService::RTMAgent, which uses LWP for sucking information from Remember the Milk using their API. For some reason, I am seeing truncated data coming back for larger response sizes.

I couldn't see anything obviously wrong in my code, or the library (I posted a ticket on CPAN though), but while searching for clues, I found this, which if true, I imagine could be the issue, since the responses WebService-RTMAgent deal with seem to be all on a massive single line.

Is this a known "problem" or is this acceptable what Perl LWP is doing?

From http://code.google.com/support/bin/answer.py?hl=en&answer=15141:

My reports are truncated when I use Perl LWP. How do I fix it?

Perl LWP module imposes a length-per-line limit that truncates report data. It appears that HTTP::Response has a hard limit to the total line size so it truncates each line. To get around this problem, you can try using wget to download the reports as it doesn't store the response as a member of the object in memory.

Replies are listed 'Best First'.
Re: Http::Response truncation issues
by Corion (Patriarch) on Sep 28, 2009 at 07:41 UTC

    Unless the claim comes with a LWP version number and/or specific indications as of where the truncation would take place, I doubt that. I've used LWP to transfer large files, and never have encountered truncation problems that were inherent to HTTP::Message or HTTP::Response.

    Note that the whole response tends to get stored in memory. So for large downloads you should consider using the :content_filename or the :content_callback argument to directly save the response content.

    If you truly believe that it's not your network connection, a proxy or anything else interfering, you will need to get a network sniffer (like Wireshark) and look at the differences between a transfer that works (without LWP::UserAgent) and the transfer that breaks (with LWP::UserAgent).

[reply]
[d/l]
[select]

      I agree it sounds weird.. but it is interesting that Google thinks its related to long line lengths, not the actual content length, which is what I was dealing with.

      Anyway - I was interested to know if this is a known/expected issue or not. Doesn't sounds like it.. but its disturbing to see this kind of mis-information from Google if its not true.

[reply]
Re: Http::Response truncation issues
by Anonymous Monk on Sep 28, 2009 at 07:44 UTC
    Is this a known "problem" or is this acceptable what Perl LWP is doing?

    Perl LWP module imposes a length-per-line limit that truncates report data. It appears that HTTP::Response has a hard limit to the total line size so it truncates each line. To get around this problem, you can try using wget to download the reports as it doesn't store the response as a member of the object in memory.

    That is complete and utter bullspit! There is no time stamp on that entry, no version numbers are mentioned (LWP dates back to May-1996), and it reports what "appears" to happen, along with imagined limitations. It is misinformation, and it appears, maybe even propaganda or libel :)

[reply]
Re: Http::Response truncation issues
by Anonymous Monk on Sep 28, 2009 at 07:06 UTC
[reply]

Back to Seekers of Perl Wisdom