Re: LWP on Windows: whitespace removed from HTML

Thanks for all of the suggestions, folks. Here is some additional information, as requested:

1) I have done a "View Source" from within Firefox to verify that there are line breaks within the source HTML, although I don't know if those are "\r\n" or just "\n". However, my cygwin environment is set up to use UNIX line endings, and it still experiences the problem.
2) Some of the sites I have tried include www.cnn.com and www.cdc.gov. In general, some of the line breaks within <script> elements seem to get preserved, but outside of those most of the HTML appears on a single line.
3) I have tried changing the agent to "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)", but I got the same results. I haven't tried using any other agent settings other than the default yet.
4) I have tried using the content returned by LWP::UserAgent with HTML::TreeParser, but it didn't seem to help, although I may need to experiment more with that.

Some additional info - I tried running my test script on another machine running Windows XP Home, and it worked fine there (i.e. whitespace was preserved), so the issue appears to be isolated to my work laptop running XP Pro. I've also tried using the Socket module to download the HTML on that machine, and I experience the same issue of whitespace being removed. Based on that, I suspect this isn't an issue specifically with LWP, but I'm not quite sure what to check next.

Thanks again for all of the help.

Comment on Re: LWP on Windows: whitespace removed from HTML

Replies are listed 'Best First'.
Re^2: LWP on Windows: whitespace removed from HTML by moritz (Cardinal) on May 06, 2008 at 18:24 UTC
This sounds very much like a http proxy that strips whitespace from the pages before delivering them to you. Is there any proxy in your way?	[reply]
Re^3: LWP on Windows: whitespace removed from HTML by Anonymous Monk on May 06, 2008 at 19:38 UTC
Not that I'm aware of. The laptop is using a cellular air-card, so it's not sitting behind a proxy server on the company network. The laptop does see line breaks if I view source from the browser; it's just when I use Perl to download the content that I experience the lack-of-whitespace issues.	[reply]
Re^4: LWP on Windows: whitespace removed from HTML by pvbcharon (Beadle) on May 07, 2008 at 13:24 UTC
Just my $0.02... I've seen providers forcing the use of compressing proxies, especially in the cellular networks here in germany. White-space gets stripped, images get resized and so on, all in order to save bandwidth. This doesn't explain why you see the whitespace in your browser but perhaps it helps.	[reply]
Re^2: LWP on Windows: whitespace removed from HTML by Anonymous Monk on May 06, 2008 at 18:19 UTC
I meant HTML::TreeBuilder in the comment above, not HTML::TreeParser. Sorry for any confusion.	[reply]