Re: Parsing of undecoded UTF-8 will give garbage

i don't know if this willbe of much help but...

if all you want to do is fetch the document, you could just use a system call to 'wget', thus bypassing the whole LWP (it's not exactly elegant, but it will get you your pages). alternatively you could run your own content test for the character encoding. that would involve building smallish statistical model, and then comparing the content of a suspected UTF-8 page against that of the model. this would be a more elegant model - its probably what mozilla does when it has to guess the character encoding for a page that is unmarked - but it would obviously require some more work on your part.

or why not check out Nutch's innards?

sorry if you read all this and found it a waste of time.

Comment on Re: Parsing of undecoded UTF-8 will give garbage

Replies are listed 'Best First'.
Re^2: Parsing of undecoded UTF-8 will give garbage by saintmike (Vicar) on Aug 03, 2006 at 06:23 UTC
From what I've found out so far, it's a bug in LWP. But instead of reverting to other methods, it's usually better to send a bug report to the module author(s) and get it fixed. I've sent it to the LWP mailing list, should be on its way!	[reply]