And here's the trace output from LWP:

That can't help, sorry :)

LWP seems to generate "Bad Request" responses.

No, LWP doesn't generate that. It may elicit that response from that particular webserver, but only because that particular webserver , like many, is being too particular.

To sucessfully scrape a website, either use a working browser, or imitate that browser convincingly, this is how to do it