Do you happen to know what purpose Javascript plays in this process? Is it possible that it's altering the browser's request? If that's the case, the page's Javascript would never be executed with your Javascript-unaware user agent. This is a pretty common issue.

Just something to consider. This could be one of those situations where WWW::Mechanize::Firefox comes in handy. If only a true browser can come up with the correct scrape, just let Perl manipulate the browser for you.