in reply to Help with LWP::UserAgent

The hard way of approaching this problem is to install a logging proxy (written with, for example, HTTP::Proxy) or a network sniffer (for example ethereal or something using Net::PCap) between your browser and the network traffic, see what gets sent between the two, and replay that from the script.

This is hard, because you will see much stuff that is unrelated and/or have to set up things.

The easy way is to use WWW::Mechanize, which tries relatively hard to emulate a browser. It handles cookies already for you and it has easy ways of masquerading as a browser as well.

If a script using WWW::Mechanize does not work, then you will have to fall back onto the above soutions.

I can't test it from here, but I think that the following WWW::Mechanize script recreates what your script does:

use strict; use WWW::Mechanize; my $agent = WWW::Mechanize->new(); my $url = '"http://www.accountancy.smu.edu.sg/facultystaff/faculty.htm +'; $agent->get($url); print "Got return code ", $agent->code, "\n"; open FH, ">", "staff.txt"; print FH $agent->content; close FH;