As you did not stated your needs, this may be overkill, but I use LWP::UserAgent to do "browsing" from a perl script. It allows you to get a URL and read the content, then you can parse it and follow more links. Eventually, you would be able to write a spider bot.