in reply to Search Engine Simulator

On the CPAN, there is WWW:Robot, which will spider your URLs, and it is also not very hard to write a spider using WWW::Mechanize. You will need to set the proper UserAgent header in both cases, so your spider (mis)identifies itself as Google, MSN or whatever.se

Replies are listed 'Best First'.
Re^2: Search Engine Simulator
by peterr (Scribe) on Dec 10, 2004 at 10:26 UTC
    Hi,

    Thanks, both of those modules look promising, especially the second one, where we can set the user agent, because we need to test quite a number of agent names to make sure the session id's get turned off for them.

    Peter
[reply]

      I expect any such module to be based on the LWP library and thus you should be able to set the user agent in any of those libraries with equal ease. But I haven't worked with WWW::Robot, so I don't know if it actually uses LWP (it should).

[reply]
        CPAN tells me that WWW::Robot has a dependancy of LWP::RobotUA
[reply]

In Section Seekers of Perl Wisdom