Wanted: Multi-page web robot for automated testing

meonkeys has asked for the wisdom of the Perl Monks concerning the following question:

I'm looking for a perl module--LWP is a good start--that can spider through a Web process, looking for links and sending new requests based on HTTP responses (eg. automate setting up a Hotmail account) anyone know of such a robot or code that would be a good model for writing one?

I've starting writing one of my own that basically sends one request, looks for a link in the page using a regex, and sends another request, until the process is complete. The multi-step process I'm testing logs a user into my site, so I know the test was successful if a particular cookie is set (I can just check the cookie jar that LWP uses). I can also just grep the content for a "Success!" message that I know should be there if the login was successful.

I have looked at HTTP::WebTest, but this seems specialized for testing particular pages and their response times, not a sequence of pages and requesting new pages by using links in the content of requested pages. Any ideas?

--
"A Jedi uses the Force for knowledge and defense, never for attack."

Comment on Wanted: Multi-page web robot for automated testing

Replies are listed 'Best First'.
(Ovid) Re: Wanted: Multi-page web robot for automated testing by Ovid (Cardinal) on Dec 13, 2001 at 08:11 UTC
You might want to check out RoboWeb on SourceForge. This was written by gregorovius and he posted a node about it. I suspect that this would be perfect for you. Cheers, Ovid Join the Perlmonks Setiathome Group or just click on the the link and check out our stats.	[reply]
Re: Wanted: Multi-page web robot for automated testing by IlyaM (Parson) on Dec 13, 2001 at 15:19 UTC
With HTTP::WebTest you could write such test as long as you know all urls in test sequence. I know it is quite limiting. It would be very nice to be able just say "click button named FooBar" and etc. I'm working right now on HTTP::WebTest 2.xx which will introduce such features. For now you probably have to look somewhere else. Maybe RoboWeb is appropriate for you. You can also write your own script using LWP::UserAgent and HTML::Form. This combo is quite powerful. -- Ilya Martynov (http://martynov.org/)	[reply]
Re: Wanted: Multi-page web robot for automated testing by miyagawa (Chaplain) on Dec 13, 2001 at 15:23 UTC
Yet another choices: HTTP::TestEngine, Deluge -- Tatsuhiko Miyagawa miyagawa@cpan.org	[reply]
Re: Wanted: Multi-page web robot for automated testing by dash2 (Hermit) on Dec 13, 2001 at 15:15 UTC
Try WWW::Robot on CPAN. dave hj~	[reply]
Re: Wanted: Multi-page web robot for automated testing by Fastolfe (Vicar) on Dec 13, 2001 at 22:00 UTC
I've used WWW::Chat for things like this with success. It acts like a source code filter, in that you can say things like "FOLLOW /search/i" and "CLICK 'submit'".	[reply]
Re: Re: Wanted: Multi-page web robot for automated testing by blakem (Monsignor) on Dec 14, 2001 at 05:47 UTC
The only confusing thing about WWW::Chat is that its actually spelled webchatpp.... oeuf, has a short demo online as well. -Blake	[reply]