in reply to web scraper testing

I'd suggest that you break down the task into a couple of different functions: One would do the login junk and grab the HTML blob. Another would parse the HTML blob and return a field list or SQL statement string or some such.

Armed thusly, you can then write a simple test module that calls your HTML blob handler with different HTML blobs and verifies that the correct junk is returned. You can also write a simple test fixture using the first chunk to simply grab a set of screens and write their HTML out to a test file (suitable for use with your first test fixture!).

Divide et impera!

--roboticus