Afternoon

I'm a big fan of Mechanize as well, enough so that I started porting it to ruby. I used ruby-htmltools (think HTML::Tree), REXML (XPATH & XML trees) and HTTP Access2 (LWP). Building it using this set of libraries gave me a few ideas for W:M-

At the moment, WWW::Mechanize doesn't try to do any general parse; it uses Tokeparser to rip out links and forms, HeadParser to do the head, etc, but doesn't really care much about the rest of the doc. One thing I found in porting, using a generic HTML parsing tool, then converting the result to an XML tree was that I could very easily pick out bits of content from the page. REXML allows the use of XPATH or XML::Twig/HTML::TreeBuilder-type scans of the parse tree, making it very easy to pick out, say, the title, or the number of <LI> elements with class 'green' in the <UL> tag with the id attribute 'foo'.

Though I've never tried it, I'm sure it would be easy to use HTML::TreeBuilder to do something similar with WWW::Mechanize (IIRC it relies on HTML::Parser, too). Then, in your subclass of W:M, you have lovely convenience methods like $mech->title(). Oh, actually W:M has that already .... hmm, how about $mech->keywords()? Or $mech->get_element_by_id('foo')?. I dunno if this was the kind of thing you were thinking of, but I think it does extend the range of possibilities.

cheers
ViceRaid