Perhaps you should see how another search does these things. Htdig (http://www.htdig.org) is the first that comes to mind, although I can't vouch for its value as an example.