If anyone is interested in helping do something with this, I could provide the rather amateurish, taped together, spider code I've managed to get running.

Any help or advice on how to improve this would be much appreciated.

Tom