I think my explanation fits rather well with the:

# sorry, but misbehaved robots have ruined it for all of you.
[download]

And sure good bots probably are smart like you said, im just saying that even if it was a naive depth first search of minimal depth, its still going to put a heavy load on the server.