Sounds like fun.

If you have URL access to the data, can you just walk through the tree, collecting and dumping data as you go? Then the dump could form the basis for the web application database.

Things get a bit more complicated if this is going to be a continuous process, rather than a one-time dump. Once the process is better understood, we can start talking about tools, although I doubt some module is going to be a silver bullet.

Tell me more.