to a new server and plan to rebuild the database into subcategories. Spidering dynamic content (as mentioned defore) is not a problem for the scripts I use.
and I am wondering if I should include a perl docs category, and spider sites like perlmonks.org and cpan etc. Thoughts??