There's some general things to think of when pulling data off of perlmonks. The first is this: get as little data as you can without sacrificing your application. If you can get away with it, use the XML node view. (http://perlmonks.org/?node_id=NNN;displaytype=xml) If that isn't sufficent, consider adding in one of the other tickers, in this case, specificly, the thread xml generator, which will show you other nodes in the thread. If /that/ won't do it either, consider the print displaytype, or print with replies, if you need that.

Going along with getting as little data as possible, reget it as little as possible. Nodes don't generaly get updated much, so see if the node in question has already been fetched and use a cached copy if you can.

As others have said, doing your crawling when there's low load on PM would be nice. Unfornatly, there's just barely such a time. The best place I know for such a statistic would be the CB stats, which seem to be fairly flat over time.

If you do want to get all nodes, the best way is to iterate over the previous and next (possibly of type) links in the node navigator nodelet, I think. Unfornatly, that doesn't mix well with my above advice to avoid rendering the nodelets. You can also simply iterate over the node_id directly, but that has the disadvantage of hitting nodes that no longer exist. A quick check (quick for a pmdev, that is) shows that the maximum node_id is about 6% higher then the number of nodes -- IE that about 6% of nodes have somehow vanished. (Note that this does not include reaped nodes, even editor-deleted nodes, only god-nuked nodes -- the first two exist, but with different contents.)

Also, note that the xml viewtypes won't mix well with nodes that aren't "normal" (root nodes in one of the sections, or note nodes, replies). That is, for newest nodes, and suchlike, or (possibly?) for documentation nodes, the section main pages, the monestary gates, etc, they won't return the content, just the author, date, and suchlike of the node itself -- not what you wanted.