There's some general things to think of when pulling data off of perlmonks. The first is this: get as little data as you can without sacrificing your application. If you can get away with it, use the XML node view. (http://perlmonks.org/?node_id=NNN;displaytype=xml) If that isn't sufficent, consider adding in one of the other tickers, in this case, specificly, the thread xml generator, which will show you other nodes in the thread. If /that/ won't do it either, consider the print displaytype, or print with replies, if you need that.

Going along with getting as little data as possible, reget it as little as possible. Nodes don't generaly get updated much, so see if the node in question has already been fetched and use a cached copy if you can.

As others have said, doing your crawling when there's low load on PM would be nice. Unfornatly, there's just barely such a time. The best place I know for such a statistic would be the CB stats, which seem to be fairly flat over time.

If you do want to get all nodes, the best way is to iterate over the previous and next (possibly of type) links in the node navigator nodelet, I think. Unfornatly, that doesn't mix well with my above advice to avoid rendering the nodelets. You can also simply iterate over the node_id directly, but that has the disadvantage of hitting nodes that no longer exist. A quick check (quick for a pmdev, that is) shows that the maximum node_id is about 6% higher then the number of nodes -- IE that about 6% of nodes have somehow vanished. (Note that this does not include reaped nodes, even editor-deleted nodes, only god-nuked nodes -- the first two exist, but with different contents.)

Also, note that the xml viewtypes won't mix well with nodes that aren't "normal" (root nodes in one of the sections, or note nodes, replies). That is, for newest nodes, and suchlike, or (possibly?) for documentation nodes, the section main pages, the monestary gates, etc, they won't return the content, just the author, date, and suchlike of the node itself -- not what you wanted.

Warning: Unless otherwise stated, code is untested. Do not use without understanding. Code is posted in the hopes it is useful, but without warranty. All copyrights are relinquished into the public domain unless otherwise stated. I am not an angel. I am capable of error, and err on a fairly regular basis. If I made a mistake, please let me know (such as by replying to this node).

In reply to Re: Best way to traverse all Perlmonks nodes? by theorbtwo
in thread Best way to traverse all Perlmonks nodes? by kvale

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.