You might try asking blakem if you can scrape thepen which is already search engine friendly.
| [reply] |
There's some general things to think of when pulling data off of perlmonks. The first is this: get as little data as you can without sacrificing your application. If you can get away with it, use the XML node view. (http://perlmonks.org/?node_id=NNN;displaytype=xml) If that isn't sufficent, consider adding in one of the other tickers, in this case, specificly, the thread xml generator, which will show you other nodes in the thread. If /that/ won't do it either, consider the print displaytype, or print with replies, if you need that.
Going along with getting as little data as possible, reget it as little as possible. Nodes don't generaly get updated much, so see if the node in question has already been fetched and use a cached copy if you can.
As others have said, doing your crawling when there's low load on PM would be nice. Unfornatly, there's just barely such a time. The best place I know for such a statistic would be the CB stats, which seem to be fairly flat over time.
If you do want to get all nodes, the best way is to iterate over the previous and next (possibly of type) links in the node navigator nodelet, I think. Unfornatly, that doesn't mix well with my above advice to avoid rendering the nodelets. You can also simply iterate over the node_id directly, but that has the disadvantage of hitting nodes that no longer exist. A quick check (quick for a pmdev, that is) shows that the maximum node_id is about 6% higher then the number of nodes -- IE that about 6% of nodes have somehow vanished. (Note that this does not include reaped nodes, even editor-deleted nodes, only god-nuked nodes -- the first two exist, but with different contents.)
Also, note that the xml viewtypes won't mix well with nodes that aren't "normal" (root nodes in one of the sections, or note nodes, replies). That is, for newest nodes, and suchlike, or (possibly?) for documentation nodes, the section main pages, the monestary gates, etc, they won't return the content, just the author, date, and suchlike of the node itself -- not what you wanted.
Warning: Unless otherwise stated, code is untested. Do not use without understanding. Code is posted in the hopes it is useful, but without warranty. All copyrights are relinquished into the public domain unless otherwise stated. I am not an angel. I am capable of error, and err on a fairly regular basis. If I made a mistake, please let me know (such as by replying to this node).
| [reply] |
Access the site only during off hours
And when should that be?
While you sleep, Monks in another part of the world
are busy at work :).
| [reply] |
While you sleep, Monks in another part of the world are busy at work :).
Yes, but if you look at percentages and trends, there are
off hours. As near as I can tell, they start about 5am
EDT, probably because that's 1am PDT, and the
California monks are going to sleep. Sure, the European
monks are all awake by then, but nevertheless the site
seems less active.
$;=sub{$/};@;=map{my($a,$b)=($_,$;);$;=sub{$a.$b->()}}
split//,".rekcah lreP rehtona tsuJ";$\=$ ;->();print$/
| [reply]
[d/l] |
| [reply] |
Fun is all nice and may further support your claim about small world theory, but what are you really aiming for?
I mean what type of positive goals that stats will have to help the community if at all possible.
There is http://perlmonks.thepen.com for the mirror of perlmonks.
{artist}
Update: My notes are more in positive 'notes' to encourage the direction of finding.
| [reply] |
I'd like to thank naChoZ and artist for pointing out the Perlmonks mirror. I had missed that in my super search.
My intial aim is to satisfy an academic curiosity about the structure of the Perlmonks. jcwren's site computes statistcs associated each monk. From a graph-theoretic point of view, the set of monks can be associated with the set of home pages, a special set of vertices in the Perlmonks graph, a set that can generate other vertices. While jcwren's site is a fun community resource that satisfies curiosity, ego, or competitive drive, it also computes some vertex structure of the Perlmonks network.
If jcwren computes vertex statistics, I'd like to look at the relationships between the vertices. Those relationships would include the population statistics mentioned previously and would also include quantities such as average number of replies to a question, whether reputation was dependent on position in the list of replies, and whether reputation was dependent on the number of previous links between questioner and responder.
The answers to such questions would be interesting academically, becasue network toplogy is a hot topic right now and many researchers are trying to discover the broad classes of behavior (physicists would call them universality classes) possible in real world networks. As I am an academic, that is my main motivation. The answers may also be interesting to monks who wonder about effects on reputation other than the quality of answer.
-Mark
| [reply] |
Fun is all nice and may further support your claim about small world theory, but what are you really aiming for?
I mean what type of positive goals that stats will have to help the community if at all possible.
I think you might be missing the point. Think of the quote at the Perl Monks Stats Page: "It's Like Crack for Statisticians" --Ozymandias
Given the number of Stat Addicts around here, I don't think anyone would complain about more stats. :)
| [reply] |