Beefy Boxes and Bandwidth Generously Provided by pair Networks
Keep It Simple, Stupid
 
PerlMonks  

Best way to traverse all Perlmonks nodes?

by kvale (Monsignor)
on Sep 24, 2003 at 19:24 UTC ( [id://293974]=monkdiscuss: print w/replies, xml ) Need Help??

Information currently gathered by jcwren at the Perlmonks site about users, experience, location, etc. is fun and interesting.

At some point, I'd like to gather information about Perlmonk network statistics, where I mean network in the graph-theoretic sense. It seems plausible to me, for instance, that for reasonable notions of connectivity of node, Perlmonks forms a small-world network . It would be fun to compute vertex degree distributions, average distance, etc., and compare against other online communities.

To do this, however, would require traversing all the nodes of the Perlmonks site and recording threads and perhaps scores as edge attributes.

Doing so in a simple-minded manner would probably get me banned, and rightly so, as I would be torturing an already overburdened server.

I ask for your advice on the gentlest manner in which to probe Perlmonks. Some possibilities I have thought of are

  • A slow spider that accesses the site every x number of minutes
  • Access an off-site mirror of Perlmonks - is there one?
  • Access the site only during off hours (as does jcwren)
  • Download a snapshot archive of the perlmonks site - is there one? It doesn't have to be current.
Or is such an idea simply ill-advised and should be dropped?

-Mark

Replies are listed 'Best First'.
Re: Best way to traverse all Perlmonks nodes?
by naChoZ (Curate) on Sep 24, 2003 at 19:51 UTC
Re: Best way to traverse all Perlmonks nodes?
by theorbtwo (Prior) on Sep 25, 2003 at 05:31 UTC

    There's some general things to think of when pulling data off of perlmonks. The first is this: get as little data as you can without sacrificing your application. If you can get away with it, use the XML node view. (http://perlmonks.org/?node_id=NNN;displaytype=xml) If that isn't sufficent, consider adding in one of the other tickers, in this case, specificly, the thread xml generator, which will show you other nodes in the thread. If /that/ won't do it either, consider the print displaytype, or print with replies, if you need that.

    Going along with getting as little data as possible, reget it as little as possible. Nodes don't generaly get updated much, so see if the node in question has already been fetched and use a cached copy if you can.

    As others have said, doing your crawling when there's low load on PM would be nice. Unfornatly, there's just barely such a time. The best place I know for such a statistic would be the CB stats, which seem to be fairly flat over time.

    If you do want to get all nodes, the best way is to iterate over the previous and next (possibly of type) links in the node navigator nodelet, I think. Unfornatly, that doesn't mix well with my above advice to avoid rendering the nodelets. You can also simply iterate over the node_id directly, but that has the disadvantage of hitting nodes that no longer exist. A quick check (quick for a pmdev, that is) shows that the maximum node_id is about 6% higher then the number of nodes -- IE that about 6% of nodes have somehow vanished. (Note that this does not include reaped nodes, even editor-deleted nodes, only god-nuked nodes -- the first two exist, but with different contents.)

    Also, note that the xml viewtypes won't mix well with nodes that aren't "normal" (root nodes in one of the sections, or note nodes, replies). That is, for newest nodes, and suchlike, or (possibly?) for documentation nodes, the section main pages, the monestary gates, etc, they won't return the content, just the author, date, and suchlike of the node itself -- not what you wanted.


    Warning: Unless otherwise stated, code is untested. Do not use without understanding. Code is posted in the hopes it is useful, but without warranty. All copyrights are relinquished into the public domain unless otherwise stated. I am not an angel. I am capable of error, and err on a fairly regular basis. If I made a mistake, please let me know (such as by replying to this node).

Re: Best way to traverse all Perlmonks nodes?
by cchampion (Curate) on Sep 24, 2003 at 20:02 UTC
    Access the site only during off hours

    And when should that be?

    While you sleep, Monks in another part of the world are busy at work :).

      While you sleep, Monks in another part of the world are busy at work :).

      Yes, but if you look at percentages and trends, there are off hours. As near as I can tell, they start about 5am EDT, probably because that's 1am PDT, and the California monks are going to sleep. Sure, the European monks are all awake by then, but nevertheless the site seems less active.


      $;=sub{$/};@;=map{my($a,$b)=($_,$;);$;=sub{$a.$b->()}} split//,".rekcah lreP rehtona tsuJ";$\=$ ;->();print$/
Re: Best way to traverse all Perlmonks nodes?
by zby (Vicar) on Sep 25, 2003 at 10:09 UTC
Re: Best way to traverse all Perlmonks nodes?
by artist (Parson) on Sep 24, 2003 at 21:03 UTC
    Fun is all nice and may further support your claim about small world theory, but what are you really aiming for?

    I mean what type of positive goals that stats will have to help the community if at all possible.

    There is http://perlmonks.thepen.com for the mirror of perlmonks.

    {artist}

    Update: My notes are more in positive 'notes' to encourage the direction of finding.

      I'd like to thank naChoZ and artist for pointing out the Perlmonks mirror. I had missed that in my super search.

      My intial aim is to satisfy an academic curiosity about the structure of the Perlmonks. jcwren's site computes statistcs associated each monk. From a graph-theoretic point of view, the set of monks can be associated with the set of home pages, a special set of vertices in the Perlmonks graph, a set that can generate other vertices. While jcwren's site is a fun community resource that satisfies curiosity, ego, or competitive drive, it also computes some vertex structure of the Perlmonks network.

      If jcwren computes vertex statistics, I'd like to look at the relationships between the vertices. Those relationships would include the population statistics mentioned previously and would also include quantities such as average number of replies to a question, whether reputation was dependent on position in the list of replies, and whether reputation was dependent on the number of previous links between questioner and responder.

      The answers to such questions would be interesting academically, becasue network toplogy is a hot topic right now and many researchers are trying to discover the broad classes of behavior (physicists would call them universality classes) possible in real world networks. As I am an academic, that is my main motivation. The answers may also be interesting to monks who wonder about effects on reputation other than the quality of answer.

      -Mark

      Fun is all nice and may further support your claim about small world theory, but what are you really aiming for?

      I mean what type of positive goals that stats will have to help the community if at all possible.

      I think you might be missing the point. Think of the quote at the Perl Monks Stats Page: "It's Like Crack for Statisticians" --Ozymandias

      Given the number of Stat Addicts around here, I don't think anyone would complain about more stats. :)

      --
      "I just read perlman:perlboot," said Tom, objectively.
      naChoZ

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: monkdiscuss [id://293974]
Approved by Paladin
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others making s'mores by the fire in the courtyard of the Monastery: (6)
As of 2024-03-28 19:20 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found