Here's the code. (Note: I'm truncating the data for conciseness.)

#!/usr/bin/perl -wl

use strict;
use File::Temp qw( tempfile );

my %rep_stats=(1=>24694, 2=>23551, 0=>22855, ... -41=>1, );

# only keep 0 < XP < 100 because we want the more mainstream
# values and not the far-out ones

my @xp_sorted = grep { $_>0 && $_<100 } sort { $a <=> $b }
                keys %rep_stats;

# generate our R commands

(my $r_commands = <<EOF) =~ s/^    //mg;
    xp <- scan()
    @xp_sorted

    count <- scan()
    @{[ @rep_stats{@xp_sorted} ]}

    summary(lm(log10(count) ~ xp + I(xp^2)))
EOF

# now, we stuff our R commands into a tempfile,
# which we'll use as STDIN

my $tmp = tempfile()
    or die "can't open tempfile: $!";
print $tmp $r_commands;
seek $tmp, 0, 0 or die "can't seek to BOF: $!";
open STDIN, ">&", $tmp or die "can't dup tmp->STDIN: $!";

# finally, we exec R, which will read our commands
# from STDIN (the temp file will be deleted automatically
# when the program exits)

my @cmd = qw(R --no-save --no-init-file --no-restore-data --slave);
exec @cmd;
die "couldn't exec @cmd : $1";  # should never get here
[download]

Read 99 items
Read 99 items

Call:
lm(formula = log10(count) ~ xp + I(xp^2))

Residuals:
      Min        1Q    Median        3Q       Max 
-0.111459 -0.032358  0.004959  0.024387  0.109470 

Coefficients:
              Estimate Std. Error t value Pr(>|t|)    
(Intercept)  4.442e+00  1.239e-02  358.49   <2e-16 ***
xp          -4.194e-02  5.719e-04  -73.33   <2e-16 ***
I(xp^2)      1.467e-04  5.541e-06   26.48   <2e-16 ***
---
Signif. codes:  0 `***' 0.001 `**' 0.01 `*' 0.05 `.' 0.1 ` ' 1 

Residual standard error: 0.04027 on 96 degrees of freedom
Multiple R-Squared: 0.9975,    Adjusted R-squared: 0.9974 
F-statistic: 1.889e+04 on 2 and 96 DF,  p-value: < 2.2e-16
[download]

sub estimate_count_from_xp($) {
    my $xp = shift;
    10 ** ( 4.442 - 4.194e-2 * $xp + 1.467e-4 * $xp**2 );
}
[download]

Just to see how good our model is, take a look at this plot comparing the actual values (dots) versus the estimated values (line). That's pretty much "on the money."

Cheers,
Tom

++ for the technique. But the result seems pretty strange - the 1.467e-4 * $xp**2 is the dominant part in the exponent so it seems that the count should grow with the xp from some point.

You're right that the quadratic term will, eventually, dominate. However, for the range I considered (0 < XP < 100), adding that term results in a slightly better fit.

But, the fit is nearly as good without it, and so for interpretive purposes (instead of get-the-best-fit purposes), dropping the quadratic term makes for a better model:

Read 99 items
Read 99 items

Call:
lm(formula = log10(count) ~ xp)

Residuals:
     Min       1Q   Median       3Q      Max 
-0.16823 -0.10095 -0.01733  0.07757  0.25553 

Coefficients:
             Estimate Std. Error t value Pr(>|t|)    
(Intercept)  4.194851   0.023379  179.43   <2e-16 ***
xp          -0.027268   0.000406  -67.17   <2e-16 ***
---
Signif. codes:  0 `***' 0.001 `**' 0.01 `*' 0.05 `.' 0.1 ` ' 1 

Residual standard error: 0.1154 on 97 degrees of freedom
Multiple R-Squared: 0.979,    Adjusted R-squared: 0.9787 
F-statistic:  4512 on 1 and 97 DF,  p-value: < 2.2e-16
[download]

With this model, our estimating function is as follows:

sub estimate_count_from_xp($) {
    my $xp = shift;
    10 ** ( 4.195 - 0.2727 * $xp );
}
[download]

From this, it's easy to see that we have classic exponential decay w.r.t. XP.

Does this match your intuition?

Tom Moertel : Blog / Talks / LectroTest / PXSL / Coffee / Movie Rating Decoder

use List::Util 'sum';

my $sum     = sum values %rep_stats;
my $entropy = sum map { -($_/$sum) * log($_/$sum) / log(2) }
                  values %rep_stats;

printf "Total entropy: %.05f\n", $entropy;
[download]

Being that $NORM is also a measurement of the trend of node reputation, how is $entropy useful as an ancillary view of the same trend?

$NORM measures the average reputation of recent nodes. It answers the question, "Are nodes rated high or low?"

Entropy measures the information content of node reputation. It answers the questions, "How much does the node's reputation tell us? How meaningful is the assignment of reputation?"

Say $NORM is 11. Well, this can happen if all recent nodes have reputation 11. If this is the case, the entropy is 0 because knowing that a node has reputation 11 tells us nothing about the node.

On the other hand, maybe among all recent nodes, an equal number of them have reputation 1, 2, 3, .. up to 22. This situation also gives us $NORM = 11. But here, knowing the reputation of a node gives us much more information. Reputation in itself is more meaningful in this scenario because it can tell us something. The something it is telling us is information in the theoretical sense.

$NORM tells us whether nodes are given high or low reputations on average (although the variance might be useful to know as well). It is an analysis of the values of a random variable. Entropy is completely orthogonal; independent of how high or low the nodes are ranked, it tells us how informative node reputation really is. It is an analysis of the uncertainty of a random variable. You can have any combination of low or high average with low or high entropy.

blokhead

print <<EOH
Content-type: text/html

<html><body>
<a href="http://www.quotegarden.com/statistics.html">Results</a>
</body></html>
EOH
[download]

Heh, nice link. :-) Its particularly amusing to me because that second quote is reduced version of a quote I know very well. A friend of my family introduced me to computers at a very young age (or it was for the time, not nowadays of course) as part of his post-doc work in using computers as an educational medium. He used to a have a line printer terminal in his office (where I got my one and only chance to mess with APL), on the wall in front of it he had a print out that said (figlet style)

If you torture the data long enough it will confess to anything

Hes been a professor of psychology focusing on statistical methods of representing data for a long time and I suppose that quote was something important to him. All i can say is its a line I'll never forget.



---
demerphq


First they ignore you, then they laugh at you, then they fight you, then you win.
-- Gandhi

Flux8

That is a great quote. I took some time too look, and it appears to originate from Ronald Coase, a British economist. So now you have an attribution. :-)

Makeshifts last the longest.

Here's one that figure out the total ammount of node XP on PM, how many days it would take a Saint to get that many votes, and the position of the total number in pi (thanks to LWP::UserAgent and http://www.angio.net/pi/piquery).

my $PI_URI = 'http://www.angio.net/pi/bigpi.cgi';

# Fill this in with the rep stats hash from parent node
my %REP_STATS = ( . . . );

my $total_rep;
while( my ($key, $value) = each %REP_STATS ) {
        $total_rep += $key * $value;
}

my $saint_days = $total_rep / 40;

my $in_pi = do {  
        use LWP::UserAgent;
        my $ua = LWP::UserAgent->new;
        
        # Couldn't get proper param passing to work, so work-around 
        # by making the query string ourselves.  Fix later.
        # 
        my $response = $ua->get( $PI_URI . "?UsrQuery=$total_rep" );
        local $_ = $response->content;
        m!The string <B>\Q$total_rep\E</b> was found at position (\d+)
+!;
        
        $1;
};      


print "Total XP: $total_rep";
print "Saint Days: $saint_days";
print "In pi at: $in_pi";
[download]

For the record, I got:

Total XP: 4068710
Saint Days: 101717.75
In pi at: 3246447
[download]

Here is a bit of code that does some basic graphing. Here is the graph. The moving averages all show the same thing. There is an unexpected data anomoly around 155-165 XP. Note I have jacked the moving averages up by 0.5 to separate them cleanly from the raw data.

Nice job!

But I cannot run it... I am on a winders machine, and ActiveState doesnt have the GD package for some reason.

I also don't have a c compiler to build it.

Oh well, I will have to be contect to admire from afar...

For now.

Whoever comes up with the coolest analysis ... will win some sort of prize

If it means I get to see, "you have -4 XP to level vroom", I'm all for it.

"There is no shame in being self-taught, only in not trying to learn in the first place." -- Atrus, Myst: The Book of D'ni.

TheEnigma

Average node value is 11.87. The median value (as many nodes have lower or higher values as the median) is between 7 and 8. 90% of the nodes have a value between 0 and 40 XP and 99% of the nodes have a value between -8 and 90. Both ranges are centered between the extremes of -223 and 571 XP, i.e. 5% of the nodes has a lower XP than 0 or a higher XP than 40, resp. 0.5% is worse than -8 or better than 90.

The XP distribution is not a standard Bell shaped Gaussian distribution, but something which peaks around the average value and quickly drops down to low values, with long low tails to lower and higher values.

What does it tell us:

• average and median are close together and most of the nodes cluster around these values: most of the nodes are thus of average value (for whatever meaning you may give to average).
• Perlmonks are quicker to give positive XP than negative XP as 95% of the nodes have no negative XP.
• Relatively speaking few nodes are very bad or very good (only 1% is worse than -8 or better than 90), but even here we are quicker to praise than to chastise (the positive tail goes much higher than the negative tail goes deeper, for an equal number of nodes).

I don't think you need statistics to draw that conclusion for you. Personally I feel safe to assume from my experience with the site generally being pleasant and informative that it is because nodes tend to be well written.

Or rather, I should say, not bad. As you saw from your analysis, the overwhelming majority of nodes end up with a handful of upvotes. To state it in a cold and detached way, that means that the average node is a tendentially better than entirely useless.

I like Perlmonks. It's a comfortable community of geeks.

Makeshifts last the longest.

To make some even more interesting analysis, it would be good to have the same data broken down to sections and node types.

• Node Rep
• Node Type
• Create Time
• Front Paged

I dont get it. You want a list of this information for all ~400k records we have in the DB? How is that supposed to happen?



---
demerphq


First they ignore you, then they laugh at you, then they fight you, then you win.
-- Gandhi

Flux8

demerphq,
I can see where making this information available in a database (as simple as SQLite for instance) offsite being a policy problem. With a couple slight modifications, it might be a bit more feasible:
• Forget the frontpaged flag
• Break things out in buckets by day, such as 2004-12-25
• The next level of buckets would be node type
• The last level of buckets would be reps and corresponding counts
I was expecting to do the breakout work off-site though.

Cheers - L~R