An experiment: gaming Slashdot's moderation system

Or: why moderation systems are trickier than they look.

Understanding the dynamics of online communities is one of my pet interests, so as a regular reader of Hacker News, I took notice when Paul Graham started experimenting recently in an attempt to stave off the gradual but unmistakable decline (AKA redditization) of what used to be my favorite online community after TurnKey.

The discussion inspired me to write a blog post about an experiment I tried a while back at my other geek haunt - Slashdot. Just to throw in my two cents.

In theory comment moderation systems are democratic and promote a high signal to noise ratio. But I've long suspected them of promoting group-think and being easy to game once you understand a little bit about the dynamics at play.

Let me back up a bit and tell how it all started. Slashdot have a pretty sophisticated moderation system that scores comments from -1 to +5. I like reading comments (for some stories), but for hundreds of comments it's just not practical without some kind of filtering, and I figured asking Slashdot to filter everything below +5 would be a good way to get straight to the good stuff which I assumed would bubble naturally to the top.

But then this one time I put my heart into a comment (where I usually just lurk) and was a bit surprised that it was promptly buried by the moderation system. It wasn't censored so much as drowned out by the noise. Sort of a censorship by glut.

This got me thinking about the dynamics of moderation and motivated me to take a closer look at one of the most famous and supposedly sophisticated moderation systems online to see if I could spot any patterns.

I came up with a few ideas regarding potential influencing factors that would skew the system and then ran a brief ad-hoc experiment to test my hypothesis. With interesting results.

6 out of 7 test posts which I submitted according to "the rules" received a perfect +5. The 7th message received a +4. From start to finish I had boosted my karma up 5 levels from neutral to excellent (the highest possible score) in 48 hours. Having "Excellent" Karma gives you a bonus point so all your posts now start from +2. This makes it even easier to game the system. Too easy really. I ended the experiment.

Common artifacts that can be used to "game" a moderation system:

  • Old boys club effect: On Slashdot it seemed pretty obvious that posts by users with low user ids have a much higher chance of getting moderated up. I think this is because of how Slashdot's moderation algorithms put most moderation points into the hands of old-timers which are more likely to moderate their own kind up.

  • First posts: there seems to be a powerful snowball effect on Slashdot whereby posts that are submitted early are much more likely to make it to a high score compared with another post of equal merit submitted later.

    Especially on low traffic stories it is quite common for a high-quality comments to not get moderated up. This is meaningful because high-scoring posts are much more likely to be read, due to the sheer volume of low-scoring posts.

I also noticed that posts have a higher chance of being moderated up if they are:

  • Promoting political viewpoints which align with popular sentiment in the community (e.g., libertarian).
  • Brief: there seems to be an inverse correlation between moderation points and length of posts. Short, concise posts are more likely to be moderated up. Probably because they are more likely to be read and enjoyed in the first place.
  • Have good grammar and no speling mistakes.
  • Not "selling" anything. Slashdot moderators are instinctively hostile to marketing. A post that seems (even superficially) to promote someone's commercial interests is far less likely to receive moderation points.

Details of the experiment

  • Low user id, check: to test my theory on low user ids I dug around for the password to my old 5-digit Slashdot account from 1998. Note I hadn't used it in many years so there couldn't be any reputation associated with it aside from the low user id.

  • Early posting, check: I payed $5 for a subscription which gave me a head-start on most stories before they show up to non-subscribers.

    You can't post a comment until the story goes live but being able to see the story minutes in advance still gives you huge advantage because you can use that time to draft one of the first posts and beat a non-subscriber to the punch.

    First post are not only much more likely to receive moderation points, they are also much more likely to be read.

In combination these simple techniques guaranteed that nearly everyone who clicked through to the comments would read my posts.

Easiest way to a high score

Aspiring Karma whores take note: all my posts save one last +4 received a perfect +5 score but some took more effort than others and took longer to reach the maximum score.

Posts that were moderated as informative got there faster and took less effort so all it took to snowball my way to +5 was to show up early, do a quick Google search and write up a few resources that complimented the story somehow.

Comments

Jake's picture

I feel the same is also true for the StackExchange family of sites. Answers from higher reputation users get upvoted alot more, and not because they are necessarily good answers.

Liraz Siri's picture

I just finished reading Malcolm Galdwell's outliers and it seems the snowball effect underlies much of success in general. Even very small initial advantages can accumulate to big differences in outcomes.

Pages

Add new comment