Reddit Front page is not a Meritocracy…. or is it?
On November 6th, Todd Schneider published a study about how Reddit’s front page is not a true meritocracy, which got a lot of people talking and even resulted in a front page Hacker News submission, Mashable republishing the study, and even the Washington Post writing about it.
Although the study is very interesting, it leaves out a very important factor, called ‘hotness’, which without really makes the assumptions made false.
On November 6th, the image below was submitted on Reddit by the user indeddit. The image appears to show how Reddit’s front page is not a meritocracy and how subreddits are clustered and not created equal.
In the study, the author explained his assumption that Reddit’s front page is not a pure meritocracy based on votes, and that subreddits themselves seem to follow a quota system that allocates certain subreddits to specific slots on pages one and two.
From the study:
“Much to my surprise, I found out that reddit’s front pages are not a pure “meritocracy” based on votes, but that rankings depend heavily on subreddits. The subreddits themselves seem to follow a quota system that allocates certain subreddits to specific slots on pages one and two, and also prevents the front page from devolving entirely into animal gifs. As a final kicker, in case it wasn’t completely obvious, I learned that links on the front pages of reddit receive a lot of traffic!”
He also classified the subreddits into 3 different clusters which where graphed and are explained below:
“Cluster 1 represents the most popular subreddits, like r/funny, which dominate the top of page one, but almost never show up on page two. Cluster 2 contains subreddits like r/personalfinance which dominate the bottom of page two, but very rarely make it to page one. Cluster 3 contains everything else: subreddits that don’t often make it to the top of page one, but aren’t stuck in page two purgatory either; cluster 3 subreddits typically represent the majority of posts at the bottom of page one and top of page two.”
A Deeper Look at the Study
When we first found the study on the 6th, we thought it was very interesting and started to debate it internally. Having been involved with Reddit for over 8 years, we are very familiar with the site and its code, so we started to notice some algorithmic elements that were missing from the study, resulting in our belief that the assumptions made by the study, that Reddit’s front page is not a meritocracy, was in fact false.
Lets take a look again at some quotes made by the author of the study:
“I found out that reddit’s front pages are not a pure “meritocracy” based on votes, but that rankings depend heavily on subreddits.”
“The subreddits themselves seem to follow a quota system that allocates certain subreddits to specific slots on pages one and two, and also prevents the front page from devolving entirely into animal gifs.”
His first statement about how rankings depend heavily on subreddits is ACCURATE, but his explanation on why is flawed.
The graph above shows some very interesting results. However, it is missing some important ranking factors that have to be included.
To get on the front page of Reddit, a submission’s total score and age is definitely considered. In his statement, he assumed that Reddit has a quota system that delimits a subreddit, thus, a submission to reach a front page. However, that is not the case.
Reddit’s front page has diversity not because it has a quota system per subreddit, but because of the ranking factor called “Hotness”
What is “hotness”?
Hotness is an algorithm factor where velocity comes into play. Basically, it is when a submission gets upvoted heavily and quickly.
This is where the ACTIVE subscribers of any subreddit have great effect. Take for example the top subreddit defined in each of the studies clusters when you add in active user counts:
- /r/aww – top 1 cluster 5mill subscribers (3804 user now)
- /r/documentaries – top 2 cluster 1.5mill (327 users now)
- /r/lifeprotips – top3 cluster 2.2 mill (652 users now)
A post with 3000 upvotes on /r/aww is not equal to 3000 upvotes on /r/documentaries or /r/lifeprotips
For example, 3000 votes on /r/aww maybe “hotter” because there are 3804 active users on that subreddit, resulting in much faster voting, while a post on /r/documentaries may have acquired the votes over X time.
Using the same example, we have to remember that these post scores are “fuzzed”, a term and practice Reddit uses to hide the true up and down votes that make up the submission’s score. The total score that we see is correct but it may mean:
- /r/aww – post has a score of 3000 = 3000 upvotes and 0 downvotes
- /r/documentaries – post has a score of 3000 = 6000 upvotes and 3000 downvotes
These factors make up ‘hotness’ and can a big impact in the ranking of submissions.
How Reddit Ranks Submissions
- Submission time is a very important parameter, generally newer stories will rank higher than older ones.
- The first 10 upvotes count as high as the next 100 (aka logarithmic voting). E.g. a story that has 10 upvotes and a story that has 50 upvotes will have a similar ranking.
- Controversial stories that get similar amounts of upvotes and downvotes will get a low ranking compared to stories that mainly get upvotes.(Source: http://amix.dk/blog/post/19588)
Or as Reddit admin /u/Deimorz explained it in Layman’s terms….
This is basically how a particular user’s front page is put together:
- 50 (100 if you have reddit gold) random subreddits from your subscriptions (or from the default subreddits for logged-out users and ones that haven’t customized their subscriptions at all) are selected. This set of selected subreddits will change every half hour, if you have more subscriptions than the 50/100 limit.
- For each of those subreddits, take the #1 post, as long as it’s less than a day old. Order these posts by their “hotness”, and then these will be the first X submissions on your front page, where X is the number of subreddits that have a #1 post less than a day old. So you get the top post from each subreddit before seeing a second one from any individual subreddit.
- The remaining submissions are ordered using a “normalizing” method that compares their scores to the score of the #1 post in the subreddit they’re from. This makes it so that, for example, a post with 500 points in a subreddit where the top post has 1000 points is ranked the same as one with 5 points where the top has 10.
Are subreddits created equal?
The simple answer is YES!
The long version: Subreddits are created equal, but there are still other factors that are not equal which are based on Redditors.
- Active Subscribers
- Types of posts submitted
Other than the ‘hotness’ which is determined by voters, there is another factor that comes into play for why /r/aww tends to rank better than /r/documentaries: TYPE OF POSTS!
“A picture is better than a thousand words”
Images are the easiest type of content to consume, which explains why Imgur, the preferred image hosting service of Redditors, generates 2 billion daily image views.
Still a Very Interesting Study
We obviously feel the studies assumptions are false, due to the omission of active user counts and ‘hotness’, but there is still some interesting elements that the study brought to light and it is great to see that people are that dedicated to understanding Reddit and how it works.
Even Admin /r/deimorz had the following observation:
“As for why certain subreddits seem to almost always be on a particular page, this isn’t actually something that’s been specifically defined.
It’s definitely interesting that it’s almost always the same set, but looking at which subreddits fell into which categories, it seems to mostly be a function of some combination of how old the subreddit is, how long it’s been a default, how much traffic or how many subscribers it has, and how well the content from it satisfies some of the biases of reddit’s hot algorithm (things that are quick to view, simple to understand, and non-controversial tend to do best).
So subreddits like /r/mildlyinteresting will almost always have their #1 post be in the top half of the eligible #1s (and thus on the first page) just because their posts are very quick, somewhat amusing images, which generally do very well.” – /u/Deimorz (Reddit Admin)
(NOTE that we reached out to the author of the study, Mashable, and the Washington Post, but have not heard from any of them about our findings)