Hi guys. I've been following speedrunning for several years now (though I don't actually run myself) and am particularly interested in glitches in games.
Since I love speedrunning and statistics I made a rankings website that aggregates results across the speedrun.com site. It's been online for a few months now so I thought I'd post about it here as my introduction. Basically, points are awarded for the primary single-player category per game based on a combination of the popularity and the competitiveness of the game. The link is below if anyone's interested.
SpeedRank news - August 5, 2019
The last few weeks have seen the top 3 spots in the rankings shuffle around a few times, but an otherwise static top 10. This week, everything changes with Marlin and tavenwebb2002 jumping into the top 10 after setting world records in Celeste and Super Mario Bros. respectively. cheese regains the top spot to rule the rankings for the 6th time, while BrawlerJesse and Lui reach new best ranks.
I don't know how you're determining the primary category, but there seems to be an issue with that. World Peace is being considered the primary category for SMO when it should be Any%. On that note, having your whole score in a game based on a single (somewhat arbitrarily chosen) category doesn't seem like a good idea. It kinda shafts people who run a lot of different categories in the same game, or people focusing on categories other than the "primary" one (or people who can't run the primary category because of a version split).
Another weird thing I've noticed, I'm getting 5 points from my Arkham City Any% WR (11 runners in the category, 43 for the whole game), and yet this person who has the Any% WR in Arkham Asylum (significantly more popular game, 45 runners in the category, 57 overall) isn't getting any points from that. That doesn't add up.
Can you go into more detail on how exactly the points are calculated?
Hi, thanks for checking out the site. I really appreciate the feedback.
I am well aware of the issue with SMO. It is because the leaderboards for that game have been set up in a nonstandard way. Basically, the game is using a custom variable to split leaderboards based on player count instead of the one defined in the API. Since I am only looking for single player games, it is choosing the wrong category because of this. Unfortunately I can't write a special case for every game and it currently works for 99% of games on the site. So basically bad data in -> bad data out.
The choice for primary category is the highest priority single player category for each game (as defined by the mods for that game). Categories appear in priority order on the site, so it's generally the one you would expect. Only one category is used because it would take too long to collect data for everything and doing multiple categories for games would mean reducing the number of games surveyed.
The reason no points are awarded this week for Arkham Asylum is because (again, for time reasons) only the top 400 most popular games on the site (according to active player count) are included in the rankings. Arkham Asylum is currently outside the top 400, while Arkham City is inside.
[quote=GoldenFox]it currently works for 99% of games on the site[/quote]
I think it not working properly for one of the most popular categories ever, on one of the most popular speedgames ever is a pretty serious issue (especially since this whole system is based on popularity). It's probably not an easy fix, but I don't think such a significant data point can just be overlooked because the system "mostly works" otherwise.
[quote=GoldenFox]only the top 400 most popular games on the site (according to active player count) are included in the rankings[/quote]
Bleugh. I understand that there are constraints to this and that doing literally every game isn't feasible, but... that's less than 2.5%. Plenty of reasonably popular games don't even get considered, which is pretty lame if you run one of those games that isn't super popular right now.
Speaking of which, having the whole thing based on active players seems problematic. The shifts in ranking might just due to changing interest in games, in which case this feels more like "who's the best at the popular games this week" than a meaningful ranking system.
I don't know. It's just kinda not working for me. Some of that is outside your control, but even those constraints aside I'm not sure how good of a system this would be.
[quote=ShikenNuggets]The shifts in ranking might just due to changing interest in games, in which case this feels more like "who's the best at the popular games this week"[/quote]
That's actually what I was aiming for, but I understand it's not for everyone. It is still possible to gain higher ranks by being good at multiple less popular games though. EndySWE was top ten in the ranks by having top 20 placings in multiple less-popular games.
Thanks for the feedback. I know the appeal is limited since every game has its own community and I think most people would prefer to be a "big fish in a small pond" so to speak. I just thought it might help to grow the sport overall if there were some unified system to recognise and quantify top runners. It makes it easier for an individual (or even the media) who is unfamiliar with speedrunning to easily find out who the more successful (and, to some extent, popular) runners are and watch their runs.
I found with while working on my app, its actually quite easy to get a dump of all 900k runs on the site in less than 2 hours with embedded players. With the games batch retrieved you could probably generate your statistics for all games pretty easily that way without having to scan the way i assume you would be now.
I'm someone who likes data for data's sake, so I'm happy you are attempting what you are doing, even though my page (http://speedrank.xyz/player.php?name=nihilistcomedyhour) seems to be... a peculiar outcome giving the games I've run, etc.
"active player count" seems to be a weird thing to go by as well, given that many games probably fall under the top 400.. (Like Fallout 3, Fallout: New Vegas, Super Mario Bros. 2, Sonic 2) but are heavily optimized. Example Sonic 2 has 77 rankings, but only 5 active players. This would mean nothing to anyone ranked, but Super Punch Out (26 rankings, 6 active players) would be.
I don't really know of ideal set up to actually get vaguely close to anything that would be considered "ranked overall speedrunners" given the way this site stores and breaks up categories, and naturally the is stuff like BrawlerJesse (http://speedrank.xyz/player.php?name=BrawlerJesse) being ranked 3 for being number one in a "full-cat" speedrun that is actually the first few races, and not the entire game versus being Rank 16 (http://speedrank.xyz/player.php?name=Torje) for having the fastest Ocarina of Time, an incredibly more popular game, but, I do find it very interesting none the less.
Going up to the discussion regarding the SMO board, I'd expect far more than 1% of the games on this site are going to have that exact issue, if not some other issue pertaining to categories and subcategories. You'd likely get a far more accurate semblance of ranking going by sub cat instead of an entire game, but I wouldn't claim to know what ultimately arbitrary set up would be more accurate, or how to even go about measuring that.
I have now added the ability to easily add special cases to my code, and have added one for SMO, which will be reflected in Monday's update. If you do find any other instances then just let me know.