The recent "performance improvement" changes to the site have broken the API and nobody seems to care. I'm posting here since the issue on the GitHub repo is being ignored https://github.com/speedruncomorg/api/issues/125
Due to this bug it's impossible to obtain every run from the most popular games the intended way rendering many projects/tools/bots unusable. Trying to work around this bug by, for example, requesting runs per category results in lost runs from deleted categories, slower results, hitting the request limit faster and won't help once categories grow past 10000 runs.
I understand that the API doesn't give ad revenue but it's an important tool for many communities and needs to be kept alive.
Hi Kohru! We definitely hear you on the API being an important tool for the communities. We have it on our road map to review the API as well as the outstanding GitHub issues in order to see what we can do to balance site optimization with community development needs. For now we've unfortunately needed to restrict non-filtered run requests to 10,000, but we appreciate the feedback that the request limit is too low for your needs and we can include that in our future evaluation.
Hi @SpeedrunCom,
Thanks for the speedy reply appreciate it. We are seeing the restriction on filtered requests as well though, like if we filter by game or by game and category we get the 10k limit. Is that working as intended?
Hi @mgmedick,
I just checked internally and to clarify all requests to the runs API endpoint are limited to 10,000 results. If an individual needs to paginate past that then they may want to apply more specific filters. If anyone feels they need more than this (or has other requests), feel free to submit it through the Feedback section of the Support Hub, all feedback is logged under the relevant feature and taken into account when we review that feature.
There's clearly some misunderstanding going on, the API already has a throttling system to limit traffic and prevent abuse, this is what should be tweaked if needed.
The bug doesn't prevent us from requesting more than 10k results at once since that was always impossible, requests are limited to a maximum of 200 results. What it prevents is accessing results with an offset of >=10k.
This means that if I want to do something as simple as graph the number of runs submitted over time of a popular game I can't because there's no straightforward way to obtain every run of that game (even if I request 200 runs every 30 minutes).
If the database is somehow actually dying from reading past 10k rows for the rate-limitted API users then enable cursor-based pagination (which every framework supports) instead of silently breaking the API
Hi @SpeedrunCom,
Yes bumping up the limit to a threshold higher than the verified run count of the largest category would help greatly. For example "Super Mario Odyssey - Any%", however many verified runs are under that category + 5k for growth would do the trick. Currently the API can't pull all the runs for that category, and we can't narrow the results by filtering any further (because we can't filter by variable).
I will submit request through Support Hub.
I have been having this issue while trying to count how many runs a moderator has verified. I think I have found a strategy for "increasing" the limit. The api allows sorting by date so including a orderby=submitted&direction=asc or sorting by any other sorting option should just sort the results. The direction=asc part is important because once you reach an offset of 9800 you can change the direction to direction=desc which would sort the list backwards. You just need to keep track of the last run that you get by sorting the list normally so that once you reach that run while sorting backwards you will know to stop the count. Using this method should simulate an offset limit of 20k instead of the usual 10k. If my guide was a little too complicated I can explain it in more depth if you ask. Also I put this solution in the related github issue too: https://github.com/speedruncomorg/api/issues/125
This thread is 2 years old but I'm bumping it because this is still an issue, you can't get runs from the API with an offset of more than 10000.
If you're trying to get all runs on the website, as I am right now for something I'm working on, going by ascending order you can only get up to November 2014, if you go descending you can only get runs from the last 4 days. This makes getting runs like this pretty much useless, so you have to specifically search for every individual game (or category) to get all runs.
However you can't even do that since some categories have more than 10000 runs, in fact the biggest Subway Surfers category has 25000. As is mentioned above, you can get 20000 runs by sorting by both ascending and descending, but that still isn't enough.
This is a serious limitation that doesn't seem to have any reason to be here. Even if you're not trying to get every single run on the boards, there are many pretty reasonable things that can't be done because of this. With this issue being 2 years old I'm a little sceptical this will be resolved, but seriously why hasn't this been fixed?
Also, I just found out that the users request is completely broken. It always returns a bad request unless you search for a specific name/id. This also appears to have been broken for a few years.