Leaderboards
7 years ago
United States

It has now been several months since the PC AI community has had access to a load remover for LiveSplit. Based on tests (see my prior posts and spreadsheets), it appears to be working accurately and as intended. This is a great addition to the community because it levels the playing field among PC runners who have dramatically different hardware (and thus, significant differences in load times).

Until this point, the leaderboards have been sorted by Real Time. This was primarily motivated by the need to validate the accuracy of the load remover, and additionally to provide runners with ample opportunity to submit new personal bests with load times removed. Those objectives have largely been achieved, although not every runner has submitted a new time and several of them are no longer running the game.

It is now time to discuss how best to integrate and merge (if at all) existing runs into a leaderboard sorted by Game Time, rather than Real Time. As of now, we are in the minority of PC games that ostensibly continues to prioritize Real Time where (1) a functional load remover is available, and (2) load times vary dramatically based on hardware.

Part of the rationale behind this persistence has been the (completely valid) concern that the leaderboard will become non-sensical and messy if sorted by Game Time where certain runs only have Real Time values. Because of the way the website works, every run that does not have a Game Time is prioritized lower than those that do. In other words, a 2:48 Real Time run will appear to be ranked worse than a 2:59 Game Time run, which is obviously not correct.

Following are several options. For purposes of this discussion, "existing runs" refers ONLY to those runs that have not been (a) timed with the load remover, or (b) manually re-timed to remove loads.

  1. We do nothing and continue to prioritize Real Time (by virtue of sorting by time with loads as the default), despite the availability of a functional load remover and our knowledge that load times vary dramatically based on hardware. This perpetuates "pay to win" - runners with better hardware will have lower load times and better overall times - and discourages runners who have anything less than the best hardware.

  2. We time out loads for all existing runs. This is a tremendous amount of work but something I would be willing to do if runners would provide (a) unobstructed YouTube VODs (or equivalent) of their runs and (b) time(s) of any death(s) that occurred during those runs. "Unobstructed" is critically important - if the load icon is not visible (i.e. because of a face cam), it is nearly impossible to accurately determine load times.

  3. We use the same values for Game Time and Real Time for existing runs timed (a) without the load remover, or (b) where the load icon was obstructed. (An example of this are my runs in Nightmare Any% and Nightmare Low%.) Essentially, we can prevent the website from prioritizing Real Time runs lower than (unequivocally slower) Game Time runs by simply giving those existing runs a Game Time. This is not technically accurate (as the Game Time would include loads), but it would avoid a non-sensical leaderboard. Additionally, it would not affect the current Top 5 runs, all of which have been either timed with the load remover, or manually had their loads removed. (As a FYI, this is the option most games have implemented when a load remover was introduced - see, e.g., http://www.speedrun.com/me )

  4. (Variation of #3) We remove some fixed amount of time as "load time" from existing runs. Based on runs completed with the load remover, and runs manually re-timed to remove loads, it is clear that PC runners have at least 4 minutes of load times, not counting reloads due to deaths. Under this approach, a 2:48 Real Time run would get a 2:44 Game Time. Again, this is not technically accurate, as load times vary greatly by hardware, and this will necessarily be more beneficial to runners with closer to 4 minute load times than those with 6 or more minute load times.

  5. We split the leaderboards into Real Time and Game Time. This will be duplicative and messy (in addition to requiring more work to maintain). Furthermore, the Real Time leaderboard would likely become obsolete.

Option 3 (or 4) is my strong preference. However, this is not a dictatorship that simply imposes draconian decisions. Comments welcomed.

Edited by the author 7 years ago
Bouvet Island

I would keep it as it is, sorted by "time with loads" (RTA), or change the sorting to "time without loads". Preferably the latter.

Option 2 is too much work, options 3 and 4 are not really accurate, and option 5 is messy.

United States

To clarify: anyone on the website can manually change how the leaderboard is sorted (by clicking the dropdown), the issue is what the default method of sorting should be.

http://i.imgur.com/36MMzQ3.jpg

If we continue to prioritize Real Time, it defeats the purpose of the load remover and will eventually lead to disputes over what the fastest run is.

United States

Option 3. It allows the sorting to be switched easily and stays accurate based on the times each run was submitted with.

Iowa, USA

I'm highly for option 3, and agree with twitchy's idea to add a mod comment that notes that those times were submitted before the load remover / the runner chose not to use it and as such they have been edited to provide a cleaner leaderboard.

A few weeks ago when Luna beat Cliffs' run with a better IGT but worse RTA the leaderboards were briefly changed to prioritize IGT because of it (see: http://puu.sh/nMeKR/27cd6303ec.png) which resulted in a completely nonsensical leaderboard. For me, it goes without saying that prioritizing by IGT when not all runs have an IGT is just a mess.

A lot more people now have IGT times on the leaderboard than they do a few weeks ago, and very few runners who are actively running / in the community still only have an RTA timing. I would say that if Info is willing to time out loads for anyone who asks then those active runners should be able to ask, especially Blueberrian as he runs on PS4 and Revas has had his loads timed out for fairness. I'd also like to do option 3 and prioritize by IGT to get Revas higher on the leaderboards. He's put a lot of effort into playing this game on an inferior platform and I think that deserves to be shown by his standing.

Option 4 seems arbitrary and if we were going to time out a "base loads" time for everyone then we may as well just do a quick-and-easy method of timing out loads by pulling up a timer and watching the run straight through and pausing it whenever loading is occuring, to the best of someone's ability. That would sadly be MORE accurate than just removing 4 minutes of loads for everyone, so I'm against option 4. And I agree that option 5 just seems messy.

Okay, sorry for the rambling. Hopefully we can reach a good compromise here!

lunalovegood likes this
United States

Regarding Option 3 (now noted in my original post), this is exactly what other games have done when a load remover was added later. See for example Mirror's Edge ( http://www.speedrun.com/me ), Portal 2 ( http://www.speedrun.com/Portal_2 ), Skyrim ( http://www.speedrun.com/skyrim ), and so forth.

Antarctica

What Curbstompers said.

I do think we should start sorting by IGT (I personally couldn't realistically run before, my hardware is just too old to be anything approaching "competitive" because of how long level/in-level load times are), but as far as older runs are concerned, if other games use option 3, then that seems like a consensus we should adopt as well.

Slightly related, I just realised I should get back into Mirror's Edge...