Timing Saturn/Maria runs
3 years ago
Lower Austria, Austria

Recently it was discovered that there's a significant difference in loading times between different Saturn consoles. The difference is so severe in fact that it would change the leaderboard ranks of some people. As this is obviously an issue for keeping the game fair and accessible for everyone I want to start a debate on what to do with this situation. There's basically two ways this could be settled: 1) Rank based on real-time and let runners figure out how to improve their load times (buying a new Saturn, burning a better disc, buying a real copy, etc.) or 2) Rank based on time without loads.

Approach #2 would require a bit more effort on the moderator's part as they'd have to go through the video and accurately cut out every load. This isn't a big deal for Any% given the small amount of runs and the length of the run but this may be a bit tedious to do for All Bosses. If we decide to take this approach we would still list RTA as well (speedrun.com supports listing multiple times for one run, look at the DOOM Eternal leaderboards for an example).

I decided to make a comparison video between the top runners of Any% to showcase the difference. (Please disregard mitjitsu's run, I forgot to remove a load)

Pros/Cons of removing load times from the ranked times: +) Runners can keep using their existing setups and don't have to try their luck on finding a Saturn with a pristine disc drive. This would also make the run more accessible for newcomers. +) It would make it trivial to allow ODEs (Optical Drive Emulators). An ODE is a device that loads disc images from magnetic or solid state media and replaces the optical drive on the host device. As there are no seek times and error correction as with optical media these devices tend to be slightly faster than a real disc drive. There's currently three such devices for the Saturn: Rhea/Phoebe (https://gdemu.wordpress.com/details/phoebe-details/) Satiator (https://www.patreon.com/prof_abrasive) MODE (https://shop.terraonion.com/en/home/17-Terraonion_MODE_Dreamcast_Saturn_ODE.html) (Note that Rhea/Phoebe are produced in extremely small batches, the Satiator is still in development and MODE is still in the pre-order phase but should be continously in stock afterwards) Whether or not we decide to allow ODEs can be part of a different discussion. +) Currently there's only one runner (mouse) who is using a legit copy of the game. Pressed discs tend to be easier to read by lasers than burnt discs, giving a slight edge to runners who use legit copies. Generally this is a good thing, however a genuine copy of SotN for the Saturn goes for $130-$200 these days and I personally find it unreasonable to expect runners to pay such a high price to stay competitive. A japanese or american Saturn will usually run you $30-$50 these days and an Action Replay/Pseudo Saturn cartridge goes for roughly the same price, making this setup significantly cheaper. +) Saturn discs have already begun to develop disc rot which may make it even harder to come by legit copies in the coming years.

-) Having to retime every Maria Any% and All Bosses run would require more effort on the moderator's or runner's part. It's also still up for debate how exactly we want to time this (e.g. which frames to cut from). -) There's more room for error when timing runs. -) It may be considered less "legit" to time runs like this as it evens out differences between original copies, burnt copies (and ODEs if we allow them) -) There may be lag differences as well between different consoles. The Saturn had 15 different motherboard revisions and there may be slight differences in processing speed. Only accounting for loading but not for these potential lag differences may be seen as a half-measure for some.

Q: What about clock rush? A: Doesn't matter on the Saturn Version as it's controlled by the system time. The system time can be set to whatever the runner desires.

ArcaniaCQ, Rasha and 3 others like this
Texas, USA

I vote in favor of timing runs without loading. It would make the saturn console maria run more accessible to newcomers since they won't have to worry about having the fastest console to compete with top times, which will become more of an issue as time goes on and working saturns become more scarce. This also removes any issues of consistency regarding retail vs burned CDs which again will be a greater issue in the future as is the case with all optical media. I will gladly volunteer for retiming current leaderboard and future runs.

Edited by the author 3 years ago
Maryland, USA

I vote for timing without loads. It is becoming more clear with time that there are many factors with a saturn console that can affect loading times. Not just loading times, but lag is also a factor. I think trying to factor in different mother board revisions and disc drives is already more of a headache than taking the effort to time runs without loads. Plus, I think now is the time to do it since Maria runs have been more active lately in both Any% and All Bosses. Any measures that can be taken to appeal to new potential runners should be taken, and I think this is one of them. This way, no one has to worry about their console setup. They can just play the game on what they have.

United States

My instinct is to agree to time without loads, but I just don't feel like it's totally justified yet. Yes, it makes sense and seems largely beneficial, but why do it now only for Saturn Maria when the other consoles have the same issue? Basically if we decided before not to equalize PS & 360, or PAL/NTSC, I don't know why the decision here would be different. I'd love to vote for timing without loads, but I guess I need a little more convincing.

miba likes this
Lower Austria, Austria

I think this is a different situation than trying to equalize PS and 360 or PAL and NTSC. The PS and 360 versions have significant differences in input latency, lag and loading. It's hard to compare runs between them for this reason. When we're talking about Maria on Saturn we're talking about the exact same run under the exact same constraints with the only difference being the load times. The PS/360 versions will also always have the problem of clockrush. As has been pointed out by Talic before, the ingame time keeps advancing during loads which would more or less defeat the point of timing runs without them since you might have to wait on the clock gate to open anyways. This isn't an issue on the Saturn due to the clock gate opening based on system time.

Another thing to consider is that the 360 version has already become the de facto standard for Alucard and Richter runs with very few people playing on a legit PS1/PS2, so it's not as big of an issue in the first place.

If we look at the Alucard Any% NSC leaderboard right now the top 23 runs were all played on X360. The X360 version is generally assumed to have consistent performance so it's easy to compare those 23 runs with each other and have an accurate leaderboard, you can't do the same for Saturn/Maria runs right now. Imagine if the X360 had huge differences in loading and lag which would for example put the current WR on 5th place instead.

X360s are also still widely available and you don't have to track down an expensive copy of the game.

Bulgaria

Just a side point to clarify something, 360 does not have consistent performance. European 360s run the game at 60 fps, due to having 60hz output, while NA ones do 59.94. Additionally some 360 seem to load drastically faster than others and some even have lag.

Lower Austria, Austria

Interesting, I was not aware of that. Do you have any sources or more info on the refresh rate difference between the different 360s? Also has anyone done proper lag/loading comparisons between them?

Bulgaria

That pretty much sums it up. I found it by frame counting videos to see why the clock statue opened faster for some. Asked a few people to send me 10 minute videos sitting in the clock room and stuff like that. We still don't know why it opens as fast for dr4gonblitz as it does for the European console, since his is confirmed to output at 59.94, yet seems to run as if it is 60. It doesn't help that they are all just designated "Xbox 360 Slim".

miba likes this
Scotland

My sentiments are very much in line with what Rasha said. Whether we need to do this right now I don't know. However, as time goes on working Saturns will become more scarce and replacing drives won't be easy. Not to mention the game goes for around ~$150 and will only continue to get more expensive. Will people who own the game want to make frequent use of what might be a $300+ game 10 years from now? I know from owning games like Panzer Dragoon Saga, Magic Knight Rayearth and Shinrei Taromaru I most definitely won't. In the future I'd most likely gravitate towards using a RHEA. Satiator or MODE. Devices that would have clear loading advantages over optical discs.

My worries are that this will open he flood gates for people doing other categories or even other games. By using a decision made here as a precedent. Also, what if this version of Maria gets a future E-shop release? Are we going to change the timing rules, because there is little to no loads on modern console or PC? After all, players generally favour using the fastest version of the game. Which is the reason no one does Richter or Alucard on the Saturn. Finally there is the issue of mods having to manually re-time runs, which will be lengthy and error prone as shown in the video posted by the OP. Especially in a All Bosses run. Also how exactly loads are being timed may also be unclear.

I'm going to vote yes. Not because it's something we need to do right now, but because it's something that will inevitably need to be done as time goes on.

Edited by the author 3 years ago
miba likes this
Lower Austria, Austria

The issue of potential re-releases is an interesting one, but I find it unlikely that the Saturn version (or Saturn Maria for that matter) will ever see a re-release considering that the PS4 version and the recently released iOS/Android version seem to be based off of the PSP PAL release for whatever reason.

As for setting a precedent for other categories: hard to say what will happen, but as Rasha pointed out similar discussions have been had in the past.

As for other games: that's up to those communities to decide. Several games already do this, on top of my head I know that several FDS games are timed without loads, such as Super Mario Bros. Lost Levels and the FDS Version of Zelda 1. It seems like the decision there was made for similar reasons: getting an FDS working again is a major chore as the rubber drive belts have all disintegrated at this point and some other mechanical parts have broken, and if you have a working one there seem to be variations in load times as well. The magnetic diskettes are also quickly approaching their expiration date. There's also lots of people using flashcarts (Everdrive N8, Powerpak) and emulators which all have different load times.

Amber24 and 3snow_p7im like this
Bulgaria

Proposed timing method for loads:

  • Every area load, internal load and boss load gets removed from the first duplicated frame up to and including the black frame before the first faded in frame.
  • Second castle warp load gets removed from the frame when white teleportation animation around maria disappears up to and including the last frame where that same animation is there in 2nd castle.

For now we will put times without loads inn the time column and put RTA in the comments. It is definitely not the desired way to do it, but there has been some conflicting reports about how turning on the integrated loadless time or IGT columns may affect an already populated leaderboard. Until I know for sure that it won't wrongly affect nearly 1000 runs I won't use that.

Edited by the author 3 years ago
Texas, USA

The moderating team has decided to go forward with the suggested change to the timing of the console Maria categories for 2 reasons:

  1. Unanimous recognition among the stakeholders involved (ie. the actual runners of the categories) that the suggested change benefits the community.

  2. Lack of a compelling argument against the change that cannot be resolved in the future with further discussion and/or clarification.

To recap how runs will be timed going forward:

• Every area load, internal load and boss load gets removed from the first duplicated frame up to and including the black frame before the first fading in frame.

• Second castle warp load gets removed from the frame when white teleportation animation around maria disappears up to and including the last frame where that same animation is there in 2nd castle.

Times for existing runs will be updated to reflect the new timing method and their run descriptions will be updated to include their original RTA times. All the Any% runs will be updated immediately, followed shortly by the All Bosses runs. Unfortunately, a few Any% videos are missing frames during transitions into loads and are unable to be timed using the new method. These runs will retain their RTA times until the runner can provide better quality video of their run.

It is important to note that the new timing method is implemented in such a way that preserves all strategies optimized for console. All times will continue to be based on the timestamps of the original video provided by the runner, with the only difference being that now the sum of all loads will be subtracted. If you’re interested in exactly how this is calculated, please use this spreadsheet as reference:

https://docs.google.com/spreadsheets/d/1OWifCUV__lzbI0-k_I8oDupQYAqzP61tPbMrYf7B32E/edit?usp=sharing

Please keep in mind that these changes are isolated to the console Maria categories and are not intended to influence other platforms or characters. While it is true that no platform is immune to some performance disparity among runners, the degree to which the various revisions of the Saturn hardware are affected seem unique in the SotN speedrunning community.

We thank everyone for their feedback and perspective on this matter, but especially Miba for his initiative in measuring the difference in loading times among the top runs and elevating the discussion around this issue. However obscure it may be, getting a competitive time on a console Maria leaderboard is now certainly more accessible thanks to him.

MASH, Rasha and 6 others like this
Scotland

Just to clarify how the teleportation loads are timed since I'm a little unclear on the wording. Is the first and last frames used in these two pics below?

https://cdn.discordapp.com/attachments/864375398364217347/1076212562397515826/MariaT1.PNG

https://cdn.discordapp.com/attachments/864375398364217347/1076212579510255676/MariaT2.PNG

Edited by the author 1 year ago
Bulgaria

The first frame is wrong, you can double check by looking at the times on the spreadsheet with avidemux.

New Hampshire, USA

Sorry to bring up an old topic, but what are the stance on the Saturn ODE. My laser died on the Saturn so I ended up getting a Fenrir ODE. After recently finishing the game, I am super interested in learning Maria%. Getting another Saturn is out of the question now due to the rising prices so the ODE is the only way I can play...

Scotland

I use a Satiator, the timing rules were changed to account for loads, because of people increasingly moving towards using ODE's.

Tishero likes this
Bulgaria

Since it turns out that there is significant performance differences between different versions of the Mednafen Saturn core it seems reasonable to adopt this methodology for emulator runs, while encouraging runners to use the latest version available.

I made an example of timing a run:

Edited by the author 11 months ago
Sestren and Tishero like this
Illinois, USA

With regard to timing Maria Console runs, I'd like to suggest the following change to timing Maria Console, All Bosses be considered:

* For leaderboard placement purposes, milliseconds for Maria Console, All Bosses should be dropped

Since timing a Maria All Bosses run for Console requires sampling over 150 timestamps, even being off by one frame on each sample can cause the final time to fluctuate by as much as 2 seconds or more. Furthermore, it seems video players are not all that great at giving consistent timestamps, even when going frame-by-frame.

I would consider two MAB Console runs that were within a second or two of each other even after controlling for load times to be within the margin of error for verification purposes. Displaying milliseconds would give a false sense of accuracy that is not possible given the timing methods used.

Bulgaria

Well I think milliseconds need to stay in if we remove loads, but thousands could be dropped since capture imperfections make that measurement meaningless. Actually on the Timing Conventions guide it already says "Precise times use milliseconds rounded to the hundredths place." for all categories.

Edited by the author 3 months ago
Sestren likes this
Bulgaria

So what's going on with Maria on Emulator? I took down 3 PBs to make sure everything is fair, but now everybody is using ancient versions of Bizhawk to get an advantage. Either we start doing loadless timing or it's a free for all.