Proposing that emulator guidelines be made into official site policy - starting with NES
6 years ago
California, USA

For most leaderboards, there is still too much leniency on emulator runs despite how easy we know it is to stream & submit a TASed run. When a console run gets cheated/spliced the way it gets caught is through pretty objective measures like audio disparities. But with emulated runs you are forced to deal in really subjective junk like whether certain movements "feel" fake, or what your opinion of the runner him/herself is. I just feel like this is a tremendous waste of mods' time having to put that much time and effort into judging something as real or fake when they really have no genuine evidence either way.

The most popular leaderboards on this site already have extensive regulations on what is required of anyone who submits an emulator run, and these guidelines have roughly the same major points. They aren't identical though. There are a lot of standards that exist in some rule sets but not in others and they don't have consistent lists of allowed emulators. But there is really no reason whatsoever why these guidelines should vary so wildly between games on the same platform. If you have a standard set of rules for say, NES emulation, those basic rules to prevent dubious, unverifiable runs will apply to nearly every single game on the system.

Now we could have this gigantic list of emulator regulations and copy-paste it into every single category of every single NES game. But that takes up a ton of space and would need to be updated constantly with the release of new emulators.

So instead how about this:

  • We make standardized lists of reasonable expectations of emulated runs + allowed emulators for each console, and publish it either as a dedicated page on speedrun.com or as a sticky.
  • For each category on each game on that system we just include a hyperlink to the page/sticky. That way we have a standard set of rules that won't have to be manually updated by individual game mods.

I'd say that a lot of the mods on this site (including me) aren't completely educated on which emulators can be trusted. But if you had an official template signed off on by people knowledgeable on the subject they wouldn't need to worry about it.

Now of course, with the way the site is currently set up and run by different groups of mods we run into the issue of territoriality & cliques who will outright reject certain changes so I don't know if I'll ever actually be successful in making this an official thing. But at the very least it could be a semi-official posting that we could recommend to current mods or anyone adding a new game on that platform.

As an example, I looked at the most popular NES leaderboards and compiled some of the guidelines they have for emulator/flash cart usage.

This is pretty exhaustive and I doubt most leaderboards would be willing to adopt it in its current length but even a hugely abridged version of it would make a huge difference.

(The first two are not restricted to emulator runs but still good general guidelines) Please ensure game audio is audible. Turbo/Autofire and left+right/up+down are banned. Multiple inputs remapping (like down+a on a single key) is not allowed.

Due to the relative ease of submitting a TASed run, emulated runs are held under higher scrutiny. To minimize the chances of your run being rejected, we recommend the following: -State the name & version of the emulator used in your submission comments. -Capture the entire emulator window, including the border. -Enable frame counter and input display. -If the emulator has a movie option you should show the "no movie" message. -Your run should begin from (and include) system reset or power cycle. -It is highly recommended that you stream all of your attempts and provide split files.

If you used a flash cart (like Powerpak or Everdrive) or some other way to play the game not on original hardware, please add that in the comments as well.

Recommended emulators: BizHawk (NESHawk core), FCEUX, Nestopia (Alternatively I found this list compiled on the Castlevania forum: https://www.speedrun.com/cv1/thread/gxkie)

Other possible lines -State whether you used a keyboard or gamepad. -Enable handcam if possible. -At the end of your run, show your key bindings (to prevent multiple input mapping).

Also, with emulators and flash carts there does exist the possibility that someone could do runs on a bad ROM dump or even on a surreptitiously hacked ROM. This could be prevented by requiring a specific ROM checksum.

Edited by the author 6 years ago
NoControl, paintophobia and 6 others like this
Canada

I'd be fully in favour of something like this as long as the platform's community can reasonably agree on what those guidelines should be (which appears to be the case for NES based on your post, but may not be the case for newer platforms with lots of multi-platform games), and that they're voluntary guidelines. If they were forced into the rules for every game site-wide then I think a lot of people would be annoyed with them and reject them, even if they otherwise would actually agree with and enforce the recommendations.

Imaproshaman and Quivico like this
Valhalla

Note: that list of accepted emulators for cv was based off of testing the first 10 seconds of the game, holding only right. The top three emulators I found to match console in frame count, the others were either faster or slower by 1 frame. I was using a Dazzle for console capture and honestly don't trust it too much, but it was only the first 10 seconds of the game, and the emulators known for accuracy held up against my NES.

Also, I am all for having site-wide rules. I've accepted a number of emulator runs that, despite not being sketchy, I was still annoyed that they would often crop their video to skip the title screen. That and the lack of seeing a system reset in a lot of PB vids means they could very well be loading a save state, which I imagine isn't just a big deal in castlevania alone.

The last thing I want to do is out right reject emulator runs, and I hope it doesn't come to that haha. Also if site-wide guidelines were put in place, I would hope that it doesn't affect emulator runs that are already posted.

Edited by the author 6 years ago
Imaproshaman likes this
Esperanto

"-Capture the entire emulator window, including the border."

Is there a way to do this in OBS using window capture; or is the only way basically to capture the full screen?

New York, USA

"cliques who will outright reject certain changes" - On this site? What the...

Imaproshaman likes this
North Carolina, USA

Do whatever you want people are still going to cheat runs with and without emulators. Don't force site wide rules that tons of people don't want to or aren't going to follow. Just educate them and let them decide for themselves.

Imaproshaman likes this
Scotland

My only problem is half the stuff you said I either don't understand what it means or don't understand how to do it. Personally if I see a run with a frame counter I am a lot more suspicious and other such things because they are used in tas a lot from what I see on the tas website. I think runs showing a reset is good though and I understand how to do that I also liked some other points you made. I think cheaters will always find a way sadly ruining it for everyone else.

And as some other people have said here I personally don't think these rules should be mandatory but nice to have especially for people who have no clue what to do maybe with a few tweaks. :) Normally I don't show my resets in emulator runs because I think the text looks ugly but mentioning it here makes me think maybe I will if it will help mods feel more at ease thank you. I don't mind doing simple things but its more of the complex stuff I don't understand I know it probably is simple to most people but me and emulators co exist but I don't know there ins and outs if you get me :)

Edited by the author 6 years ago
Germany

[quote]-Capture the entire emulator window, including the border.[/quote] Not possible in OBS Studio unless you use Monitor capture which is 1) extra work to set up every time the emulator changes position for some reason, in the case of bizhawk it changes size all the time depending on what console you emulate. 2) doesn't work properly on a lot of graphics cards. [quote]-If the emulator has a movie option you should show the "no movie" message.[/quote] On some emulators this causes the game to freeze, while the menu is open and therefore would invalidate that VoD by the standards of the next one: [quote]-Your run should begin from (and include) system reset or power cycle. [/quote] which also has bad wording, because forcing emulator to run with different timing gives them a disadvantage instead of providing extra fairness. Should be "Your submission video should begin [...]". [quote]-It is highly recommended that you stream all of your attempts [...][/quote] Not everyone can stream, for example my family is forced to use a trash tier ISP provided router that crashes if I stream or upload while any other device is already doing something, therefore I can't stream.

I'm also not a huge fan of cluttering up the gameplay with input displays and framecounters (which probably add a tiny bit of extra lag too) for games with a small resolution, but that's subjective. There are better ways of doing that, like Nohboard, which you can put next to the gameplay, for example under livesplit. Ideally the footage is recorded at a framerate and quality where frame-by-frame comparisons are easy to make. As mentioned above banning Turbo/Autofire is also subjective, some communities have opted to make the use of this allowed with the compromise of it being a seperate category.

In addition using a whitelist of emulators can be a bad idea, because emulators still get developed, therefore could become more or less accurate over time. For example Gameboy and Gameboy Color switch from Gambatte to BGB and back to a fork of Gambatte called Gambatte-Speedrun over the past 2 or 3 years. Everytime something like that happens, the whitelist would need to be updated by site staff, who are already busy enough as is.

EDIT: Oh yeah, while we're already mentioning Bizhawk, which really is mostly just a collection for multiple emulators, like mGBA for GBA (it also offers VBA-Next, so that's an extra thing to show before every run), these collections would be banned due to whitelists, unless they are the only ones offering a core made for that specific program.

Edited by the author 6 years ago
Alayan, paintophobia and 2 others like this
England

It's a good idea in principle but I also believe this will result in a whole lot of extra work for moderators, whilst many runners are just not going to bother to remain clued up on the rules and standards, resulting in a much higher proportion of runs being rejected for frivolous reasons. This would need careful consideration and application to work effectively.

It's also not really clear about which rules are really necessary. Two immediately jump out at me:

"-Capture the entire emulator window, including the border."

"-Your run should begin from (and include) system reset or power cycle."

I can't speak for other games/series, but in Classic Sonic we used to go by the first listed rule as it was a generally effective method of ensuring that a movie file wasn't being played. Nowadays it's been superseded by the second rule which is easier for runners to adhere to whilst achieving the same result, so it makes little sense to include both rules. If anything it only adds an extra rejection criteria, for no good reason.

paintophobia and Quivico like this
California, USA

OK I may have originally written this in a moment of intense paranoia & frustration. I know that emulation is a massive boon to accessibility and I don't want to block it off, and it also wasn't my intention to be looking down at & punishing people just for being emu runners. But I want to call attention to this to prevent current & future emulator runners from making mistakes that will prevent their genuine efforts from going to waste. Anyone who chooses to speedrun on emulator should be aware of two things:

  1. Which emulators are the most accurate, and why using an inaccurate one can provide an unfair advantage.
  2. Making & playing back a TASed run is easy and, if done right, completely indistinguishable from a genuine run. If these facts become common knowledge & you take reasonable precautions in advance then the tens/hundreds of hours you spent doing a really amazing emulator run won't get trashed when someone correctly points out that there is no definitive way to prove it actually happened. Regarding the 2nd point, people have said that people who want to cheat will cheat anyway, by splicing video for instance. Maybe, but this is ignoring how monumentally trivial producing a tool-assisted run is. Multiple instances of splicing have been exposed by things like audio disparity, missing or additional frames, and impossible RNG patterns. A covert TAS completely bypasses every single one of these points of detection. I should cede on the following right away though:
  • The issue of turbo does indeed depend on the game and is usually pretty easy to identify on video. It really doesn't have anything to do with my concern about secretly TASed runs. So scratch that.
  • Everything I've said should really only apply to reasonably competitive runs. Top 3 at the least. I don't think there is any point in putting this level of scrutiny on every single run that comes in. Most people submitting runs to this site are completely honest and on the level & there is obviously little to no motivation or point in cheating an unremarkable run.
Valhalla

I would love some standards for what emulators are used, for sure. Not so much a site-wide rule maybe, but just some guidelines.

I notice in SNES and N64 games people are pointed towards pretty much only one or two emulators. I get that probably has more to do with emulating those consoles compared to NES which seems to be more figured out, but I hate seeing an emu run pop up and I have to guess which emulator it is. It's made even worse when that is combined with dropped frames from twitch vods.

Germany

I support this idea. Like others have mentioned, I would also prefer this to be implemented as a centralized "official" suggestion, that the particular game pages can refer to. Not every game moderator is knowledgeable about emulators or even aware of emulation inaccuracies, so having a general guideline would help those out. Furthermore, there might be special cases in which a community can formulate exceptions for their game (e.g. categories with memory manipulation in Game Boy Pokémon games requiring a particular emulator). I assume there are knowledgeable people for pretty much every platform out there who can help putting this database together, initially. I have a lot of experience with Game Boy speedruns and the various emulators used for it. I manage 2 moderately active leaderboards with Super Mario Land and Super Mario Land 2 with between 1 and 10 run submissions per week. It still happens that people do not read the rules and submit a run on an inaccurate emulator; and they may have also ignored this information if it had been present in a speedrun.com knowledgebase, but putting it somewhere at least increases the chances of it being seen. If it spares even just one person from the disappointment and discouragement of getting their run rejected because they just didn't know better, then it is already worth the effort. And at the same time it educates those who are actively seeking out that information. It also can bring together people from different communities to exchange ideas and approaches and help each other out. More things should be standardized for particular games (e.g., framerate standards for Game Boy games). The Megaman leaderboards have had an automatic time conversion for years, depending on the platform the game was played on. Imo, no harm will come from having emulator standards. And just like the majority of this site is handled, sr.com should provide the canvas and let the specific communities provide the content - probably with some mediation involved.

Edited by the author 6 years ago