Star Fox Adventures Memory Leak Discovery
3 years ago
New York, USA

For anyone that's on the Star Fox Discord, this is the same method/video that I've posted there about a month ago. I've decided to repost this here so that it doesn't get lost to time over there and to expose this to anyone that's not on the Discord server.

For anyone who's been following this channel for a bit, I've mentioned that this game leaks memory but we couldn't figure out how or why the game does this. Just last week I've pinpointed what causes the game to leak memory. If you were to somehow go OoBs and unload the map, the game leaks memory each time you unload the map via OoBs. My guess is that the game doesn't know how to properly clean up objects when unloading maps this way and the game orphans these objects in memory. So far I've managed to get some textures and particles to unload from messing around with this but I can't single handily figure out every single possible skip you can do with this. For anyone that's interested in glitch hunting this game, this is a huge discovery that should be messed around with.

Since the last post I've made on the Discord server I've found several things that I didn't know about a few months prior. Some things that I've found with this method of leaking memory since then are: -This method of memory leaking works on all versions of the game. -This method works on both the GC and Wii with little to no differences on how many times you can unload/load a map before crashing the console. -Can confirm that you can unload certain triggers in Thorntail Hollow such as the Egg Minigame trigger not loading but I haven't found a way to do this consistently. -This method works in Thorntail Hollow and Lightfoot Village -I wasn't able to get the game to crash when doing this at Snowhorn Waste. I know that I wasn't memory leaking in that area since I've unloaded and loaded the area about 100 times with nothing getting affected by this method there. However, this doesn't mean that it's not possible to memory leak at Snowhorn Waste since I was only able to test this method out in only two areas of the Waste.

Things that still need to be looked at: -Find out what is being orphaned. This can help eliminate some places to test this out if it's a specific prop/props that are being orphaned when doing this. -Find out if we can overwrite key memory addresses of props and/or cutscenes with junk data with this method.

We've narrowed down what causes the memory leak at Thorntail Hollow to be the WarpStone which makes this memory leak only exclusive in some places in Thorntail Hollow where the WarpStone is loaded in. However, I still think there's some other objects that are causing a memory leak in Thorntail Hollow since doing this method at different locations and at different points of the game require a different amount of zoom-ins before crashing. Also we can confirm that objects from other maps can get unloaded from this memory leak at Thorntail Hollow as seen in this video below.

I must mention that in order to unload objects in other maps you'd have to save at the map which you want to unload certain objects then go to Thorntail Hollow and perform the memory leak.

I should have posted these when I first made this post, but the codes below are to enable the debug menu for this game if anyone else is interested on trying to figure out what causes this memory leak. You'd also need to have enabled the debug text cheat which. If you don't know how to do this, read through this section here https://tcrf.net/Star_Fox_Adventures#Development_Stuff

Gecko Code (Only works for 1.0) c6148b78 80137948 c62510cc 80137948 c6246e04 80137948 c61378a8 80137948 c213798c 0000000d 91410024 7d6802a6 91610068 3c60803e 8063bc14 80810008 80a1000c 80c10010 80e10014 81010018 8121001c 81410020 81610024 3c008028 6000f688 7c0803a6 4e800021 3c80803e 80a4bc14 7ca51a14 38a50001 90a4bc14 80a10068 7ca803a6 60000000 00000000

Adding c6148bc8 80137948 will give additional Tricky debug text (I doubt he's causing the memory leak).

Adding c607d6dc 80137948 will display all OSReport messages.

Adding 003dcded 00000001 will give you map cords.

Action Replay (Only works for 1.0) 04137948 4BF45D94 04148B78 4BF34B64 04148BC8 4BF34B14 042510CC 4BE2C610 04246E04 4BE368D8 0428F7DC 4BDEDF00

Also it seems that the Warpstone doesn't have as much of a role in this memory leak than we first thought he had. The game uses about 75%-85% of memory in the areas where the memory leak occurs. I have no idea what is using this much memory in these locations but I'd say that this is the likely culprit to the memory leak than the Warpstone being the sole reason at this point.

Edit: Seems like the magic gem and the particle effects don't get freed up when Thorntail Hollow gets unloaded. This might be one of the major factors to the memory leak but I doubt it's the only thing since it still doesn't make sense that the amount of zoom-ins require before the game crashes changes based on your location and your game progress.

Image of the Warpstone area before it's unloaded: https://cdn.discordapp.com/attachments/141969068956712960/804188310041133086/image.png

Image of the Warpstone area when it's unloaded (Magic gem and it's effects are still loaded in behind Tricky's food counter) https://cdn.discordapp.com/attachments/141969068956712960/804188311286710282/image2.png

Seems like there's a pattern with the memory leak, if there's an adjacent map that we can load from OoBs a memory leak will occur. There's another place where we can do this memory leak besides Thorntail Hollow and it's at the corner of Lightfoot Village closest to Cape Claw. Here's a picture that shows the positioning of all of the maps that this game has to give you an idea of where a memory leak can theoretically occur. Thanks to Rena for finding this out and for making this image: https://cdn.discordapp.com/attachments/141969068956712960/804513208219861052/map.png

Based off this picture, we can also theoretically leak at Cape Claw with Krazoa Palace and at Cloudrunner Fortress (Although there's no way to get OoB in these areas normally) but we haven't been able to leak at these areas yet.

As I've mentioned in my Discord post, this is probably one of, if not, the largest discovery for this game since ESW that can lead to huge changes for both Any% and possibly the 100% routes for this game. However, since this trick involves overwriting memory addresses, it'll probably take some time to find any use for this trick. It doesn't help that I also don't have as much free time to mess around with this glitch. If anyone is interested in glitch hunting for this game I'd still say that this glitch should be looked at.

Edited by the author 7 months ago
zcanann and BvqRzxi5 like this
San Francisco, CA, USA

Edit: Ignore a lot of my initial findings. I used emulator settings that apparently would otherwise crash on an actual console.

For those who are new, this glitch is set up by performing an air swim first. See https://www.speedrun.com/sfa/guide/oe1h4

For basic routing information, check out https://www.speedrun.com/sfa/guide/m3cys

=======================

I'm time constrained too, but I think I'll try and poke at this some when I can.

Things that I think are worth looking into:

  • v1.00 and v1.0.1 differences
  • Whether this can be used to unload parts of Cape Claw to get OOB and skip needing early Frost Blast (~5 minutes saved). Maybe OOB isn't even needed, and we can keep the fire unloaded. (https://www.speedrun.com/sfa/thread/j98z7)
  • Unloading the moon pass door (I tried this several ways with no luck)
  • Obviously K4 early
  • The effects on creating new games
  • Unloading the cage at the top of the Krazoa Palace. This alone would save like 2 hours I think.
  • Everything imaginable

It seems that the current state of ThornTail Hallow, as well as StarFox's position influences the # of zooms required to finish the memory leak.

Early game takes ~10, later it takes ~17.

If you jam yourself into the bottom left corner of the map, it usually takes 1-2 extra zooms to crash. Edit: Even then, this seems RNG based.

The only thing I've been able to keep unloaded are exploding door textures, and the shop. It caused the camera to get weird too.

=======================

To manipulate a particular zone, I've been using this strategy:

  • Get to the zone you want to mess with
  • Save and quit
  • Enter game, and backtrack to ThornTail Hallow
  • Swim OOB and perform this glitch
  • Now when you quit without saving and re-enter, the memory leak will be active in the zone of choice
Edited by the author 2 years ago
BvqRzxi5 and Dark-X-rane like this
San Francisco, CA, USA

Edit: Ignore a lot of my initial findings. I used emulator settings that apparently would otherwise crash on an actual console.

I've been attempting to unload the K4 gate, but no luck.

I've been able to unload:

  • The log bridge before the K4 gate. This seems pretty significant, actually.
  • Icons for items in the C stick menu (usually the power chamber key)
  • Campfires
  • Sharpclaws (rare)
  • Textures for mamoths
  • The lock to the snowhorn gate, but not the gate itself
  • Textures for spider webs in the pass between Thorntail Hallow and Snowhorn areas
  • The Snowhorn Trials mammoth
  • I've had some terrain not load (walking on it causes a void out)

However I've never gotten the ice gate to unload, nor the gate that requires a key. The only time the ice gate was gone was when I had already burned it.

Makes me wonder if unloading doors is possible.

Some attempts here, with 16 unloads on Dolphin

Edited by the author 2 years ago
Dark-X-rane likes this
San Francisco, CA, USA

Strange scenario:

  • Perform a low swim ()
  • Hit the shop loading triggers (one by climbable bricks, one by the wall after the climbable ledge)
  • Clip above the shop
  • Zoom Leak spam. The game will start lagging, especially for zoom #7 and a few after that. But it never crashes, and eventually the lag goes away.
  • I soft-locked after trying to use FireBlast.

Skip to ~0:40 if you do not care about the setup. Best viewed on 2x speed.

In other attempts, the game crashes after about ~7 zooms ONLY if you attempt to open the pause menu or continue the game normally. Zooms will not crash the game by themselves in this scenario.

This seems to indicate there is something even weirder going on than a simple memory leak.

Edited by the author 3 years ago
Dark-X-rane likes this
Japan

One of the interesting unloads, which may not be related to memory leak, is the Volcano Force Point door. twitch.tv/videos/48385732 I don't know how to reproduce this, but I think it does exist.

zcanann likes this
San Francisco, CA, USA

I tested this at various points in the game to see how many zooms it took before a crash on Dolphin. I used the original glitch setup.

Disclaimer: GC/Wii may be different. As always with this trick, there seems to be a margin of error of ± 2 zooms.

WarpStone first available - 9 Rescued Tricky - 11 Obtained SS1 - 19 Obtained SS2 - 18 Obtained K3 - 18 Turned in K3 - 17 Obtained SS3 - 17 SS3 Placed - 9 Obtained K4 - 11 Turned in K4 - 17 Obtained SS4 - 16 SS4 Placed - 16

======================

Untested, since these are not in the any% route: SS1 Placed K2 Obtained/Placed (K1 is irrelevant since that is done with Krystal)

======================

It seems that Thorntail Hallow can end up in two main states, with (roughly) 17± 2 zooms required, and 10 ± 2 zooms required.

Edit: It's also positioning based. I guess it depends on what objects are loaded.

It may have nothing to do with the game progression, and instead only which maps or objects have been recently loaded.

Finally, I could have swore there was a time when I was first trying this trick that it took me like 25-27 zooms. I can't reproduce it, maybe I'm wrong/crazy. If anyone figures that out, please post here!

Edited by the author 3 years ago
Dark-X-rane likes this
San Francisco, CA, USA

Edit: I thought this setup crashed less, but it also seems to be dependent on the state of the game. See youtube comments for details, but I don't think this setup is perfect. :'(

Edited by the author 3 years ago
New York, USA

In the case of preventing the game to crash here, I don't think this is possible to do since on actual hardware, the game doesn't lag when doing a zoom-in, it just crashes after 17-18 zoom-ins. I'd assume that if your game starts lagging on Dolphin that it's a safe bet to assume that's the equivalent to the game crashing on hardware.

zcanann likes this
San Francisco, CA, USA

Just adding more info to the info dump:

  • It is possible to gain control during the CapeClaw intro cutscene. However, you are more likely to crash, glitch out the camera, or fall through the ground. If there is a cutscene we would want to gain control of, it's worth trying this glitch (success rate is likely < 5% though).
  • When the game is under load (textures are weird), sometimes using Tricky's ball plays a sound effect, but does not spawn
  • I suspect this was known, but if you save while the game is super broken, a console reload returns the game to normal.
  • Spamming this glitch in Thorntail Hollow slowly destroys that area. GrubTub Well can become unloaded, the shop triggers can become unloaded, DIM path can unload, Cape Claw path can unload, MM Pass path can unload. The collision for Fox's ship can become weird and much larger. Bombable doors can reseal (but become uninteractable). Unfortunately, there is no way I've found to destroy other areas quite as severely.

The more I look into this glitch, the less I seem to understand it.

Also, I managed to perform a low-swim and unload the WarpStone, and the glitch no longer works

I tried this 3 different times, including in the original spot, and the glitch no longer worked.

I also tried the same low-swim into the shop with the WarpStone unloaded, and again the trick no longer works.

This is more evidence that the WarpStone or something that loads with the WarpStone is responsible for the leak.

I also managed to later find the location of the load triggers for the WarpStone. Unfortunately zooming near the sewers crashes at around ~10 zooms (After SS1 obtained).

Edited by the author 3 years ago
josejavier1158 likes this
New York, USA

I should have posted this earlier since I've just remembered this, there's a Gecko code and an Action Replay code to enable the debug menu for this game. This should help out with figuring out what's causing the memory leak Zac.

Gecko Code (Only works for 1.0) c6148b78 80137948 c62510cc 80137948 c6246e04 80137948 c61378a8 80137948 c213798c 0000000d 91410024 7d6802a6 91610068 3c60803e 8063bc14 80810008 80a1000c 80c10010 80e10014 81010018 8121001c 81410020 81610024 3c008028 6000f688 7c0803a6 4e800021 3c80803e 80a4bc14 7ca51a14 38a50001 90a4bc14 80a10068 7ca803a6 60000000 00000000

Adding c6148bc8 80137948 will give additional Tricky debug text (I doubt he's causing the memory leak).

Adding c607d6dc 80137948 will display all OSReport messages.

Adding 003dcded 00000001 will give you map cords.

Action Replay (Only works for 1.0) 04137948 4BF45D94 04148B78 4BF34B64 04148BC8 4BF34B14 042510CC 4BE2C610 04246E04 4BE368D8 0428F7DC 4BDEDF00

I've messed around with the Debugger a bit and I've noticed one thing that seems to be present when the memory leak occurs. When it come to Thorntail Hollow itself, this area takes up about 65%-85% of memory to begin with. I've noticed that the areas where the memory leak occurs, memory usage is at about 75%-80% in those areas before we do the zoom-ins, whereas areas where the memory leak doesn't occur, memory usage is less than 75%. I'd assume that if an area is using 75% or more memory before we perform a memory leak, it's safe to assume that this would be enough to cause one.

Edited by the author 3 years ago
zcanann likes this
San Francisco, CA, USA

I've been playing around with these a bit, they seem like they will be useful. I noticed in the Dolphin logs there are both invalid reads AND invalid writes. The invalid writes give some hope of arbitrary code execution 🤔

One question, what are you using to track memory usage?

I'm not seeing any memory info with these debug codes. Also Dolphin has some memory debug features but they seem incomplete and not super useful.

Edit: For the nerds here, it looks like somebody is actively reverse-engineering this game with Ghidra: https://github.com/RenaKunisaki/StarFoxAdventures -- Building on this work is probably crucial for understanding this bug fully. I'll reach out to the author and see if they have any free cycles to look at this.

Edit 2: They responded -- reposting some info here from Discord

  • Boneheaded trial and error approaches are unlikely to be useful here. This will require real reverse-engineering.
  • ASLR is not in play, so all behavior should be reproducible based on positioning & previous events. Zelda OOT bugs are quite similar in this regard, so we should be learning as much as we can from their methodology.
  • Disabling Dolphin MMU should give us crashes consistent with real hardware.
  • "The number of objects loaded does not increase each time, and drops to 3 (as expected) when out of bounds, so the problem isn't an object not getting unloaded, but failing to unload some resource of its own"
Edited by the author 3 years ago
New York, USA

I could have sworn that the Gecko codes I gave would have shown memory usage. I was actually looking at a few of Rena's streams of the Kiosk version of the game where he had Memory usage enabled on the debugger. Since what I saw was on the Kiosk version of the game, memory usage could be lower in the final game. However, I highly doubt that the memory usage would be dramatically different in the final game though.

zcanann likes this
San Francisco, CA, USA

Giving an update from Discord, based on Rena Kunisaki's findings:

  • It's definitely caused by crystal plants. For some reason only the one(s) near the WarpStone cause the glitch (still working out the details on exactly which ones)
  • Chopping down a plant after performing this glitch causes it to not grow back. It will grow back if you go back out of bounds or reload the area otherwise.
  • We have not been able to replicate this on other plants in other areas.
  • 1 EXPGFX slot (80 max) and 3 EXPGFX objects, and a couple heap slots are leaked per zoom
  • The crash occurs due to a lack of heap slots, not EXPGFX slots.
  • This means size of objects is not a factor, just the amount of them. This could be a good thing, as we do not have to worry about "object size Tetris", just the number of leaked objects.

Edit 2: Rena also shared the map that was added above to the original post. It seems the leak may occur when passing between the boundaries of two areas while voided out.

Edited by the author 3 years ago
Dark-X-rane likes this
United States

Just to double check, is the gate object for K4 stored in a heap slot? Also, does each object use just one heap slot?

United States

I have an update with some bad news...

Looking through memory, it seems that the K4 gate always loads before the Warp portal, and the K4 object is also smaller than the Warp portal in size. This means that even if we could introduce memory leaks wherever we want and with as many as we want, it would still be impossible to both make the gate not load AND make the warp portal load. Thus, it looks like this trick is a dead end with respect to K4 early potential.

The model for the gate is apparently more complicated than the portal (thanks to Rena for noticing this), so it could still theoretically be possible to use this to prevent the gate from spawning, but this would only be possible if preventing the gate's model from spawning also disabled collision with the gate. Otherwise, the already low odds of this being possible drop to 0%...

San Francisco, CA, USA

It seems even 1x zoom causes nearby zones to unload.

Unlikely to be memory-leak related, this seems to be tied to their zone loading logic. We're slowly learning how this works when looking into void traveling.

====

If we can keep the link loaded, but unload MM Pass, then we can skip the key here.

Edited by the author 2 years ago
josejavier1158 likes this
San Francisco, CA, USA

Can also be used to unload shop textures:

As well as LFV link, but not LFV/CC itself.

Also, this can be used to cause Queen EarthWalker / portal door models not to load, but does not work as a Queen EarthWalker skip.

josejavier1158 likes this
San Francisco, CA, USA

The unloading works outside of ThornTail (except for the leak part). It seems to keep adjacent zones partially unloaded.

This means we can skip the first Tricky gate, but would need to use Cannon clip + save/quit. The time save might be minor, but could be a TAS strat.

Edited by the author 2 years ago
josejavier1158 and BvqRzxi5 like this
United States

To add on to Zac's strat, we can fall in the lava and die to skip having to save and quit. There's a save checkpoint on the platform right before the lava pit, so this lets us respawn immediately on the edge of the platform that we want to be on, with all of the textures fixed, and this is faster than saving and quitting.

Fox can take fire damage from walking into the torches in this section, and can touch the edge of the save trigger while falling from the top ledge into the lava, which deals 1/2 heart of damage from the fall.

Also, if we wait until Tricky is next to the tunnel we have to dig through before feeding him his first grub tub, then the grub tub dialog that Tricky has will cancel his dialog about how he can dig us out through the tunnel from playing, which also saves a bit of time.

zcanann and josejavier1158 like this
Ontario, Canada

Disabling Dolphin MMU should give us crashes consistent with real hardware.

Wrong way around, you need it enabled.

The memory info and such is part of my mod. The original game doesn't show much information.

As far as I know all the magic plants in TTH fail to unload their glow effect when you leave the map by going out of bounds. I don't know why this is. It doesn't seem to happen elsewhere, or when leaving the map normally.

The plants not growing back when you cut them is false. It turns out some plants are just set to take a very long time (as much as 18 hours real time!) to grow back. (Probably some oversight...)

The fact that it seems to be related to having another map nearby makes me think something more than just the plants is involved. If you travel from one map grid cell to another it should load the next map automatically (if one is there), but to manage memory and load times, there are many invisible triggers around that load and unload things, including entire maps. Bypassing these is likely to cause all sorts of odd behaviour. I don't know why the plant effects would have to be manually freed, but maybe?

As for why it takes a different amount of zooms at different points in the game, this is almost certainly just because different amounts of objects are loaded. Each map has two associated values: an "act number" and an "object groups" value. (Not every map uses them, and a few are shared.)

Act Number ranges from 1 to 15 (in practice, I think only 1 to 8 are used) and determines which objects will load when you enter the map. This is what differentiates CC1 from CC2, etc. This can't be changed while you're on the map (or rather, doing so would have no effect until you left); it's usually changed by scripts on some other map.

Object Groups is 32 individual flags (not all are necessarily used) that can also control whether objects appear, but these can be changed while you're on the map. So these are usually used for things like opening doors and blowing up walls.

In addition, there are a lot of random and/or hard-to-predict factors such as weather effects, the general memory layout (a factor of everything you've done from power-on), where exactly everything is (NPCs move around, and objects can load/unload based on distance), etc.

Also: causing memory leaks or loading maps in unusual ways sometimes causes things to "reset" (walls become un-exploded, etc) - this is because you've prevented the "LevelControl" object from loading, which is responsible for loading/unloading objects based on the Object Groups flags (and various other tasks which vary from one map to the next).

Saving and quitting doesn't fix memory leaks, but the reset button should. (Makes me wonder how the new game sequence handles them...) The game also has invisible triggers in the "link" maps (the winding corridors between large areas) which perform a memory "defragmentation" - copying active assets into a reserved memory region and back in a set order to remove gaps. This is when you're likely to see crashes due to low memory.

Edited by the author 2 years ago
zcanann and Dark-X-rane like this
Game stats
Followers
214
Runs
235
Players
40
Latest news
UPDATE: Rules and Category Changes [01/23/24]

As of today, new categories and rules have been added to the leaderboard. The following has changed.

Here are the following new categories and changes added to the leaderboard:

  • Spellstone 2
  • Spellstone 1 will now be a sub-category under the new category "Spellstones" along with S
2 months ago
Latest threads
Posted 13 days ago
0 replies
Posted 2 months ago
6 replies
Posted 8 months ago
190 replies
Posted 9 months ago
25 replies