i saw a runner using custom palette, so i'm wondering if it's allowed to use custom palette to submit it for runs?

The rules are:

Must beat Wario without using glitches. Timer starts when entering the start pipe and stops on the 3rd hit on Wario 3. (Emulator submissions: please include which emulator you are using! VBA is banned for the entirety of GB/C speedrunning! BGB and Gambatte on their default settings are recommended. Use other emulators at your own risk.)

custom palette is not a glitch, and it cant be used to hide splices so I don't see why not. I'm not a mod though, so I can't be 100% sure.

Custom SGB(2) palettes are perfectly fine and do not affect the game.

Some additional info: There's a distinction to make for "SGB-enhanced" games with a custom game-specific border or special sound effects. Those special assets have to be loaded during gameplay which usually results in the games running slower than without those enhancements (e.g. if played on a regular Game Boy or a GB Player on the Gamecube). You can find out whether a game uses SGB enhancements or not here: https://en.wikipedia.org/wiki/List_of_Super_Game_Boy_games Only games that use custom (changing) borders and/or custom audio are affected. A mere custom default palette is not an indicator for an SGB enhancement, nor do the default SGB borders affect gameplay.