Is it possible to submit a valid run even if your 3DS has Custom Firmware?

do it. if mods will reject the run, have a little chat with them.

Unless stated otherwise in the rules then yes, I run a number of games through emulators on my 3DS and use kit-kat to wirelessly stream/record them, just don't cheat and you'll be golden