I have had experience with recording my 3ds XL with my phone, but I would really like to get better quality. I was wondering what I should use, a camera to record it then crop, or invest the money into a 3DS mod to capture it. (Which can cost from $250-$500)

I mean a Capture Card is always the best looking option. But if you are new to speedrunning maybe just try some EMU runs ? and if you feel like it invest it later on.