I am trying to set up Project64 1.6.1 to use my switch pro controller. However whenever I try to bind buttons like B or Y I get an issue where the control just says "Keyboard: ". I've been able to set up the analog sticks using the N-Rage's Direct-Input by closing and reopening the bind window until it finally works. Using any of the other Controller Plugins causes Mario's movements to walk everywhere instead of running and no amount of changing the deadzone's or range fixes it. The other plugins take the button inputs just fine but not the analog inputs, however N-Rage's Direct-Input takes my analog stick movement correctly but doesn't take button inputs. Has anyone been able to set a pro controller up properly?

I'm not sure, but I don't think that the switch pro controller is fully compatible with pj64.

do you have any XBOX or PS4/5 controllers laying around that you can connect via USB? that's what I used to use before I got a refurbished n64 controller and an adapter, and it worked absolutely fine. same layout(generally, although you do have pressure-sensitive triggers, not button triggers like on the switch, although I think you can mod an Xbox/ps4/5 controller to have the same button style triggers as the nsp), better analogue input honestly, and probably most important of all, more compatibility.

I use ProconXInput, I wasn't having the same problem as you but it's worth a shot

https://github.com/MTCKC/ProconXInput

I was finally able to get the pro controller to work in P64 V1.6
For a copy and paste solution see the last paragraph, to see how I solved it in case my solution doesn't exactly work for you, read on.

It's near impossible to get any of the input plugins to correctly get analog stick inputs and the ones that do don't seem to correctly get the gamepad buttons to work. However, the one that I got to work was N-Rage Input Plugin, though not by conventional methods. I save a profile to a file and work backwards through how they encode the inputs to end up getting my controller to actually work.

For anyone trying to get this to work in the future this is how I did it. Basically every input slot in the configuration screen is represented by 8 hexadecimal digits, for example 00020001. The way we can figure out what this means is by splitting up the first 4 and last 4 digits. The first 4 digits represent the actual input value while the last 4 digits appear to represent the "device" that the input is coming from. 0001 meaning it comes from the gamepad buttons, 0002 meaning button 2 on the gamepad. To figure out which 8 digits are for which slot I just swapped one of them between inputs and look for which 8 digits change in the file (I will reply to this later with a labeling of which 8 digits are for which input slots).

This is the encoding that I've been able to work out so far, I may edit this with more info later.
First 4 digits from the left when the last 4 digits are 0001 (i.e. in 00020001 focus on 0002)
0000-0003 represents the 4 letter buttons (A, B, X, Y) though not in order
0004 represents L-Trigger
0005 represents R-Trigger
0006 represents Z-Trigger
0009 represents Start Button

First 4 digits from the left for Axis and Rotation Positive Movements
0000 represents the X Axis (Left Stick)
0001 represents the Y Axis (Left Stick)
0003 represents the X Rotation (Right Stick)
0004 represents the Y Rotation (Right Stick)

First 4 digits from the left for Axis and Rotation Negative Movements
0000 represents the Y Axis (Left Stick)
0001 represents the X Axis (Left Stick)
0003 represents the X Rotation (Right Stick)
0004 represents the Y Rotation (Right Stick)

Last 4 digits from the left (i.e. in 00020001 focus on the 0001)
0001 represents Gamepad Buttons (I don't think this includes the dpad but I haven't looked much into it)
0005 represents Keyboard Inputs (Say you wanted to press W the whole thing would be 00110005)
0102 represents GamePad Axis and Rotation negative movements (Axis is left stick, Rotation is right stick)
0002 represents Gamepad Axis and Rotation positive movements (Axis is left stick, Rotation is right stick)

The final control combination I got to work for me was this
Controls=00010003000200030003000300000003000900010006000100020001000000010003000200030102000400020004010200050001000400010000000200000102000100020001010200000000000000000000000000000000
You should be able to copy and paste this into an N-Rage Input Plugin profile file right where you see the controls section. To open one of these files just use some sort of text editor. This control combination only includes the left analog stick, right analog stick, back buttons, letter buttons, and start but this should be all you need for SM64.

holy moly, what a solution

Can you share the joystick save file?

I recommend Octiminos Controller Input

I recommend simply connecting your controller to your PC with USB and it worked for me. I don't use a switch controller anymore but I used to.