I noticed I was getting better times for Time Attack when I was grinding with the Joy-Con "D-Pad" vs the pro controller's stick and pad. Could have also been a me thing, but also curious what the consensus is or preference.

Here is a better clarification, Pro Controller is used for speedruns most of the time. Using a D-pad on Pro controller is a bit difficult to play with despite how Nintendo designed it before they release the Pro Controller after Switch console release launch. (I have that problem all the time). For example, you can press the d-pad down completely but sometimes it goes right or left since they're close to each other. Personally, I would use the circle pad for a comfortable position without having d-pad jinx your run, but again that is a preference.