I don't think I've ever had trouble with the skips on controller. Probroski's run on episode 3 he destroys the breakfast skip with a controller and even if they are harder with a controller, the drag function more than brings a pretty monumental disadvantage. However, the filter for no-controller can pretty much act as the leaderboard, right?

I think it's a big enough difference to warrant it. People who can't get a controller have a decent disadvantage, honestly.