Well sadly this game loads way too slow unless you use the gamepad, and even then the loading time isn't great.

it has very long loading times in general yeah. not much to do about that ;)

Yeah, if you compare the pro controller loads with gamepad loads it might vary ~2 or 3 secs each loading screen in worst case scenario, it's pretty dumb.