Game suggests you to skip the dialogues with "A" button (on Xbox controller), but if you skip them with "B", it's much faster.

Not sure if holding "A" and mashing "B" is even faster than that, but that's what I'm doing.