As soon as you hit the act 5 trigger (move to the rocks behind the shed at the end) the priest will walk to the temple on his own.

So I tried launching myself there and ended up being faster, yay!