Sorry that this took so long to get back to, Im kind of out of the sun haven scene and I couldnt reply to this for a while for whatever reason (Like legit website wouldn't let me) But yeah, time ends as soon as the farm loads after you take the griffon or the ferry back to the farm (whichever you unlocked second)