I think it shouldn't be allowed for all levels because the game doesn't count it as finished even if you get the key, so in the end it doesn't look like you completed all levels

PD: I want to split when I decide to run this category :(

You have to wait for the animation to play out and then after the fadeout the level counts as complete and you can walk past it. Ideally, you want to have 100% of the levels complete when you view the world, like for example in Bonus island I where it says 22/22. If you skip the animation and just keep the key, you have not technically completed the level, since you cant go past it, therefore, you have not done All Levels.