It turns out I was missing something glaringly obvious in my runs; in Level 4, you clearly do not need to enter the middle-left room to complete the level.

I think when I first started playing, my eyes jumped to the key in that room and I thought "Okay, how do I get that?", and never corrected this mistake.

Many thanks to the commenter who pointed this out to me. A time under 23:40 definitely seems achievable now.