Why do runners not use the Star% glitch when they run Any%? Is it against the rules and if so, why isn't "Any% with Star% glitch" a category on its own?

I assume you're talking about the trick called Bruncheon. It's not that it's not allowed, it's that it isn't viable. It isn't viable in Any% because the Odyssey doesn't spawn in Luncheon Kingdom when you do Bruncheon to reach Luncheon Kingdom early. Thus, you'd have no way of progressing the game. This trick is viable in Star% because there's a painting in Luncheon you can access at any time, and it takes you directly to the star you need to collect.

Actually, optimally it is only around 3 seconds slower than doing Luncheon normally, because it allows you to skip 2 slow Story moons. However, it is also much harder to do optimally so nobody even considers practicing it right now

If you are to use the painting to warp to mushroom after using Bruncheon the odyssey spawns there, which could theoretically make this trick work, but the jump down there is impossible due to an invisible wall. The jump has been proven possible by Wet Nut Jumping up from the mainland, which removes the barrier, but as far as we know there is no way to pass through the wall without coming from the mainland. Theoretically we could then take the painting to Snow (assuming the odyssey does spawn there when we do this) , which would end up skipping a fair portion of the game, but the tricks required would be incredibly difficult and either way it's impossible right now.