are there any rules like everyone has to be alive

Nah, I don't think so.

so what your saying is one person can die as fast as possible then the other can play normally?

If the run is faster, while one player is dying repeatedly, then I don't see why not.
Honestly, I just assumed there wouldn't be any other rules, than in Any%.

huh that kind of destroys the co-op run now as it could be faster than normal run

like get 4 people 3 kills and 1 goes through the level as fast as possible then on boss levels the 3 people don't die

thats the best way but some levels you might have to find a way to keep everyone alive because it might take longer to have 3 killed at the start.
I have another idea, only 2 people for the normal levels but then on the boss levels two people jump in. unless this is against the rules.

Nobody has ever really taken the time to determine rules for coop. I made one of the rules. I'm pretty proud of it. I'm the reason you can't run coop solo. Of course to get this rule change I had to submit a WR time and have it be rejected. The community is very inactive.
So long as we are discussing coop rules. I think as long as 2 bros are present at some point in each level, the run should be valid. Even if the run was done with mostly 1 bro at all times, the faster times would use the second (or third and fourth) players to improve the chances of getting the right bro for each level (and presumably whoever gets the fastest character would play that level). I think that is interesting enough to warrant the coop category.