In the rules for the category it says online is banned, but says nothing about whether co-op is allowed or banned. If it is banned, can there be a different category for co-op?

There is a variable for controllers on Colosseum in which you can use if you believe you used Co-op mode during the run. I don't recommend Co-op mode, though, as it causes the player's damage output to be reduced to compensate for having a second player.