As long as you don't attack the towers and instead tank them with rams, the A.I. won't aggro and building the castle is just a joke.