Buildings should have more armour. Although they have different armour types, it looks lame to have only 5 armour for a super-fortified castle. So give them more armour
Also, I believe that buildings shouldn't be destroyed by swords and guns. Seriously, what can swords and guns (normal guns, not crazy rifles or something) do against buildings in the real world? Maybe you should make only certain units able to inflict decent amounts of damage against buildings.
For example you could make all units have 2 attacks, one for units and another for buildings. Then Footman (or whatever) would have decent damage against units but only, say, 2-3 damage against buildings. I know buildings have armour good against almost all attack types, but this could be useful in the following example:
You have some caster who uses explosive magics. It would deal more damage against buildings than units (supposedly) but then the damage would be reduce by the armour so it can be both stronger against buildings (in terms of damage) yet weaker (in terms of attack type, magic).
Finally, make more buildings like Sacrifical Pit. More buildings with special abilities like Bloodlust or something. It would also be cool if you had a building that could train Peasants to Warriors or something like that. It would be cool