General idea (would require custom models and advanced coding)
Tank tower can rotate separately from the tank itself. Missile is shot in the line, hitting any obstacle. Tank moves not like WC3 unit, but more like a real tank (no deliberate turns on spot, etc.). Tanks have different points where a missile can hit (front armor, side armor, cannon, tracks, fuel tank, etc.), with different effects from that.
Destroyed tanks leave nigh-indestructible debris, which can be used as obstacle.
(All of this is inspired by a very cool russian game, roughly translated as "Behind Enemy Lines". There was a very realistic tank combat. Such system allows very interesting tactical combat and requires quite a skill to master. A skilled player can alone defeat several heavier and more powerful tanks, while riding a lighter and weaker one, simply by using cover smartly and shooting only when the moment is right.
Although such system is hard to make.