You have to tear down any building that you build on land that you do not own by attacking it. Destroy only works on land that you own,.
I have noticed that players build a Ballista and then build a palisade wall around it. They should have to tear down the Palisade wall the hard way and NOT be able to use Destroy to remove the wall.
My idea is to ONLY allow the Destroy command to function on the p[ayers own land.