Slight means of little consequence, barely perceptible or scanty. If a person is described as slight they'll be slender or frail looking. If you refer to someone slightingly you'll be treating them as though they're insignificant. I think that's more than slightly rude.
Historically castles were slighted to make them useless for military purposes. This kind of damage can be seen at Kenilworth. (There are more shots of the castle and it's grounds on my gardening blog)