short answer is yes.
But really that is up to you, and that person ( and there fed ? )
If said person is inactive then feel free to take there slaves. ( they are not using them anyhow )
But, if they are in a good fed, the fed will either stand up for them, or feel it is there right to raid said inactive person.
if teh person is active, and you just go in and and dec them, there may get mad at you, and start a drawn out war.
I personally would send them a message acing if they are active, or atleast message them as i dec them. if i have no " good reason " i claim manifest destiny.
ON a side note, I am big on the " role playing " of games. and think it is funny to play if you draw out a DoW, and either send it to them or in forums. stating why I feel there land should be mine and so on.