IF someone post in your clan DSC after you told them not to, can you report them for it and they get in trouble?

Yes, it's an unwanted invasion and as long as you blatantly told the person(s) to refrain from posting, it's a bannable offence.

Thank you for telling me.
Next time they post after I posted not to post again, I will just report

Although it's obviously highly dependent on the actual post made.
A clan mod will essentially be the mediator in all of it, but yeah, as a general rule that's right.

But, if they are talking shit about me and putting me down, and not a staff at all in any way, and spamming up the DSC

Well, they did that before I told them not to.

But, if they do it again, I can report?

Of course, talking shit or 'flaming' in itself is actually not allowed.
So the perpetrator has more or less broken two rules.
I'd suggest reporting the person straight away if it happens again.

E: I looked into all of this, and what the members were saying about you doesn't really classify as flaming.
It's still an unwanted invasion though.