What if you fight back without actually "hitting"? Shoving or pushing them down would be harmless wouldn't it?