I see that as noise, and I prefer to downvote noise. Not voting is reserved for comments with very small contributions or opinions I disagree with and don't seem properly justified. But that's just me.

I agree that downvoting is a harsh punishment, but it's also my opinion that this "noise" should be differentiated from the above cases, so I don't see any way around.