As far as general SE policy is concerned, you are entitled to downvote whatever you want for whatever reasons you choose, but targeted downvoting is not allowed. However, policy is different from the generally agreed upon etiquette of the site, and the generally agreed upon etiquette is that you should downvote answers that are incorrect, incomplete, incomprehensible, or don't answer the question asked. But since voting is private, you can do whatever you want, as long as you don't target a specific user or group of users and arbitrarily downvote them (the system automatically detects targeted voting patterns and reverses the votes in question; the user responsible for the votes may be suspended or banned for the irregularities).
Meta SE FAQ Recommendations for Downvoting Answers:
Try to comment whenever applicable when downvoting.
[Reasons to downvote]
Does not answer the question. Flag at your discretion.
Makes no sense
FGITW [Fastest gun in the West, i.e., submitting an inferior or incomplete answer as quickly as possible just to beat everyone else to the punch] post with minimal answer: This depends upon the answer really.
Link-only answer
Is wrong (in your opinion).
Do NOT downvote just because:
It's a competitor. Voting is for post quality, and you end up harming the system. Also, upvoting competing answers gets you something shiny.
Again, grammar that does not affect comprehensibility too much.