I reviewed a number of questions that were suggested for closure. For most of them it was a judgement call as to whether they were, for example, off-topic. Several of the questions had already been upvoted more than 4 times.
My question is, when considering to vote a question for closure, should the number of upvotes be considered in making the judgement? After all these votes indicate that the topic is of interest.
Yes, it should be a factor in the decision.
Closing for duplicate is a special case - there is no reason to avoid upvoting the question, or closing a highly upvoted question in that case. The duplicate question could be better asked than the original and worthy of upvotes and attention, which lead people to the other question to actually post answers. That is one of the functions of duplicate questions and why they are not deleted.
Regarding other questions, if the community here thinks that the question is "useful and clear" (language from the tooltip on the upvote button) then it is an indication that it is relevant to the community. However, the votes don't determine the on-topic nature of the question since up votes may be for ancillary reasons emotional reasons about the question, or they represent a different opinion about the on topicness that you, as a user, have the right to vote differently. They are one piece of evidence to weigh in the decision.
The main indication that they serve is that while you can get votes to close a question, SE doesn't support counter-voting to keep it open. The up vote and comments are the only voice that those who disagree with the initial VTC have to express their opinion, and they should be analyzed to see if that appears to be their motivation and if so, consider, as a close voter, to respect that view.
After all these votes indicate that the topic is of interest.
Unfortunately, interesting, does not an on topic question make.
To borrow from this excellent answer: the cons of letting an off topic question through even if it is upvoted include:
.. the noise of a non-technical non-helpful question.
.. and the bad precedent that confuses new users about what is OK on the site.
.. and the noise of having many of the high-vote questions be 3 year old off-topic questions.
Just because it's cool or popular doesn't mean it's appropriate, and that's the root of this issue. Some questions are not appropriate for the site.
There's a thousand pages with cooking info, historic info, technical info, etc.
However, this is really only one of the only places to get extremely high signal-to-noise-ratio technical questions and answers, and as the site gets more popular we need to be more careful about sticking to that core value.
It's what makes very gifted people who excel in their fields continue to contribute here, and it's why we all spend so much time here learning and teaching.
For topicness determination, we have the FAQs.
