- Sign up for more than one PC per year, year after year (even if you are not underrepresented)
The PC chairs are asking you because you are an expert (though they may not know how many PCs you have been/will be on in a year). The community needs your expertise and you need to be active in the community. So sign up even if you have review fatigue and much work from your day job left over after your previous PC. Never decline because the PC chair’s job will become harder, even if it means your review quality may suffer. After all, it’s not like the raw number of acceptances have gone up so PC chairs can choose from a wider set of proven expertise.
- Squish your reviewing to the last 1-2 weeks before the rebuttal deadline
This squishing gives you a focused time to do the reviews (the two-month review window is precisely so you can squish it to under two weeks). Never mind that typical loads of 15-20 papers mean reviewing more than one paper per day on top of your day job. This is the only way to manage the review load (forget point #1). Besides, authors should be able to make their points clearly and succinctly irrespective of the complexity of the problem they are addressing. If you don’t understand something in your quick scan/read, It’s the authors’ fault for being unclear. Always.
- Try hard to reject (if you are over-negative) (or accept, if you are over-positive)
Your review should be based largely on your personal likes and dislikes, and your own unilateral standards. Historical acceptance rates of the conference and the general community standards don’t mean much. Besides, it’s not your fault that you get exceptionally bad (or exceptionally good) papers, PC after PC. It’s not like the probability of this happening is really low. After all, many PC chairs ask you to be positive, so you should overlook basic flaws in the papers – no paper is perfect (applies only if you are over-positive). If all the claimed contributions are already done then elevate some detail from page 8. When in doubt, give a weak accept. Especially so that you can make up for the many negative reviewers, irrespective of each individual paper’s merit. If you are over-negative, then nit-pick, make mountains out of mole hills, ignore big strengths and go after small imperfections, and, when all else fails, inject vague doubts. Your job is to ensure that if a paper is not perfect it should not get in. When in doubt, give a weak reject.
- Your review should be based more on your opinions than on facts
Even if a paper cannot be rejected (or accepted) based on facts, you should do so. You are the reviewer – your opinions matter more than facts. If you don’t know the material, simply give a weak reject instead of sending the paper back to the PC chair when there is time (by checking the abstracts within the first week of paper assignment). Of course, the PC chair may notice your low expertise and ask for a new, late review which is perfectly fair because the new reviewer is an expert (that he/she may be super-negative or that the hurried review could be wrong without the authors getting a chance to rebut are irrelevant). (I have never seen a late review that is positive.)
- “New and better” is bogus. Novelty is super subjective and a paper need not (show to) be better than previous work in any aspect
Novelty is so subjective that it is hard to agree on what it is. After all, it is so hard to apply the rule that if a claimed contribution is in a previous paper then it is not novel, otherwise it is. Further, if the problem is in a new context then the paper is novel even if the solution is old and well-known. Conversely, if the problem is old, then the paper is not novel even if the solution is new and better than previous work. If a paper on a relatively new topic makes some points, however vague, without being better than previous work in any aspect then accept the paper because someone else may build upon the paper. After all, progress is not made by concrete, new and better ideas but by vague promise of potential future extensions. Also, ignore whether the problem is real or not; as long as there are plenty of graphs and high speedups or you feel the topic is new and important, accept it.
- Published work cannot be wrong (especially if it’s your own)
After all, the work was reviewed by experts like you so it cannot be wrong. And since when have we made progress by correcting past mistakes? Further, don’t declare intellectual conflict of interest with papers that question your work. Conflicts are only for authors – not for reviewers. Besides, you are the expert on the topic, so how will the PC chair find someone else? It’s not like there is more than one person working on a topic.
- Value hard work and numerous graphs over “new and better”
If the paper has prototyped a chip or a 100-node cluster, has hacked a million-line OS, or shows 20 graphs, just take it. At that point, we don’t care about “new and better”. Period.
- In post-rebuttal discussions, simply go by the scores
Not changing your score would mean you are correct which is all that matters. There is no need for further discussions. Your reasons being wrong is irrelevant as long as the majority’s scores agree with yours. After all, who cares that the reviewers may not agree with each other’s reasons as long as they agree on the accept/reject score? There is no need for you to actually read the others’ reviews to make sure you agree on the key reasons for their scores.
- Rebuttals cannot change your mind
Even if a rebuttal blocks all your objections, ignore or dismiss the rebuttal or stand your ground by claiming that you are not convinced. When all else fails, object to the tone of the rebuttal while ignoring the tone of your review. If another reviewer tries to engage you, simply ignore him/her or raise new issues. If your score remains unchanged, most likely the paper will be rejected silently even before the PC meeting. And, no one will know (especially if all the reviewers of a paper adopt this wonderful strategy). Silent rejects are the best.
- If a majority of the PC reviewers disagrees with you, just fold irrespective of how high or low your score is
We don’t want disagreements. If a majority, especially including someone claiming to be an expert, disagrees with your score, they must be right so simply change your score. After all, the PC is not about finding the true merit of a paper, it is about avoiding altercations with your esteemed colleagues. It’s not like we get five reviews for independent evaluation, it is for one big kumbaya. You can fold either after the rebuttal or at the PC meeting. This is especially important for late, unrebutted reviews – there is no need to verify such hurried reviews. A late review, especially by a self-declared expert, is always correct (see point #4).
- You are always right and the authors are mostly wrong
In most cases, you can write/say whatever you want. Everything is confidential and anonymous, and no one will hold you accountable because no one will notice (except for the authors but they can’t do anything).
About the Author: T. N. Vijaykumar is a Professor of Electrical and Computer Engineering at Purdue University and works on computer architecture.
Disclaimer: These posts are written by individual contributors to share their thoughts on the Computer Architecture Today blog for the benefit of the community. Any views or opinions represented in this blog are personal, belong solely to the blog author and do not represent those of ACM SIGARCH or its parent organization, ACM.
