Many CS systems and architecture conferences have a rebuttal phase or an author response period, where authors can respond to the questions asked by reviewers, and correct misunderstandings by reviewers. We discuss the pros and cons of this phase, and whether it is a net positive or negative for the community.
The author response phase
The author response phase works like this: once all the reviews are in, they are released to the authors, and authors get a limited period of time (ranging from 48 hours to two weeks) to respond to the reviews. The response usually has a word limit (ranging from 500 to 3000 words), beyond which the reviewers are not obligated to read. Participating in the author response phase is nominally optional.
There is some disagreement about what the author response should respond to. One view is that the author response should be limited to correcting factual mistakes in the reviews. (“You didn’t do X”, “we show results for X on page 4”). Another view is that it can be used to argue about subjective things as well (“your paper is not novel”, “here is why it is novel”). Call for Papers and Instructions from PC Chairs are typically vague about this point.
The author response phase can be implemented in different ways. One implementation choice is whether reviewers are required to update their reviews and scores after reading the reviews. If this is implemented, the authors get feedback from the reviewers on how the rebuttal affected their reviews (and ultimately the final decision for the paper). Unfortunately, this is not a common implementation choice. Typically, author response phases are one-way streets: authors send in their rebuttals, but reviewers are not required to change their reviews or scores to indicate that they have even read the rebuttals.
Yet another implementation difference is in whether reviewers have to provide a list of questions that should be answered in the rebuttal, or whether authors should attempt to respond to everything in the review. This second option can be quite tricky for authors, given reviews can be quite long and detailed.
The cost of the author response phase
An author response period is not without cost – it is a massive time commitment for the entire community. For example, over a million words were written in rebuttals for CHI 2020. For the typical systems conference with 250 submissions and a 500 word limit, a conservative estimate is that over 125,000 words will be written for this phase.
Apart from the time commitment, an author response phase causes significant stress among authors (especially students). Authors are forced to work under time pressure (typically a couple of days) to craft a good response. This can be very stressful, since the final fate of the paper can depend on the material in the response.
Why have an author response phase?
So given this cost, why should a conference institute an author response period? There are two main camps of thought here.
The first camp is well represented by Kathryn McKinley, who argued in her 2008 article (updated in 2015 here), that the author response period can have three main benefits: 1) It helps keep reviewers accountable, 2) It helps get the reviews in before the PC meeting, 3) It helps the PC chairs get additional reviews if the paper seems controversial. So overall, the benefit here is a better review process. If this is the primary goal, not providing feedback to the authors in the form of changing scores or reviews after the rebuttal is understandable.
For the second camp, the primary benefit of this phase is more down-to-earth: it helps to hopefully increase the score of their paper by correcting factual errors and misunderstandings. For this camp, a rebuttal phase where paper scores and reviews won’t change would not be a good use of their time and energy.
Why not have an author response phase?
Author response phases, as commonly implemented today, have several drawbacks.
Rebuttals simply don’t affect most papers. First, and most concretely, they do not change the scores or reviews for the majority of papers. For example, for 41% of papers at CHI 2020, the rebuttals did not change the mean score. Only 0.5% of papers had the rebuttal change the decision from a reject to an accept. Over a million words were written by the community for this small effect. I do not know of a similar study for systems/architecture conferences, but I would estimate the effect to be similar.
Rebuttals don’t actually keep reviewers accountable – other reviewers do. You might argue rebuttals are worth doing since it provides a sorely needed mechanism for keeping reviewers accountable. Sadly, this only works if the paper has a champion who can use the material in the author response to argue for the paper in the PC meeting. Without a champion, reviewers who make fundamental mistakes about the paper simply ignore the rebuttal and move on – there is no one to hold them accountable since most reviewers are overloaded and focus their attention on the papers they want to champion.
Optional, but not really. The author response phase is indicated as optional in most conferences. However, this is not really the case: I’ve seen several reviewers negatively rate a paper because the authors did not choose to respond, even though the phase is marked as optional. So authors are forced to respond, even for papers that are clearly on the path to acceptance, for fear of annoying the reviewers.
Time pressure and stress. The author response period is a stressful time for all the authors involved. You have limited time to put together a response. You might have to comb the reviews to figure out which questions are worth answering. There might be dozens of complicated questions, and you have to answer each one for fear of annoying reviewers. I don’t know anyone who actually enjoys this process (on the author’s side).
The hidden curriculum. At conferences where the word limit is not enforced, I’ve seen some authors send in short rebuttals under the word limit; reviewers then get annoyed when their questions are not answered. More savvy authors typically send in rebuttals that focus on common/important questions under the word limit, but then answer every reviewer question in detail. Reviewers simply search for their questions and see what authors have responded. While some conferences explicitly ask authors to format their rebuttals in this way, many do not.
How can we improve the author response phase?
There are a number of simple changes we can do to improve this phase significantly:
- Require reviewers to edit their reviews in response to the rebuttal. There is nothing more demoralizing than spending several days of time and effort into writing the rebuttal, only for neither scores or reviews to change. If the author response phase is important enough for the community as a whole to spend several days on this, the reviewers should be able to spend a few minutes updating their reviews and scores.
- Only open the author response phase to papers that are going to be discussed at the PC. First-round rejections should not be allowed to participate. If rebuttals can cause a rejected paper to get an entire second round of reviews, there should be more time budgeted between the rebuttal phase and the final PC discussion (usually it is about one week, time enough to get one new review, but not several). Similarly, the author response phase should not be open for papers that will clearly be accepted. We should limit the effort on the part of the community (and the PC).
- Have reviewers list questions they want answered. Limit the questions. Disallow “please see my review for questions”. The author response should be focussed on these questions.
- Be clear about whether the word limit is strict or “more like guidelines than actual rules”. If authors are allowed to submit responses past the word limit, suggest how they should format the rebuttal.
- If the main intended benefit of rebuttals is reviewer accountability, assign a reviewer for each paper to read the rebuttal and engage the other reviewers. Reviewers need to hold other reviewers accountable.
Overall, I feel like author response periods today result in a lot of work without a net advantage for the community. Senior researchers tell me it was much worse without rebuttal phases; however, this has not been my experience either as a PC member or an author in conferences with and without author response periods.
I have argued previously for one-shot revisions: such revisions include a cover letter where the authors can respond in detail to the reviewers without worrying about word limits. I feel this achieves some of the intended benefits of rebuttal phases without many of the negatives: not all papers are given revise decisions (focussing community and reviewer time), there is no immediate time pressure, and you can respond to reviewer comments in detail.
Our conferences are undergoing rapid and significant change: things like multiple deadlines, revisions, and extended abstracts are being introduced. I feel the time is ripe for our community to revisit the rebuttal question, consider what we are aiming to get out of a rebuttal, and if the way we are implementing rebuttals is indeed the best way to obtain those goals.
About the author: Vijay Chidambaram is an Assistant Professor in the Department of Computer Science at the University of Texas at Austin. His research group, the UT Systems and Storage Lab, works on all things related to storage.
Disclaimer: These posts are written by individual contributors to share their thoughts on the Computer Architecture Today blog for the benefit of the community. Any views or opinions represented in this blog are personal, belong solely to the blog author and do not represent those of ACM SIGARCH or its parent organization, ACM.
