[Editors’ Note: We close our series on conferences/travel with a discussion of an alternative review model from the most recent ISCA PC Chair.  We welcome your inputs and alternative proposals in the comments section.]

Our review model is under stress because of the growing number of submissions and the increasingly large Program Committees. Natalie Enright Jerger, in a recent thought-provoking and well-articulated blog post, argues to abandon the in-person PC meeting. My experience as Program Chair for ISCA 2020, in combination with Natalie’s blog post and other discussions, led me to think about a new review model. 

For ISCA 2020, I assembled a diverse Program Committee (PC) with 62 experts. We received 428 submissions each of which got reviewed by 3 PC members plus at least 2 additional reviews from the External Review Committee (ERC) or external reviewers. The authors were invited to submit a rebuttal and/or a revised submission. There was extensive online discussion involving all 5 (or 6) reviewers. The PC met in person for 1.5 days to make the final decisions.

My experience from running the ISCA 2020 PC is that an in-person PC meeting is invaluable to “set the bar”, to find consensus if possible, and, if there is no consensus, to clearly spell out disagreements among the reviewers before making an informed decision. I agree that managing a large in-person PC meeting is challenging, and it is true that some PC members get to discuss only a few papers at the PC meeting, which makes it hard to justify the time and effort spent on the PC. However, achieving a similarly effective process through a full-PC virtual meeting, a series of small-group virtual PC meetings, completely online decision making, or a full-journal model as suggested in Natalie’s blog post, in my opinion, seems hard, if at all possible. The problem is not the in-person PC meeting per se, but the review model. 

In this blog post, I propose a novel review model with an in-person PC meeting. The intent is to provide inspiration for rethinking the review model as an alternative to a completely virtual or online PC in case the community’s consensus is to keep the in-person PC meeting. In addition, as we rethink our review model, we should make sure it is scalable with the increasing number of submissions. Finally, we need to guarantee the best possible conference program and provide a fair and high-quality review process for all submissions and authors. 

There are several considerations that led me to this proposal:

• There needs to be a PC meeting to discuss papers. This is important (i) to calibrate scores and paper outcomes across the entire PC, and (ii) to train junior people and learn from more senior PC members. An in-person PC meeting is preferred over a virtual or online meeting so as to have engaging and lively discussions. The carbon footprint is minimized by co-locating the PC meeting with another conference.
• The PC should be small to enable lively discussions at the PC meeting and to ease PC meeting logistics and cost. On the other hand, our community is growing (e.g., ISCA 2020 received 428 submissions, or 17% more submissions than ISCA 2019), and we should engage the untapped potential by soliciting reviews from the broader community. There is a big gap between the number of people who work in our community and the relatively small number of people who review submissions.
• The review model needs to be scalable as the number of submissions keeps increasing. We will soon be reaching 600 submissions in our conferences, and we need to be ready for even more.
• The review process would benefit from a (distributed) mechanism that guarantees an equally high review quality across all submissions, in fairness to all authors.

 I propose a novel review model based on the following principles:

• We need a Program Committee (PC) of limited size which meets in person to make the final paper decisions. Each PC member is the primary reviewer for about 15 submissions, and each submission is assigned to one (primary) PC reviewer. This means that for 600 submissions, we’d need 40 PC members.
• We need a Review Committee (RC) that is large in size and performs the bulk of the reviews. Each RC member reviews, say, 6 submissions, and each submission gets reviewed by three RC members. The RC does not meet in person but heavily participates in the online discussions. Assuming 600 submissions, we’d need 300 RC members.
• The review process works as follows:

1. Each submission is assigned three RC members who review the submission and write a formal review.
2. Each submission is also assigned one primary PC member who reads the submission and who writes a meta-review based on the RC reviews (and her/his own perspective) with detailed instructions for the authors to address in the rebuttal and/or revision.
3. The authors are given the opportunity to rebut the reviews and/or submit a revised version of the paper addressing the review comments.
4. The PC member and the RC members read the rebuttal/revision, discuss whether the rebuttal/revision has addressed the concerns, and decide through online discussion whether the submission needs to be brought up for discussion at the PC meeting. The PC member updates the meta-review. It is expected that about one-third of the submissions will be discussed at the PC meeting (this includes the online-accept papers).
5. A secondary PC reviewer is assigned for the submissions that are brought up to the PC meeting. The secondary PC reviewer reads the (revised) submission, the reviews, and the rebuttal, and writes a second meta-review. For our running example with 600 submissions and a 40-person PC, there will be approximately 200 submissions to be discussed at the PC meeting, which means that each PC member reads an additional 5 submissions.
6. At the PC meeting, each submission is introduced by the primary PC member. The secondary PC member chimes in to provide another perspective. The entire PC makes an informed decision based on the comments provided by the primary and secondary PC members.

There are several pros to this proposal:

• We have a (relatively) small PC that facilitates discussion and calibration at the in-person PC meeting.
• The load per PC member remains approximately 20 papers as is common today.
• Each PC member will be discussing ~10 papers at the PC meeting (~5 of the primary review assignments plus ~5 secondary review assignments).
• By having a large RC, we engage a broader set of the community to review submissions and to participate and contribute to the review process.
• Each submission that gets discussed at the PC meeting receives five reviews: 3 RC reviews plus 2 PC (meta) reviews. Other submissions receive four reviews: 3 RC reviews plus one PC (meta) review.
• The meta-reviews streamline the process, and help identify and remedy potential problems or inaccuracies in the reviews, and also help maintain an equally high review standard across all submissions.

One challenge that remains in this proposal (or any proposal) is the large number of submissions that need to be discussed at the PC meeting. For our running example of 600 submissions, this would mean 200 submissions to be discussed at the PC meeting. This can be addressed by making online-accept decisions for the top-10% submissions – this is ~60 online-accept papers. The remaining ~140 submissions effectively need to be discussed at the PC meeting. This is doable during a two-day PC meeting. If needed, the PC could break out in parallel sessions after an initial calibration phase. 

How long would this review process take? This model is implementable within a period of 13 weeks (or ~3 months). Here is a suggested schedule:

• Weeks 1-2: the Program Chair(s) assign reviews (3 RC members plus 1 primary PC member)
• Weeks 3-4: the RC members review their 6 assigned reviews
• Weeks 3-6: the PC members read their assigned 15 submissions (as primary PC) and write a meta-review
• Weeks 7-8: the authors rebut the reviews and/or revise their submission
• Weeks 9-10: online discussion among the primary PC member and the RC members
• Weeks 11-12: the PC members read the assigned 5 submissions (as secondary PC) and write a meta-review
• Week 13: PC meeting and author notification

Is this model scalable to even more submissions? Yes! Assuming 800 submissions, we’d need 53 PC members who would all read 15 submissions as a primary PC reviewer and 5 more submissions as a secondary PC reviewer. In addition, we’d need 400 RC members each reviewing 6 submissions. Assuming one-third of the submissions are left for discussion at the PC meeting (~260 submissions) out of which the top-10% are online-accepts (~80 submissions), we’d be left with ~180 papers to discuss during a two-day, partially parallel, PC meeting.

To conclude, we as a community need to reach consensus on whether we want to keep the in-person PC meeting. I would argue that we should be careful not to throw the baby out with the bathwater. An in-person PC meeting has several important upsides which we may lose if abandoned. Rethinking the review model may lead to an overall better (and scalable) process with an in-person PC meeting as an alternative to current practice and a completely virtual/online PC.

About the Author: Lieven Eeckhout is a Full Professor at Ghent University, Belgium, and serves as the Program Chair for ISCA 2020.

Disclaimer: These posts are written by individual contributors to share their thoughts on the Computer Architecture Today blog for the benefit of the community. Any views or opinions represented in this blog are personal, belong solely to the blog author and do not represent those of ACM SIGARCH or its parent organization, ACM.