Peer-reviewed articles in top conferences are both the result and the driving force of computer architecture research. As our community expands and the number of submissions to top conferences continues to grow, it will be imperative to improve the review process. Maintaining high quality, fairness, and integrity while not overly burdening reviewers and authors will require innovation. This blog post summarizes a proposal for improving and streamlining the review process for our top architecture conferences: ISCA, MICRO, and HPCA.
The Proposal
The proposal seeks to improve paper quality and reduce review workload by coordinating reviewing across the conferences. Reviews would follow papers as they are rejected and resubmitted. For example, a paper rejected from ISCA and resubmitted to MICRO would include the ISCA reviews and author response, and be reviewed by mostly the same reviewers. A paper submitted to a given conference would have three possible outcomes: accept (possibly subject to revision/shepherding), reject, or invitation for a larger revision and submission to one of the next conferences. Program chairs would cooperate to ensure that there is significant overlap between the reviewers for the same paper between conferences. This process would have several beneficial effects:
- Knowing that many of the same reviewers would be reviewing the paper again would encourage authors to take heed of the feedback given in reviews, rather than continue to resubmit the same old paper hoping to get a “lucky” set of reviewers.
- Keeping many of the same reviewers would ensure consistency in the review process for authors. Reviewers could make clear what improvements would be required for acceptance, and commit to accept the paper if the changes are made.
- Reviewer workload would be reduced as reviewers already familiar with a resubmitted paper can check whether the paper has improved sufficiently rather than treat each paper as they would a new submission.
- Coordinated reviewing makes explicit a concept that most of us already know intuitively: we are one architecture community with multiple top conferences which, although they have some different emphases, are equivalent in quality.
The proposal would require the leadership from each of the three conferences to cooperate. Program chairs would agree to keep some members of a previous conference’s program committee (PC) on their external review committee (or PC) to review resubmitted papers. Program chairs would agree to make reviews available to the next conference in a way that protects privacy and respects new conflicts of interest. The steering committees for the conferences would agree on details of the implementation. The proposal also includes the idea of having a journal jointly sponsored by ACM and IEEE to publish papers accepted at any of the three conferences.
What Do You Think?
At this point, we would like to ask the broad computer architecture community to provide feedback on this proposal. We have posted the draft proposal on the TCCA website (http://ieeetcca.org/feedback_community_proposal/). We have set up a web form to collect anonymous feedback, and a board for public, optionally anonymous discussion. We would like to know what you think. Does this sound like a reasonable idea? What suggestions do you have for us?
History of this Proposal
The proposal is authored by nine members of the TCCA Executive Committee named in the linked document. The TCCA Executive Committee was asked to vote whether to post the proposal for broad community feedback. The vote passed by 11 to 6. A group of dissenting voters authored a thoughtful critique of the proposal. They raised several concerns. Among them: 1) Different envisioned rates of accept/reject/revise may lead to substantially different operation of this proposal; 2) It is unknown what fraction of submitted papers are resubmissions; 3) It is unclear how to handle rejected papers that are resubmitted to the next conferences without being invited; 4) Conflicting policies within IEEE and ACM may present challenges to a common journal and to sharing reviews. These and other concerns raised are important issues we believe should be studied and addressed.
About the Author: Daniel A. Jiménez is a Professor in the Department of Computer Science & Engineering at Texas A&M University. He is interim Chair of the IEEE Computer Society Technical Committee on Computer Architecture (TCCA).
Disclaimer: These posts are written by individual contributors to share their thoughts on the Computer Architecture Today blog for the benefit of the community. Any views or opinions represented in this blog are personal, belong solely to the blog author and do not represent those of ACM SIGARCH or its parent organization, ACM.
