As with all scientific communities, the field of computer architecture is driven by publications. At the heart of a meaningful scientific publication model is a fair and sound peer-reviewing process. If this process is compromised, some papers might be unfairly rejected while others might be accepted for reasons other than sheer merit.
One may think that the reviewing process starts and ends with reviewers; however, that is not so. The program committee (PC) chair, authors and reviewers all have significant responsibilities in this endeavour. For instance, the PC chair helps ensure a fair and unbiased review process by verifying stated conflicts of interest and assigning reviews taking into account the validated conflicts and reviewers’ expertise. Authors, too, have a number of responsibilities, including stating all of their legitimate conflicts of interest, and only those. Authors also must not take any action to somehow identify and/or sway the reviewers of their paper.
At present, the process of stating the responsibilities of all parties in the reviewing process — i.e., the PC chair, authors and reviewers — is somewhat ad hoc. The responsibilities of PC chairs for our field’s top conferences are typically set by the steering committee of each individual conference with little or no coordination across conferences. The responsibilities of authors and reviewers are determined by the PC chair in coordination with the steering committee for the particular conference. Recently, our conferences have moved to spell out the various rules and responsibilities related to reviewing. Along the way, they have borrowed ideas from each; however, there remains significant variance across different conferences and even instances of the same conference with regard to reviewing guidelines.
In an effort to clearly define the responsibilities of all of the parties in the reviewing process, ACM SIGARCH and IEEE TCCA formed a working group composed of two members from each organization’s executive committees (ECs). The working group examined guidelines from recent conferences and the 2013 document providing guidance to PC chairs. Using that as a starting point, and following extensive discussions with input from each of the two ECs, the committee created a draft document titled “SIGARCH/TCCA Best Practices for Conference Reviewing Process”.
The document spells out what PC chairs, authors and reviewers must do, as well as what they must not do. The document also suggests a number of recommended practices in regard to reviewing. Ultimately, PC chairs retain significant latitude, but they can now operate with a robust and uniform set of rules.
Those in the community who have had recent experience submitting and/or reviewing papers for our leading conferences will be familiar with much of the document’s content. A few items are, however, different from established practices. One such item of note is that the duration of a conflict of interest for recent collaborations has been set to four years. In recent past, this number has varied across conferences, but five years has been common.
The document also spells out a policy for how investigators from different institutions who are part of a research structure under a shared umbrella funding award, such as an SRC JUMP center in the US or a Horizon 2020 project in the EU, should be treated for reviewing purposes. The concern is twofold: First, the appearance of a financial incentive among members to promote each other’s work — indeed, publications and research visibility are often part of the performance review of these research structures. Second, routine exposure to non-published work in a community where double-blind review is long-standing practice. This issue proved somewhat contentious. In the end, it was agreed that conflicts with other members would apply systematically to the leadership of these research structures, as well as among members who regularly participate in non-public meetings sponsored by the award where they are exposed to presentations or discussions of unpublished work. (Naturally, traditional rules for active research collaborations still apply.)
Notably absent from the document are policies governing two hot topics. The first is whether students are allowed to assist with reviews. This is left under the discretion of the PC chair; however, the document does mandate the PC chair to have an explicit policy governing student reviewing. The second is the use of social media to promote/discuss a paper under review. While the document does not disallow the use of social media, it does prohibit attempting to sway the reviewers (who may be exposed to the social media posts from the authors) and also bans disclosure of the content of reviews until submission decisions have been publicly announced.
The draft document has been approved by both SIGARCH and TCCA ECs. Once finalized, it will apply to all conferences exclusively sponsored by TCCA, SIGARCH or both jointly, which includes ISCA, HPCA and PACT. We hope that other organizations in our community will adopt these guidelines and we would be happy to engage with them in the process.
We now invite the broader community to review the document and to provide comments using a brief survey form available at http://bit.ly/confreview-survey. We would appreciate your feedback latest by May 31, 2021.
About the Authors: Daniel A. Jiménez (Texas A&M) and Moin Qureshi (Georgia Tech) are members of the TCCA Executive Committee. Boris Grot (University of Edinburgh) and José Martínez (Cornell University) are members of the SIGARCH Executive Committee.
Disclaimer: These posts are written by individual contributors to share their thoughts on the Computer Architecture Today blog for the benefit of the community. Any views or opinions represented in this blog are personal, belong solely to the blog author and do not represent those of ACM SIGARCH or its parent organization, ACM.
