TL;DR

The ASPLOS Steering Committee is considering two changes to the ASPLOS submission process: 1) three submission deadlines spread over the year, and 2) the possibility for papers near acceptance to be revised and resubmitted. This proposal outlines these changes.

The steering committee welcomes comments and suggestions for improvement (please read the whole thing first) in this online survey (by December 31, 2021).

Rationale

ASPLOS is a unique conference that lives at the intersection of computer architecture, programming languages, and systems. It provides a venue for the best interdisciplinary research in these fields and supports a strong community of researchers who share a cross-disciplinary focus. The conference has a long and distinguished history, and has remained relevant by evolving while remaining a premier venue for publishing work on the challenges in building and applying complex computer systems.

Like many successful conferences, ASPLOS has recently received significantly more submissions, which has taxed its review process and frustrated researchers and paper authors. For example, ASPLOS 2021 received 398 submissions, while only 5 years earlier, ASPLOS 2016 received 232, around the average volume for many years. This increase is a natural consequence of the growth of the academic and industrial research community and increased cross-disciplinary innovation in cloud and mobile computing.

A larger volume of submissions has led to larger program committees as part of a concerted effort to improve the quality of reviewing and feedback to authors by reducing the number of papers a committee member reads. ASPLOS 2016 had 42 PC members (and 79 external review committee members), but ASPLOS 2021 had 75 PC members (and 115 external members). After a certain size, both in-person and online meetings become less effective and more challenging.

Despite this increase and other innovations, the ASPLOS Steering Committee feels that further changes are necessary to improve reviewing for both authors and reviewers. The Steering Committee has heard many concerns raised by the community, and it believes that a significant change to the ASPLOS submission process is needed to address them.

This paper describes our proposed change to this process. We welcome your feedback and suggestions and will consider them before we implement the change. Please submit them by December 31, 2021 in this online form.

Proposed Change

This document presents a proposal for a new review process for ASPLOS. In practice, ASPLOS Program Committee Chairs have considerable discretion in running the review process, so the Steering Committee expects there to be several years of experimentation and innovation before we settle on a process that fully meets the community’s needs. While these ideas are new to ASPLOS, top conferences in many other communities have implemented similar approaches.

If the proposal is adopted, the next ASPLOS would have three submission deadlines. A paper submitted to any of the conference’s three deadlines will be reviewed by the ASPLOS program committee (PC) and considered for acceptance at the conference in the same way as is currently done. Each submission will receive one of three responses by a fixed date after the submission deadline:

1. Accept. The paper is accepted for the conference. The PC may suggest (minor) corrections.
2. Reject. The paper is not accepted for the conference and may not be resubmitted to a subsequent deadline for the same year’s conference.
3. Revise. The paper is not accepted or rejected. Instead, the PC will suggest changes to the paper or request additional results. The authors may submit a revised paper to the subsequent ASPLOS deadline (i.e., second or third). A revised submission can only receive an Accept or Reject, but not a second Revise. 

Papers submitted at the third deadline will not receive a Revise but should instead be submitted afresh the following year.

We anticipate that only papers close to acceptance will receive a Revise. This option is an opportunity for the PC to ask for changes or additional experiments to push a borderline paper into the Accept category. However, the amount of work is bounded by one month by the submission schedule, so immature or weak papers will not be offered this opportunity.

The ASPLOS PC will serve for a whole year, handling the submissions for three deadlines. The submissions will be reviewed in the same manner as the current PC: a set of members of the PC and EPC will be assigned to write reviews, and the submissions will be discussed and decided upon by the full PC.

Aside: Why 3?

The Steering Committee is not unanimous in believing that 3 deadlines per year are a better choice than 2, 4, or even 12. Two deadlines only offer an opportunity for papers submitted at the first deadline to be revised and would therefore likely not reduce the number of papers submitted at this time. Four deadlines leave only 90 days between milestones, which is insufficient for the current ASPLOS reviewing process and a revision period. The Steering Committee, looking at the experience of VLDB, did not want to move to a monthly reviewing cycle because of the continuous, year-long burden on the PC. We are not sure that 3 deadlines are the best choice and are willing to change based on experience. We will carefully monitor this new process and change it if it does not appear successful or if serious problems arise.

Example

Looking ahead a few years, when the new process has stabilized, ASPLOS 2025 will occur in March 2025. It could have the following schedule:

• Submission deadline 1: March 1, 2024, around ASPLOS 2024.
• Response deadline 1: May 15, 2024
• Submission deadline 2: June 15, 2024
• Response deadline 2: Sept 1, 2024
• Submission deadline 3: October 1, 2024
• Response deadline 3: December 15, 2024

These dates assume a similar review period to ASPLOS’s current one and leave a month for revision before the subsequent deadline. If the volume of papers submitted at each deadline decreases, the review period could be reduced, and the interval for revision increased. Without some experience, it is difficult to predict what will work for the community and the PC.

Three deadlines offer researchers a different set of trade-offs than today’s schedule. For example, Alice is a student working on programming languages to solve NP-hard quantum computer problems and believes ASPLOS is a great place to publish her research. She needs to decide when to submit. If her work is mature, submitting it to an early deadline allows her to revise her paper if needed. On the other hand, submitting to a later deadline gives her additional time to improve her paper and reduces the amount of time her paper is “under submission.”

Discussion

We have several hypotheses as to why this change will improve the reviewing process for ASPLOS. All need to be measured and evaluated.

1. The workload of the PC will not increase significantly, although the duration of its work will extend over more of the year. (Not every Steering Committee member agrees with this contention.) 
2. Three smaller PC meetings per year will be possible. 
3. Spreading the reviewing over the year permits higher quality reviews and discussion.
4. The submissions will be spread among the three deadlines without too great of an imbalance. Authors will not all rush to submit at the first deadline.
5. Multiple deadlines will encourage authors to submit finished work that is easier to evaluate and more likely to be accepted. Authors will not submit early work in the hope of getting a second chance.
6. The Revise option will improve the conference and reduce reviewing randomness by providing authors of papers close to acceptance with an opportunity to address PC concerns directly by providing additional work and experiments.
7. The additional work of evaluating the revisions will not be a significant burden on the PC because it starts from the context of its previous effort and its specific questions and requests.

About the Authors: The ASPLOS Steering Committee currently includes Sarita Adve, R. Iris Bahar, Emery Berger, Luis Ceze, Babak Falsafi, Jeff Foster, Maurice Herlihy, James Larus, Alvin R. Lebeck, Shan Lu, Christos Kozyrakis, Tim Sherwood, Karin Strauss, Emmett Witchel.

Disclaimer: These posts are written by individual contributors to share their thoughts on the Computer Architecture Today blog for the benefit of the community. Any views or opinions represented in this blog are personal, belong solely to the blog author and do not represent those of ACM SIGARCH or its parent organization, ACM.