For 2023, ASPLOS embarked on a new multi-deadline review model. Multiple deadlines are meant to encourage authors to submit their papers when ready, to facilitate the selection of some papers for revision and to distribute the reviewer workload better. In this post, we provide an overview of our experience as ASPLOS’23 Program Co-Chairs, offer thoughts on further improvement and discuss long-term opportunities.
ASPLOS’23 featured 3 deadlines: March 31, 2022, July 7, 2022 and October 20, 2022. Papers submitted to each deadline could receive an accept, reject or major revision decision. Papers given a major revision decision would be given a concrete set of changes to make and 6 weeks to make them before being re-reviewed by the same reviewers. They were also given additional space to accommodate revision requests. If the revisions requests were met, the paper would be accepted. Across 3 deadlines, we received 600 submissions. Of those submissions, 151 were accepted–128 appeared in ASPLOS’23 and an additional 23 will appear in ASPLOS’24. The overall acceptance rate was 25%. Of 600 submissions, 90 papers were given a major revision and 69 of those were ultimately accepted (77% acceptance rate). We awarded 11 Distinguished Paper Awards. ACM Award guidelines place an upper limit of 10% of accepted papers that can be designated for an award. We aimed for hitting this upper limit–giving more awards is good for our community and for showcasing the outstanding work of our researchers.
Multiple Deadline and Major Revisions
Multiple deadlines enabled authors to submit their paper when ready. The first deadline received the smallest number of submissions (90)–authors were given fairly short notice about the new early deadline. We anticipate future years will see more even submissions across deadlines as our community acclimates to the process. Multiple deadlines also opened up the opportunity to give papers a major revision decision. The majority of major revision papers were ultimately accepted and reviewers felt that papers were significantly improved through the major revision process; 2 out of 11 distinguished paper awards went to papers that underwent a major revision decision–attesting to the improvement seen by reviewers. An open question remains regarding what is the right threshold for how many major revisions decisions should be given out. Did we give major revisions to papers that we would have accepted outright if major revision was not an option? Although reviewing a revised paper is easier than reviewing a new paper, major revisions do represent an increase in workload for PC members. On the other hand, papers that went through a major revision generally did not need shepherding and may not have needed as much work for the camera-ready version. Finally, we wanted to have all deadlines offer the same opportunity; thus, the Fall deadline was offered major revisions that would appear in ASPLOS’24. This has two drawbacks: 1) there is a long window between when the paper is accepted and when it appears at the conference and 2) it extends the service of PC members beyond the conference itself and lengthens service to more than 1 year.
Reviewer Workload
With each deadline, we expanded our PC to accommodate the higher than anticipated workload. Despite a large PC, PC members reviewed an average of 23 papers which is higher than a typical load of ~15 papers. The higher load was offset by a longer window for the service; those 23 papers were not all reviewed in a 2-3 month period of a typical PC. However, many reviewers found frequent deadlines and the long duration of service weighed on them; ASPLOS service also overlapped with other commitments. 78% of our PC members felt that service on the ASPLOS’23 was either “A little more work” or “Far more work” than previous ASPLOS PCs. To alleviate pressure on the PC, larger PCs can be recruited or PC members can be invited to serve for fewer than 3 deadlines. Additionally, we encourage those accepting an invitation to serve on the ASPLOS PC to think carefully about additional commitments as serving on multiple overlapping PCs is not good for individual researchers, their productivity and their review quality. We have a large and diverse community and need to ensure that the service load is being well-distributed across that community.
Moving forward
To accommodate 128 papers at ASPLOS’23, the program was organized in 3 parallel tracks across 3 days. Each talk had a 16.5 minute time slot – 12 minutes to present followed by 4.5 minutes for questions. The program featured 2 poster sessions–to allow for more meaningful engagement between authors and attendees. We anticipate next year’s ASPLOS program will be larger as we have already accepted 23 papers from Fall 2022 Major Revisions. This leads to questions of how to scale the conference–more tracks, more days, shorter talks, spotlight papers, etc.?
The revision model for ASPLOS also opens the opportunity to publish ASPLOS proceedings as part of the Proceedings of ACM series. Publishing ASPLOS as a journal in this form would benefit colleagues at institutions that still prize journal papers.
We are grateful to the steering committee for their input throughout the process, our program committee members for their dedication to the process especially in the face of a high and unanticipated workload and to Tor Aamodt who as General Chair pulled off an exceptional conference experience for all!
We welcome thoughts from the community on the overall process used in ASPLOS’23; this is a new experiment and there is always room for improvement. We encourage you to comment below with your thoughts. The interested reader may also wish to view the ASPLOS’23 business meeting recording which featured some open discussion of the process. More details can also be found in our 3 program chair messages in the ACM DL: Volume I, Volume 2, and Volume 3.
About the Authors: Natalie Enright Jerger (University of Toronto) and Mike Swift (University of Wisconsin-Madison) served as the ASPLOS’23 Program Co-Chairs.
Disclaimer: These posts are written by individual contributors to share their thoughts on the Computer Architecture Today blog for the benefit of the community. Any views or opinions represented in this blog are personal, belong solely to the blog author and do not represent those of ACM SIGARCH or its parent organization, ACM.
