Problem: The Disappearance of Product Papers from ISCA

Industry research groups in computer architecture (like at IBM, Intel, and NVIDIA) have as much support for architectural exploration and publication as academic groups, but product groups certainly don’t. Few industrial product architects have permission—let alone the time or motivation—to write papers for the International Symposium on Computer Architecture. 

When I started going to the ISCA in the 1970s, 40% of the papers were on real products from industry, including classics that many still read and quote today [BellStrecker76, EmerClark84]. Figure 1 shows that industry papers fell from 40% in the 1970s to 10% recently, and even that 10% includes papers based more on industrial research rather than on industrial products. If these trends continue, the historically strong bond between computer architecture research and practice could fade, making it harder to understand the problems facing industry and for our research to have impact.

When I complained over dinner at ISCA 2019 about the lack of papers on real industrial products, SIGARCH chair Sarita Adve was sitting next to me. She didn’t take the complaint lightly. With the blessing of the executive committees of ACM SIGARCH and IEEE TCCA, Sarita assigned me to help fix the problem for 2020. 

Figure 1. Percentage of industry versus academia affiliation of first authors (from Publication Trends at ISCA, 2019). 

Solution: An ISCA Path Tailored to Real Product Papers

I thought the only hope was a separate submission process with a separate program committee (PC) whose members believed that retrospective papers on industrial products were valuable complements to academic research papers. The PC members also understood that company concerns about patent issues or trade secrets may mean some details are not revealed. The resulting PC was a mix of junior and senior people from industry and academia. The nine of us represent eight universities and six companies. We kept the PC small to facilitate discussion of the papers.

Joel Emer	MIT and NVIDIA	Olivier Temam	DeepMind and Inria
Mark D. Hill	Wisconsin	Caroline Trippel	Stanford
Chris Hughes	Intel	Carole-Jean Wu	Arizona State and Facebook
Sophia Shao	UC Berkeley	Yuan Xie	Alibaba and UCSB

We do not intend the industry track for intern projects where a student spends a summer in industry or a faculty member spends a sabbatical and writes a paper about it; those papers can already appear at ISCA. We decided that the first and virtually all authors of such papers must work in industry. The PC also strongly preferred papers on real products, not prototypes developed in industry research labs. In addition, authors should not anonymize industry papers; reviewers want to know the company and the product being evaluated and whether the authors of the paper are the architects of the product. Fortunately, we had many submissions that matched our requirements. 

Management must approve industry papers before submission (often involving multiple rounds of redaction), and there can be restrictions about filing patents before submitting a paper. We delayed the paper deadline to January 30, two months after the ISCA deadline, to improve the odds of receiving papers.  Besides, reviewing ≈20 papers takes less time than ≈400.

Given the paucity of product papers, we tried to encourage submissions, so in November we reached out to nine companies with the top 10 reasons to publish at ISCA. This recruiting effort led to three of our submissions.

HPCA 2020 also had an industry track, publishing six papers in two sessions. It had a separate PC but its policies differed: there was no paper recruiting, the industry paper deadline was not later, academics could be and were first authors, they allowed intern/sabbatical papers, they used double blind reviews, and its PC was much larger at 21 members.

The ISCA 2020 Industry Track Paper Selection Process

The competition was stiff for the 19 papers we received, with the majority of papers being strong. They were so strong that Sophia Shao and I are guest editing a special issue of IEEE Micro for the papers that we couldn’t accept. PC members who were also on the regular ISCA PC found that the industry papers were very engaging and brought interesting perspectives with different values than the typical ISCA submission. 

We did two rounds of reviews in 3 weeks, adding one or two external expert reviewers to three reviews by PC members in the first round. The final 10 papers of the second round averaged 9 reviews, or 11 papers per PC member. We barely had time to get authors’ responses to these reviews before our meeting on February 20, where we discussed each paper for about 15 minutes. The regular ISCA PC accepted 77/428 or 18% and we accepted 5/19 or 26% (only one was recruited):

• “Data Compression Accelerator on IBM POWER9 and z15 Processors,” IBM
• “High-Performance Deep-Learning Coprocessor Integrated into x86 SoC with Server-Class CPUs,” Centaur
• “The IBM z15 High Frequency Mainframe Branch Predictor,” IBM
• “Evolution of the Samsung Exynos CPU Microarchitecture,” Samsung
• “Xuantie-910: A Commercial Multi-core 12-stage Pipeline Out-of-order 64-bit High Performance RISC-V Processor with Vector Extension,” Alibaba

Each paper proudly bore the label “Industrial Product.” 

It was not our intention a priori that the papers selected focus on micro-architecture, but it was these five papers that stood out from the rest at the end of the process. Interested academics should read them to understand the important issues that arise when deploying designs in the real world, which I hope will lead to new, even more fruitful research directions.

A Quantitative Evaluation of the Industry Track

It delighted us that they appeared in the honored first session of ISCA. The big question was the ISCA audience reaction, which was harder to gauge at a virtual conference. José F. Martínez, the general co-chair of ISCA 2020, sent a survey after ISCA to attendees and received 132 responses. Figure 2 shows the results of the survey questions about the industry track.  

I’m not sure what other questions would get 80% agreement from the traditionally skeptical computer architects —free alcohol and ice cream at the ISCA reception? Given the many changes we tried, the results thrill our industry PC, the ISCA steering committee, and the executive committees. I presume ISCA will have an industry track again at least next year.

Figure 2. Summary of industry track questions from José F. Martínez’s survey of ISCA 2020 attendees (132 responses). 

Reflections and Recommendations

In retrospect, we suspect that few of the selected papers would have survived the traditional ISCA process, as reviewers would expect different content and evaluations. Thus, agreeing with the survey, we believe a separate PC is critical to the success of the industry session. It also felt like having the industry papers in their own session highlighted their distinction, so we agree with the survey and recommend following this precedent in the future. The survey also supports our decision to accept only papers on real products by actual industry authors, as we think that perspective is what made attendees say the ISCA program was stronger and more exciting than ISCAs of the recent past. Thus, we suggest ISCA continues this policy too.  

The executive committees allowed us a maximum of five papers, and we surely would have accepted two or three more if we had their permission. The upper limit in the original industry session proposal to the committees was no more than 10% of the papers, or a maximum of 8 of 85 papers for ISCA 2020. I’d recommend a higher target than five for ISCA 2021, in part because the several authors that didn’t make the cut in 2020 plan to revise and resubmit better even stronger papers next year. I also suspect that we’ll get more new submissions now that more architects are aware of the industry track at ISCA.

The rest of our suggestions are small tweaks from 2020.  We’d recommend an earlier recruiting meeting (September versus November) to give industry authors more time to write, an earlier paper deadline  (January ≈11 versus  January 30) to give the PC more time for reviews, and a little larger PC (≈12 versus 9 members) to cope with the likely more submissions. And we need a new Industry Track Program Chair, as my Google colleagues and I plan to submit a paper to the ISCA 2021 industry track!

Given

• the importance of good ties between industry and research in computer architecture;
• the decline in visibility of industrial products in recent conferences; and 
• the strong positive reaction from both authors and the audience to this approach; 

perhaps other conferences should follow this path to industrial product papers?

About the Author: David Patterson is a Google distinguished engineer,  a UC Berkeley Professor, the RISC-V International Vice Chair, and the RISC-V International Open Source (RIOS) Laboratory Director. His newest book is The RISC-V Reader: An Open Architecture Atlas and his best known is Computer Architecture: A Quantitative Approach. He and his co-author John Hennessy shared the 2017 ACM A.M. Turing Award.

Disclaimer: These posts are written by individual contributors to share their thoughts on the Computer Architecture Today blog for the benefit of the community. Any views or opinions represented in this blog are personal, belong solely to the blog author and do not represent those of ACM SIGARCH or its parent organization, ACM.