“If you cannot measure it, you cannot improve it.” – Lord Kelvin
The IEEE International Symposium on Workload Characterization (IISWC) just celebrated its 25th anniversary. This is an excellent opportunity to look back and forward—at what has happened over the past 25 years and what is to come. This blog post shares some of what was said at the IISWC 25th-anniversary panel at the most recent conference, which was held in November 2022 in Austin, TX. The panelists included John Carter (IBM), Lizy K. John (UT Austin), David Kaeli (Northeastern), Vijay Janapa Reddi (Harvard), Carole-Jean Wu (Meta), and Neeraja J. Yadwadkar (UT Austin). I had the honor of moderating the panel.
The quote at the head of the blog post from Lord Kelvin captures the essence of what IISWC is about, namely IISWC covers measurement techniques, characterization methodologies, and workload analyses, which form the foundation for any architecture and systems research and development cycle to optimize computer system performance, power, reliability, security, etc. Without a proper understanding of the workloads, it is simply impossible to design a computer system that is well-optimized and balanced for the workloads it will run. Building and developing representative benchmark suites and methods for workload analysis is hard and takes a lot of time and effort. IISWC provides a forum for researchers in academia as well as practitioners and developers in industry to present and discuss recent advances in workload characterization.
The mission statement of the IISWC is to foster research and application of workload characterization studies and techniques and to provide a forum for presenting and discussing the latest research results and insights related to workload characterization. To quote a former steering committee chair, “IISWC is the only symposium of its kind anywhere. If you are interested at all with what computing workloads look like today, IISWC is where you can get answers.”
IISWC began in 1998 in Dallas, Texas, as the Workshop on Workload Characterization (WWC), which was held at the same time as the IEEE/ACM International Symposium on Microarchitecture (MICRO). Lizy K. John and Ann Marie G. Maynard (IBM) co-founded and co-organized the first editions of WWC for several years until 2004. In 2005, the workshop became a symposium with a steering committee and a set of bylaws to steer the event into the future and create a vibrant community around the conference. The conference was held annually in a variety of places all over the US, from the West Coast (Seattle, Portland, San Jose, San Diego), the South (Austin, Dallas), and the East Coast (Boston, Providence, Raleigh, Atlanta, Orlando), as well as virtually from Beijing, China.
The workloads analyzed over the years at IISWC have changed dramatically, on par with how real-life computer workloads have evolved over the past 2.5 decades. While the initial emphasis was on Java and server workloads, embedded computing, memory analysis and tracing tools (1998–2002), the focus shifted towards commercial, bioinformatics, parallel workloads, and high-performance computing (2003–2007). The introduction of chip multiprocessors shifted the emphasis to multi-core, multi-threaded, and transactional memory workloads; additionally, GPGPU computing, as well as mobile and interactive Internet-scale workloads, began to emerge (2008-2012). The emergence of hyperscale datacenters and the end of Dennard scaling drove the conference’s focus toward cloud computing, big data, and power/energy characterization (2013–2017). The key focal point of interest has recently moved towards GPU and graph analytics workloads, as well as deep neural networks and machine learning workloads (2018–2022). The above word cloud was taken based on paper titles over the past 25 years.
It is humbling to see the conference’s impact over the years. IISWC has come up with several benchmark suites. Some of these have been widely adopted by the community or are expected to have an impact soon. MiBench, an embedded computing benchmark suite published at WWC in 2001, has been cited 4,637 times, according to Google Scholar. Rodinia, a GPGPU benchmark suite published at IISWC in 2009, has been cited 3,320 times, and STAMP, a transactional memory benchmark suite published at IISWC in 2008, has been cited 1,277 times. ILLIXR, a virtual/extended-reality benchmark suite, recently published at IISWC 2021, was selected as an IEEE Micro Top Pick.
IISWC is much more than a conference that publishes benchmark suites. In fact, most of the papers that are published at IISWC can be put into three main groups: methodology, workloads, and systems. Methodology papers cover topics as broad as benchmarking, profiling, tracing, simulation, hardware performance counters, sampling, phase analysis, workload synthesis, etc. Workload papers describe new benchmark suites and analyze, characterize, and explain new application domains in depth. Systems papers investigate how workloads affect systems and how workloads affect systems, not only in terms of performance but also of power, energy, thermals, reliability, and so on. A variety of methodology papers and insightful characterization studies have had a significant impact with over 100+ citation counts.
The future for workload characterization, and IISWC, is bright. Workload analysis and understanding are at the foundation of our field and our research and development cycle. Workloads are constantly changing, and we, as a community, must stay ahead of the curve. At the end of the day, we are designing computer systems that will run our future workloads, not just today’s, let alone yesterday’s, benchmarks. The workloads emerging on the horizon will be inevitably different from today’s workloads, as we are witnessing the emergence of hybrid cloud, edge and Internet-of-Things computing, quantum computing, artificial intelligence, machine learning, deep learning recommendation, domain-specific computing, virtual and augmented reality, heterogeneous and accelerator-rich systems, robotics, health care, microservices and serverless computing, wearables, etc.
In terms of workload characterization methodologies, the use of novel machine learning and data analysis techniques could foster a fresh breadth of characterization and analysis methodologies to be able to see the forest for the trees in the massive amounts of data that current performance counter, profiling, instrumentation, and simulation techniques provide when analyzing ever more complex computer systems. Furthermore, we need cross-cutting technologies to accelerate benchmark innovation and technology for rapidly emerging domains to be ahead of workload trends. We also need robust and reproducible methodologies to collaborate across the community, which has been a challenge for many years and might become even more challenging as machine learning gets involved in the system stack. We should develop methodologies to model and characterize workloads at extreme scale in hyperscale datacenters and across the many mobile and edge devices featuring heterogeneous hardware. Sustainability is a major challenge for our society and generation, and in line with IISWC’s motto “to measure is to know”, we need to develop methodologies, metrics, and characterization techniques to analyze and eventually improve computer system sustainability. Given the breadth of challenges, this calls for a community-driven approach to unify our efforts to collectively accelerate progress and adoption.
In summary, IISWC has served the community well over the past 25 years, and it is expected to continue to do so in the next 25 years and beyond. Workload characterization is and remains a critical foundation for our field. There are major challenges and fantastic opportunities ahead of us for generating impact.
This post was written by Lieven Eeckhout, with input from the IISWC 25th-anniversary panelists.
Disclaimer: These posts are written by individual contributors to share their thoughts on the Computer Architecture Today blog for the benefit of the community. Any views or opinions represented in this blog are personal, belong solely to the blog author and do not represent those of ACM SIGARCH or its parent organization, ACM.
