Bench 2021

2021 BenchCouncil International Symposium on Benchmarking, Measuring and Optimizing
(Bench’21)
http://www.benchcouncil.org/bench21/index.html

Nov. 14th – Nov. 16th, 2021

==========================================================

Introduction
—————-

Benchmarks, Data, Standards, Measurements, and Optimizations are fundamental human activities and assets. The Bench conference has two essential duties: promote data or benchmark-based quantitative approaches to tackle multidisciplinary and interdisciplinary challenges; connect architecture, system, data management, algorithm, and application communities to better co-design for the inherent workload characterizations.

The Bench conference has three defining characteristics. First, it provides a high-quality, single-track forum for presenting results and discussing ideas that further the knowledge and understanding of the benchmarks, data, standards, measurements, and optimizations community as a whole. Second, it is a multidisciplinary and interdisciplinary conference. The past meetings attracted researchers and practitioners from the architecture, system, algorithm, and application communities. Third, it includes both invited sessions and contributed sessions.

There are three submission opportunities, and the reviewing process is double-blind. Upon acceptance, papers will be scheduled for publication in the BenchCouncil Transactions on Benchmarks, Standards, and Evaluation (TBench https://www.benchcouncil.org/journal.html ) and presentation at the Bench’21 conference.

Regularly, the Bench conference will present the BenchCouncil Achievement Award ($3000), the BenchCouncil Rising Star Award ($1000), and the BenchCouncil Best Paper Award ($1000). To encourage reliable and reproducible research using the benchmarks from all organizations, the Bench conference presents the BenchCouncil Award for Excellence for Reproducible Research to the papers using publicly available benchmarks. Each article receives a $100 prize, maximally up to 12 articles. We welcome the industry leaders as sponsors to provide the testbed and the honorariums for the above awards.

Call for papers
————————

We solicit papers describing original and previously unpublished research. Specific topics of interest include, but are not limited to, the following.
**Benchmark and standard specifications, implementations, and validations of:
-Big Data
-Artificial intelligence (AI)
-High performance computing (HPC)
-Machine learning
-Big scientific data
-Datacenters
-Cloud
-Warehouse-scale computing
-Mobile robotics
-Edge and fog computing
-Internet of Things(IoT)
-Block chain
-Data management and storage
-Financial domains
-Education domains
-Medical domains
-Other application domains

**Data:
-Detailed descriptions of research or industry data sets, including the methods used to collect the data and technical analyses supporting the quality of the measurements.
-Analyses or meta-analyses of existing data and original articles on systems, technologies and techniques that advance data sharing and reuse to support reproducible research.
-Evaluations of the rigour and quality of the experiments used to generate data and the completeness of the descriptions of the data.
-Tools generating large-scale data while preserving their original characteristics.

**Workload characterization, quantitative measurement, design and evaluation studies of:
-Computer and communication networks, protocols and algorithms
-Wireless, mobile, ad-hoc and sensor networks, IoT applications
-Computer architectures, hardware accelerators, multi-core processors, memory systems and storage networks
-HPC
-Operating systems, file systems and databases
-Virtualization, data centers, distributed and cloud computing, fog and edge computing
-Mobile and personal computing systems
-Energy-efficient computing systems
-Real-time and fault-tolerant systems
-Security and privacy of computing and networked systems
-Software systems and services, and enterprise applications
-Social networks, multimedia systems, web services
-Cyber-physical systems, including the smart grid

**Methodologies, abstractions, metrics, algorithms and tools for:
-Analytical modeling techniques and model validation
-Workload characterization and benchmarking
-Performance, scalability, power and reliability analysis
-Sustainability analysis and power management
-System measurement, performance monitoring and forecasting
-Anomaly detection, problem diagnosis and troubleshooting
-Capacity planning, resource allocation, run time management and scheduling
-Experimental design, statistical analysis and simulation

**Measurement and evaluation:
-Evaluation methodologies and metrics
-Testbed methodologies and systems
-Instrumentation, sampling, tracing and profiling of large-scale, real-world applications and systems
-Collection and analysis of measurement data that yield new insights
-Measurement-based modeling (e.g., workloads, scaling behavior, assessment of performance bottlenecks)
-Methods and tools to monitor and visualize measurement and evaluation data
-Systems and algorithms that build on measurement-based findings
-Advances in data collection, analysis and storage (e.g., anonymization, querying, sharing)
-Reappraisal of previous empirical measurements and measurement-based conclusions
-Descriptions of challenges and future directions that the measurement and evaluation community should pursue

**Optimization methodologies and Tools.

Important Dates
————————

There are three submission opportunities over the year.

Spring submission website: https://bench2021.hotcrp.com/
-Abstract registration: May 15, 2021
-Paper submission: May 21, 2021
-First-round author notification: June 21, 2021
-Rebuttal and Revision Period: June 21-July 21, 2021
-Second-round author notification: August 10

Summer submission website: TBD
-Abstract registration: August 1, 2021
-Paper submission: August 7, 2021
-First-round author notification: September 7, 2021
-Rebuttal and Revision Period: September 7-October 7, 2021
-Second-round author notification: November 7, 2021

Winter submission website: TBD
-Abstract registration: December 15, 2021
-Paper submission: December 21, 2021
-First-round author notification: January 21, 2022
-Rebuttal and Revision Period: January 21-February 21, 2022
-Second-round author notification: March 21, 2022

Paper Submission
————————

There are three submission opportunities, and the reviewing process is double-blind. Upon acceptance, papers will be scheduled for publication in the BenchCouncil Transactions on Benchmarks, Standards, and Evaluation (TBench) and presentation at the Bench’21 conference. All accepted and eligible papers will be considered, by a panel of reviewers, for the BenchCouncil Best Paper Award and the BenchCouncil Award for Excellence for Reproducible Research.

Papers must be submitted in PDF. For a full paper, the page limit is 12 double column pages in TBench format (All research article page limits do not include references and author biographies). For a short paper, the page limit is 8 double column pages in TBench format, not including references and author biographies. The submissions will be judged based on the merit of the ideas rather than the length.

At least one author must pre-register for the symposium, and at least one author must attend the symposium to present the paper. Papers for which no author is pre-registered will be removed from the proceedings.

Submission site:
https://bench2021.hotcrp.com/
TBench Latex template:
http://mirrors.ctan.org/macros/latex/contrib/els-cas-templates.zip

Awards
———————

* BenchCouncil Achievement Award ($3,000)
– This award recognizes a senior member who has made long-term contributions to benchmarking, measuring, and optimizing. The winner is eligible for the status of a BenchCouncil Fellow.

* BenchCouncil Rising Star Award ($1,000)
– This award recognizes a junior member who demonstrates outstanding potential for research and practice in benchmarking, measuring, and optimizing.

* BenchCouncil Best Paper Award ($1,000)
– This award recognizes a paper presented at the Bench conferences, which demonstrates potential impact on research and practice in benchmarking, measuring, and optimizing.

* BenchCouncil Award for Excellence for Reproduceable Research (Each winning paper earns a $100 prize, maximally up to 12 papers).
– BenchCouncil incubates and hosts benchmark projects, and further encourages reliable and reproducible research using the benchmarks from BenchCouncil or other organizations. To this end, we present the BenchCouncil Award for Excellence for Reproducible Research to the papers using all publicly available benchmarks.

The Reviewing Process
—————–

The reviewing process for submissions is a hybrid of the traditional conference and journal models. There are three possible outcomes from the first round of submission:

-Accept with Shepherding: a PC member will shepherd every accepted paper to ensure that the reviewers’ essential suggestions are incorporated into the article’s final version. This is similar to the “Minor Revision” outcome at a journal.

-One-shot Revision: This is similar to the “Major Revision” outcome in a journal. In such cases, the authors will receive a list of issues that must be addressed before the paper can be accepted. Authors may then submit a revision of the paper during the rebuttal period. The revision should include an author’s response to the reviewers’ issues as part of the appendix of the article. If this paper’s revision is not submitted within this time, then any resubmission will be treated as a new paper. The outcome after resubmission of a “one-shot revision” will either be “Accept with Shepherding” or “Reject.” The one-shot revision may be rejected, for example, if the reviewers find that the issues they raised were not satisfactorily addressed in the revision.

-Reject: If the paper is rejected, it may not be resubmitted to any Bench deadline within 12 months following the paper’s initial submission.

Organization
—————–

Bench Steering Committees
Prof. Dr. Jack Dongarra, University of Tennessee
Prof. Dr. Geoffrey Fox, Indiana University
Prof. Dr. D. K. Panda, The Ohio State University
Prof. Dr. Felix, Wolf, TU Darmstadt
Prof. Dr. Xiaoyi Lu, University of California, Merced
Dr. Wanling Gao, ICT, Chinese Academy of Sciences & UCAS
Prof. Dr. Jianfeng Zhan, ICT, Chinese Academy of Sciences &BenchCouncil