Tuesday, 12 December 2017

Session 5: Benchmarking

Workload Generation for Microprocessor Performance Evaluation

Authors:

Luk Van Ertvelde (Ghent University)
Lieven Eeckhout (Ghent University)

Abstract:

This PhD thesis [1], awarded with the SPEC Distinguished Dissertation Award 2011, proposes and studies three workload generation and reduction techniques for microprocessor performance evaluation. (1) The thesis proposes code mutation, a novel methodology for hiding proprietary information from computer programs while maintaining representative behavior; code mutation enables dissemination of proprietary applications as benchmarks to third parties in both academia and industry. (2) It contributes to sampled simulation by proposing NSL-BLRL, a novel warm-up technique that reduces simulation time by an order of magnitude over state-of-the-art. (3) It presents a benchmark synthesis framework for generating synthetic benchmarks from a set of desired program statistics. The benchmarks are generated in a high-level programming language, which enables both compiler and hardware exploration.

DOI: 10.1145/2188286.2188313

Full text: PDF

[#][]

Performance Evaluation and Benchmarking of Event-Based Systems

Authors:

Kai Sachs (SAP AG)

Abstract:

Event-based systems (EBS) are increasingly used as underlying technology in many mission critical areas and large-scale environments, such as environmental monitoring and location-based services [3]. Moreover, novel event-based applications are typically highly distributed and data intensive with stringent requirements for performance and scalability. Since their reliability is crucial for the whole IT infrastructure, a certain Quality-of-Service (QoS) level has to be ensured. The motivation for our work was to support the development and maintenance of EBS that meet their QoS requirements. Given that EBS differ from traditional software in fundamental aspects such as their underlying communications paradigm, specific solutions and concepts are needed. System architects and deployers need tools and methodologies, which allow them to evaluate and forecast system performance and behavior in certain situations to identify potential performance problems and bottlenecks. Common approaches are benchmarking and performance modeling. However, no general performance modeling methodologies focusing on EBS had been published. Furthermore, there was a lack of test harnesses and benchmarks using representative workloads for EBS. Consequently, we focused on the development of a performance modeling methodology of EBS as well as on approaches to benchmark them. We summarize now our main contributions and proposed approaches.

DOI: 10.1145/2188286.2188314

Full text: PDF

[#][]

Efficient Update Data Generation for DBMS Benchmarks

Authors:

Michael Frank (University of Passau)
Meikel Poess (Oracle Corporation)
Tilmann Rabl (University of Toronto)

Abstract:

It is without doubt that industry standard benchmarks have been proven to be crucial to the innovation and productivity of the computing industry. They are important to the fair and standardized assessment of performance across different vendors, different system versions from the same vendor and across different architectures. Good benchmarks are even meant to drive industry and technology forward. Since at some point, after all reasonable advances have been made using a particular benchmark even good benchmarks become obsolete over time. This is why standard consortia periodically overhaul their existing benchmarks or develop new benchmarks. An extremely time and resource consuming task in the creation of new benchmarks is the development of benchmark generators, especially because benchmarks tend to become more and more complex. The first version of the Parallel Data Generation Framework (PDGF), a generic data generator, was capable of generating data for the initial load of arbitrary relational schemas. It was, however, not able to generate data for the actual workload, i.e. input data for transactions (insert, delete and update), incremental load etc., mainly because it did not understand the notion of updates. Updates are data changes that occur over time, e.g. a customer changes address, switches job, gets married or has children. Many benchmarks, need to reflect these changes during their workloads. In this paper we present PDGF Version 2, which contains extensions enabling the generation of update data.

DOI: 10.1145/2188286.2188315

Full text: PDF

[#][]

Studying Hardware and Software Trade-Offs for a Real-Life Web 2.0 Workload

Authors:

Stijn Polfliet (Ghent University)
Frederick Ryckbosch (Ghent University)
Lieven Eeckhout (Ghent University)

Abstract:

Designing data centers for Web 2.0 social networking applications is a major challenge because of the large number of users, the large scale of the data centers, the distributed application base, and the cost sensitivity of a data center facility. Optimizing the data center for performance per dollar is far from trivial.

In this paper, we present a case study characterizing and evaluating hardware/software design choices for a real-life Web 2.0 workload. We sample the Web 2.0 workload both in space and in time to obtain a reduced workload that can be replayed, driven by input data captured from a real data center. The reduced workload captures the important services (and their interactions) and allows for evaluating how hardware choices affect end-user experience (as measured by response times).

We consider the Netlog workload, a popular and commercially deployed social networking site with a large user base, and we explore hardware trade-offs in terms of core count, clock frequency, traditional hard disks versus solid-state disks, etc., for the different servers, and we obtain several interesting insights. Further, we present two use cases illustrating how our characterization method can be used for guiding hardware purchasing decisions as well as software optimizations.

DOI: 10.1145/2188286.2188316

Full text: PDF

[#][]

Benchmarking Decentralized Monitoring Mechanisms in Peer-to-Peer Systems

Authors:

Dominik Stingl (TU Darmstadt)
Christian Gross (TU Darmstadt)
Sebastian Kaune (TU Darmstadt)
Ralf Steinmetz (TU Darmstadt)
Karsten Saller (TU Darmstadt)

Abstract:

Decentralized monitoring mechanisms enable obtaining a global view on different attributes and the state of Peer-toPeer systems. Therefore, such mechanisms are essential for managing and optimizing Peer-to-Peer systems. Nonetheless, when deciding on an appropriate mechanism, system designers are faced with a major challenge. Comparing different existing monitoring mechanisms is complex because evaluation methodologies differ widely. To overcome this challenge and to achieve a fair evaluation and comparison, we present a set of dedicated benchmarks for monitoring mechanisms. These benchmarks evaluate relevant functional and non-functional requirements of monitoring mechanisms using appropriate workloads and metrics. We demonstrate the feasibility and expressiveness of our benchmarks by evaluating and comparing three different monitoring mechanisms and highlighting their performance and overhead.

DOI: 10.1145/2188286.2188317

Full text: PDF

[#][]