Wednesday, 13 December 2017

Session 8: Measurements and Benchmarks - Part 1

RMS-TM: A Comprehensive Benchmark Suite for Transactional Memory Systems

Authors:

Gokcen Kestor (Barcelona Supercomputing Center)
Vasileios Karakostas (Barcelona Supercomputing Center)
Osman S. Unsal (Barcelona Supercomputing Center)
Adrian Cristal (IIIA - Artificial Intelligence Research Institute CSIC - Spanish National Research Council)
Ibrahim Hur (Barcelona Supercomputing Center)
Mateo Valero (Universitat Politécnica de Catalunya)

Abstract:

Transactional Memory (TM) has been proposed as an alternative concurrency mechanism for the shared memory parallel programming model. Its main goal is to make parallel programming for Chip Multiprocessors (CMPs) easier than using the traditional lock synchronization constructs, without compromising the performance and the scalability. This topic has received substantial research attention and several TM designs have been proposed using various TM benchmarks. We believe that the evaluation of TM proposals would be more solid if it included realistic applications, that address on-going TM research issues, and that provide the potential for straightforward comparison against locks.

In this paper, we introduce RMS-TM, a Transactional Memory benchmark suite composed of seven real-world applications from the Recognition, Mining and Synthesis (RMS) domain. In addition to featuring current TM research issues such as nesting and I/O and system calls inside transactions, the RMS-TM applications also provide a mix of short and long transactions with small/large read and write sets with low/medium/high contention rates. These characteristics, as well as providing lock-based versions of the applications, make RMS-TM a useful TM tool. Current TM benchmarks do not explore all these features. In our evaluation with selected STM and HTM systems, we find that our benchmark suite is also scalable, which is useful for evaluating TM designs on high core counts.

DOI: 10.1145/1958746.1958795

Full text: PDF

[#][]

Automatic Estimation of Performance Requirements for Software Tasks of Mobile Devices

Authors:

Simon Schwarzer (University of Bonn)
Patrick Peschlow (University of Bonn)
Lukas Pustina (University of Bonn)
Peter Martini (University of Bonn)

Abstract:

This paper introduces a new method to predict performance requirements of mobile devices’ software tasks using system models describing the hardware and software. With the help of clustering algorithms and linear regression, behavioral models of software tasks are generated automatically. These models are used to project the runtime of representative parts of the software tasks. The runtime of representative execution parts is determined with instruction-accurate simulations which are not feasible for whole executions. The inputs for the projection task a model of the hardware platform and input data parameters, especially the data size. A major advantage of this approach is that the developers do not have to estimate the performance requirements themselves. In this way the method helps to seamlessly integrate the performance analysis process into the development process. The paper introduces the ideas in detail and presents an evaluation of the proposed method for typical software tasks of mobile devices.

DOI: 10.1145/1958746.1958796

Full text: PDF

[#][]

Improving the Efficiency of Information Collection and Analysis in Widely-used IT Applications

Authors:

Sergey Blagodurov (Simon Fraser University)
Martin Arlitt (Hewlett-Packard Laboratories)

Abstract:

Modern IT environments collect and analyze increasingly large volumes of data for a growing number of purposes (e.g., automated management, security, regulatory compliance, etc.). Simultaneously, such environments are challenged by the need to minimize their environmental footprints. A general solution to this problem is to utilize IT resources more efficiently. This paper describes our work to systematically evaluate the inefficiencies in the information collection and analysis of several widely-used IT applications, to implement a more efficient solution, and to quantify the improvements. In particular, the logging of HTTP transactions by the Apache Web server and of network events by the Bro intrusion detection system are converted from text files to DataSeries [24]. The costs of recording, storing and analyzing the information in the different formats are thoroughly evaluated and compared. We converted the text logs to DataSeries online, with no discernable overhead on the logging applications. We achieved upto a 7x decrease in the logfile sizes relative to the sizes of the default text logs, and speedups of 3x-8.4x to analyze the logfiles.

DOI: 10.1145/1958746.1958797

Full text: PDF

[#][]

A Little Language for Rapidly Constructing Automated Performance Tests

Authors:

Shaun Dunning (NetApp, Inc.)
Darren Sawyer (NetApp, Inc.)

Abstract:

In order to e ffectively measure the performance of large scale data management solutions at NetApp, we use a fully automated infrastructure to execute end-to-end system performance tests. Both the software and user requirements of this infrastructure are complex: the system under test runs a multi-protocol, highly specialized operating system and the infrastructure serves a diverse audience of developers, analysts, and eld engineers (including both sales and support). In this paper we describe our approach to rapidly constructing automated performance system tests by using a lightweight, little, or domain-speci c language called SLSL in order to more e ectively express test speci cations.

Using a real world example, we illustrate the efficacy of SLSL in terms of its expressiveness, exibility, and ease of use by showing a complex test con guration expressed with just a few language constructs. We also demonstrate how SLSL can be used in conjunction with our performance measurement lab to quickly deploy performance tests that yield highly repeatable measurements.

DOI: 10.1145/1958746.1958798

Full text: PDF

[#][]