Wednesday, 13 December 2017

Session 9: Performance Measurement and Experimental Analysis

Automated Detection of Performance Regressions Using Statistical Process Control Techniques

Authors:

Thanh H. D. Nguyen (Queen's University)
Bram Adams (Queen's University)
Zhen Ming Jiang (Queen's University)
Ahmed E. Hassan (Queen's University)
Mohamed Nasser (Research in Motion)
Parminder Flora (Research in Motion)

Abstract:

The goal of performance regression testing is to check for performance regressions in a new version of a software system. Performance regression testing is an important phase in the software development process. Performance regression testing is very time consuming yet there is usually little time assigned for it. A typical test run would output thousands of performance counters. Testers usually have to manually inspect these counters to identify performance regressions. In this paper, we propose an approach to analyze performance counters across test runs using a statistical process control technique called control charts. We evaluate our approach using historical data of a large software team as well as an open-source software project. The results show that our approach can accurately identify performance regressions in both software systems. Feedback from practitioners is very promising due to the simplicity and ease of explanation of the results.

DOI: 10.1145/2188286.2188344

Full text: PDF

[#][]

Capturing Performance Assumptions using Stochastic Performance Logic

Authors:

Lubomír Bulej (Charles University in Prague & Academy of Sciences of the Czech Republic)
Tomásš Burešs (Charles University in Prague & Academy of Sciences of the Czech Republic)
Jaroslav Keznikl (Charles University in Prague & Academy of Sciences of the Czech Republic)
Alena Koubková (Charles University in Prague)
Andrej Podzimek (Charles University in Prague & Academy of Sciences of the Czech Republic)
Petr Tůma (Charles University in Prague)

Abstract:

Compared to functional unit testing, automated performance testing is difficult, partially because correctness criteria are more difficult to express for performance than for functionality. Where existing approaches rely on absolute bounds on the execution time, we aim to express assertions on code performance in relative, hardware-independent terms. To this end, we introduce Stochastic Performance Logic (SPL), which allows making statements about relative method performance. Since SPL interpretation is based on statistical tests applied to performance measurements, it allows (for a special class of formulas) calculating the minimum probability at which a particular SPL formula holds. We prove basic properties of the logic and present an algorithm for SATsolver-guided evaluation of SPL formulas, which allows optimizing the number of performance measurements that need to be made. Finally, we propose integration of SPL formulas with Java code using higher-level performance annotations, for performance testing and documentation purposes.

DOI: 10.1145/2188286.2188345

Full text: PDF

[#][]

Refactoring Access Control Policies for Performance Improvement

Authors:

Donia El Kateb (University of Luxembourg)
Tejeddine Mouelhi (University of Luxembourg)
Yves Le Traon (University of Luxembourg)
JeeHyun Hwang (North Carolina State University)
Tao Xie (North Carolina State University)

Abstract:

In order to facilitate managing authorization, access control architectures are designed to separate the business logic from an access control policy. To determine whether a user can access which resources, a request is formulated from a component, called a Policy Enforcement Point (PEP) located in application code. Given a request, a Policy Decision Point (PDP) evaluates the request against an access control policy and returns its access decision (i.e., permit or deny) to the PEP. With the growth of sensitive information for protection in an application, an access control policy consists of a larger number of rules, which often cause a performance bottleneck. To address this issue, we propose to refactor access control policies for performance improvement by splitting a policy (handled by a single PDP) into its corresponding multiple policies with a smaller number of rules (handled by multiple PDPs). We define seven attribute-set-based splitting criteria to facilitate splitting a policy. We have conducted an evaluation on three subjects of reallife Java systems, each of which interacts with access control policies. Our evaluation results show that (1) our approach preserves the initial architectural model in terms of interaction between the business logic and its corresponding rules in a policy, and (2) our approach enables to substantially reduce request evaluation time for most splitting criteria.

DOI: 10.1145/2188286.2188346

Full text: PDF

[#][]

Hirundo: A Mechanism for Automated Production of Optimized Data Stream Graphs

Authors:

Miyuru Dayarathna (Tokyo Institute of Technology)
Toyotaro Suzumura (Tokyo Institute of Technology and IBM Research - Tokyo)

Abstract:

Stream programs have to be crafted carefully to maximize the performance gain that can be obtained from stream processing environments. Manual fine tuning of a stream program is a very difficult process which requires considerable amount of programmer time and expertise. In this paper we present Hirundo, which is a mechanism for automatically generating optimized stream programs that are tailored for the environment they run. Hirundo analyzes, identifies the structure of a stream program, and transforms it to many different sample programs with same semantics using the notions of Tri-Operator Transformation, Transformer Blocks, and Operator Blocks Fusion. Then it uses empirical optimization information to identify a small subset of generated sample programs that could deliver high performance. It runs the selected sample programs in the run-time environment for a short period of time to obtain their performance information. Hirundo utilizes these information to output a ranked list of optimized stream programs that are tailored for a particular run-time environment. Hirundo has been developed using Python as a prototype application for optimizing SPADE programs, which run on System S stream processing run-time. Using three example real world stream processing applications we demonstrate effectiveness of our approach, and discuss how well it generalizes for automatic stream program performance optimization.

DOI: 10.1145/2188286.2188347

Full text: PDF