Thursday, 14 December 2017

Session 1: Keynote Address

IaaS Cloud Benchmarking: Approaches, Challenges, and Experience

Authors:

Alexandru Iosup (Delft University of Technology)

Abstract:

Over the past five years, Infrastructure-as-a-Service (IaaS) clouds have grown into the branch of ICT that offers services related to on-demand lease of storage, computation, and network. One of the major impediments in the selection and even use of (commercial) IaaS clouds is the lack of benchmarking results, that is, the lack of trustworthy quantitative information that allows (potential) cloud users to compare and reason about IaaS clouds.

In this talk we discuss empirical approaches to quantitative evaluation of IaaS clouds, toward cloud benchmarking. Both industry and academia have used empirical approaches for years, but with limited success for IaaS clouds and similar systems (e.g., grids) due to the complexity and size of challenges. We present initial results of our research into cloud workload characterization, including Big Data applications. We present the lessons we have learned in developing the SkyMark framework for cloud performance evaluation and the results of our SkyMark-based investigation of three research questions: What is the performance of production IaaS cloud services? How variable is the performance of widely used production cloud services? and What is the impact on performance of the provisioning and allocation policies that interact with IaaS clouds?

In contrast to previous attempts, our research combines empirical and other approaches, for example modeling and simulation, for gaining more insight; is based on a combination of short-term and multi-year measurements, for ensuring the longevity of our results; and uses large, comprehensive studies of several real clouds, for reducing the threats to validity related to the experimental environment. This presentation can also provide useful insights for fields related to benchmarking, for example experimental evaluation conducted in other large-scale distributed systems.

Last but not least, we present a road-map toward cloud benchmarking and the way we plan to progress on it with other members of the RG Cloud WG of the Standard Performance Evaluation Corporation (SPEC) (http://research.spec.org/working-groups/rg-cloud-working-group.html).

Note: talk based on the homonym article [1].

DOI: 10.1145/2462307.2462309

Full text: PDF

[#][]