SPEC RG Member Receives Best Impact Award at SuperComputing'12

Friday, December 14, 2012 08:43

Alexandru Iosup (TU Delft) received the Best Impact Award of the MTAGS workshop at the prestigious ACM/IEEE conference on High Performance Computing and Networking (SC).

The Best Impact Award of the MTAGS workshop at the prestigious ACM/IEEE conference on High Performance Computing and Networking (SC) was received by the SPEC RG member Alexandru Iosup (TU Delft). The award represents the impact of new research work on the research, development, and deployment efforts of large-scale many-task computing (MTC) applications on large scale clusters, Grids, Supercomputers, and Cloud Computing infrastructure. The winner has conducted benchmarking and performance evaluation work on four commercial IaaS clouds. The award was accompanied by a presentation of benchmarking IaaS clouds.

URL: datasys.cs.iit.edu/events/MTAGS12/biggest-impact-award.html

More details about the presentation

Talk title: IaaS Cloud Benchmarking: Approaches, Challenges, and Experience (Slides, Paper)

Abstract: Over the past five years, Infrastructure-as-a-Service clouds have grown into the branch of ICT that offers services related to on-demand lease of storage, computation, and network. One of the major impediments in the selection and even use of (commercial) IaaS clouds is the lack of benchmarking results, that is, the lack of trustworthy quantitative information that allows (potential) cloud users to compare and reason about IaaS clouds.

In this talk we discuss empirical approaches to quantitative evaluation, which we find to be a necessary bumpy road toward cloud benchmarking. Both industry and academia have used empirical approaches for years, but the limited success achieved so far for IaaS clouds and similar systems (e.g., grids) is perhaps indicative of the complexity and size of challenges. We present the lessons we have learned in developing the SkyMark framework for cloud performance evaluation and the results of our SkyMark-based investigation of three research questions: What is the performance of production IaaS cloud services? How variable is the performance of widely used production cloud services? and What is the impact on performance of the user-level middleware, such as the provisioning and allocation policies that interact with IaaS clouds? We discuss the impact of our findings on large-scale, many-task and many-user applications; notably, we discuss not only cloud performance, but also operation and behavior.

In contrast to previous attempts, our research combines empirical and other approaches, for example modeling and simulation, for deeper analysis; is based on a combination of short-term and multi-year measurements for better longevity of results; and uses large, comprehensive studies of several real clouds for an overall broader study. This presentation can also provide useful insights for fields related to benchmarking, for example experimental evaluation conducted in any large-scale distributed system.

Last but not least, we present a roadmap toward cloud benchmarking and the way we plan to progress on it with other members of the RG Cloud Group of the Standard Performance Evaluation Corporation (SPEC) [ research.spec.org/working-groups/rg-cloud-working-group.html ].

Bio: