Theodolite

Description

Theodolite is a framework for benchmarking the scalability of cloud-native applications, running in Kubernetes. It automates the benchmarking process by deploying the system under test (SUT) to a Kubernetes cluster, generating load on the SUT, and collecting performance metrics during load generation. Theodolite comes as a Kubernetes Operator being installed inside the Kubernetes cluster.

Theodolite allows both designing new benchmarks and running existing ones. Designing a benchmark involves specifying the SUT, a load generator, a dimension along load is increased, a dimension along resources are scaled, and SLOs which are evaluated by Theodolite. All of this is defined in declarative files, which can be written and managed using established Kubernetes tooling. Likewise, running a benchmark only involves defining the experimental setup in declarative files and deploying them to Kubernetes.

Theodolite comes with a set of benchmarks for distributed stream processing engines. These benchmarks represent typical use cases for analyzing Industrial Internet of Things sensor data such as writing measurements to a database or performing different types of aggregations on streaming data. Along with a configurable load generator, Theodolite provides benchmark implementations for the stream processing engines Apache Flink, Kafka Streams, Hazelcast Jet as well as other engines, which are supported by the Apache Beam SDK.

Requirements

Kubernetes ≥ 1.23, Helm ≥ 3.7

Additional resources

Website

https://www.theodolite.rocks

Contributors

Maintainers

  • Sören Henning, soeren.henning(at)email.uni-kiel.de
    Christian-Albrechts-Universität zu Kiel, Institut für Informatik
    24098 Kiel, Germany

Version

0.8.2

License

Apache License 2.0

Related publications and projects

  • S. Henning and W. Hasselbring. "A Configurable Method for Benchmarking Scalability of Cloud-Native Applications". In: Empirical Software Engineering 27. 2022. DOI: 10.1007/s10664-022-10162-1.
  • T. Pfandzelter, S. Henning, T. Schirmer, W. Hasselbring, and D. Bermbach. "Streaming vs. Functions: A Cost Perspective on Cloud Event Processing". In: IEEE International Conference on Cloud Engineering. 2022. DOI: 10.1109/IC2E55432.2022.00015.
  • S. Henning and W. Hasselbring. "Theodolite: Scalability Benchmarking of Distributed Stream Processing Engines in Microservice Architectures". In: Big Data Research 25. 2021. DOI: 10.1016/j.bdr.2021.100209.
  • and others (https://www.theodolite.rocks/publications.html)