benchkit

Description

benchkit is a declarative framework for composable performance evaluation of system software. Instead of the ad-hoc shell scripts that performance studies typically accumulate, an experiment is described once as a campaign, and the framework automates the full fetch, build, run, and collect pipeline. Benchmarks, system mechanisms, and profilers are composed as reusable components, so the same experiment description runs reproducibly and unattended across machines and architectures, and produces self-documenting result directories (raw data, the exact commands executed, run metadata, and figures).

benchkit ships integrations for standard benchmarks (SPEC CPU 2017 and SPEC CPU 2026, LevelDB, RocksDB, KyotoCabinet, and microbenchmarks) and makes it straightforward to add custom ones, composable system mechanisms (alternative lock implementations injected via LD_PRELOAD with tilt, user-space scheduling policies with schedkit, and CPU placement with taskset), and profiling and analysis tools (perf stat, perf record, differential flame graphs, and paper-quality plotting). Official SPEC CPU suites are driven non-intrusively: benchkit orchestrates SPEC’s own runcpu/specinvoke harness without modifying or redistributing SPEC.

A complete worked example is provided as a companion reproduction artifact: the runnable experiments and figures that reproduce the ICPE 2026 benchkit paper are packaged in a separate repository (see Additional resources), which received the ICPE 2026 Artifact Available, Functional, and Reusable badges.

Requirements

Core:

• A Linux system (validated on Ubuntu 20.04 to 24.04 and Manjaro)
• Any CPU architecture (validated on x86, armv8, and riscv64)
• Python 3 (validated on 3.10 to 3.12)
• Standard build tools (build-essential, cmake)

Recommended, only for specific benchmarks or features:

• Linux perf, for profiling and flame graphs
• Docker, for container-based experiments
• A SPEC CPU license and ISO image, for SPEC CPU experiments

Additional resources

• Framework source code and documentation: https://github.com/open-s4c/benchkit
• Reproduction artifact for the ICPE 2026 paper, a worked example that reproduces the paper’s experiments and figures (tagged v1.0.0): https://github.com/softwarelanguageslab/icpe26-benchkit-ae

Website

https://benchkit.dev

Contributors

• Antonio Paolillo
• Mats Van Molle
• and others (https://github.com/open-s4c/benchkit/graphs/contributors)

Maintainers

• Antonio Paolillo, antonio.paolillo(at)vub.be (or antonio(at)paolillo.be)
  Vrije Universiteit Brussel
  Brussels, Belgium
  

Version

0.0.2

License

MIT License

Related publications and projects

• Publications:
  • A. Paolillo, M. Van Molle, and K. Hasselmann. "benchkit: A Declarative Framework for Composable Performance Evaluation of System Software". In: Proceedings of the ACM/SPEC International Conference on Performance Engineering (ICPE 2026). DOI: 10.1145/3777884.3796997.
• Companion tools and libraries:
  • tilt: lock interposition via LD_PRELOAD (https://github.com/open-s4c/tilt)
  • schedkit: user-space thread scheduling on Linux (https://github.com/open-s4c/schedkit)
  • libvsync: verified synchronization primitives and concurrent data structures (https://github.com/open-s4c/libvsync)
  • pythainer: composable Docker builders and runners for reproducible research (https://github.com/apaolillo/pythainer)