Saturday, 05 December 2020

Session 7: Keynote Address II

Supporting Operations Personnel through Performance Engineering


Len Bass (NICTA)


Operations costs are around 50% of the worldwide costs of Information Technology. Yet operations problems are mainly viewed as the problem of IT and not of the problem of product developers or performance engineers. I will discuss some areas where performance engineers could be of great assistance to operations and where an understanding of operations process could inform the models created by performance engineers.

Some of these areas are:

  • Upgrades from version N to version N+1 are a common occurrence. Some systems are updated as frequently as 10 times a day. Approximately 10% of these updates fail and one common source of this failure is incorrect configurations. Configuration settings that adversely affect performance are among the most difficult to find and correct. For example, a mistaken configuration setting caused a routing of messages unnecessarily through a different region in the cloud. Inconsistent specification of thread pools between two different libraries used by a product is another example.
  • Operations personnel monitor system wide performance to determine appropriate resource assignments. This monitoring could be much more effective if applications exported relevant information about their particular performance characteristics and needs.
  • Operations processes dictate particular steps when managing applications. Again, considering the upgrade problem, a rolling upgrade causes servers to be systematically taken out of service, upgraded, and restored to service. The workload assumptions of performance models are, typically, not automatically connected to process descriptions. Why could an embedded performance model not be aware of the operations process currently being executed and could adjust the behavior of an application in response.

In general, my argument is that operations personnel are important stakeholders for systems. Performance engineers could make a significant contribution to smoother operations activities and operations processes provide an important source of problems for performance engineers.

DOI: 10.1145/2479871.2479897

Full text: PDF