Highlighted Publications

Selected joint member publications as well as technical reports published by SPEC RG are available at the publication page.

Group Publications

In the following, we list a selection of relevant publications by members of the RG IDS Benchmarking working group.

Flora, J., Gonçalves, P., Teixeira, M., & Antunes, N. (2021, October). My Services Got Old! Can Kubernetes Handle the Aging of Microservices?. In 2021 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW) (pp. 40-47). IEEE.
[ bibtex | abstract ]
Keywords: aging; kubernetes; fault injection; microservices.

The exploding popularity of microservice based applications is taking companies to adopt them along with cloud services to support them. Containers are the common deployment infrastructures that currently serve millions of customers daily, being managed using orchestration platforms that monitor, manage, and automate most of the work. However, there are multiple concerns with the claims put forward by the developers of such tools. In this paper, we study the effects of aging in microservices and the utilization of faults to accelerate aging effects while evaluating the capacity of Kubernetes to detect microservice aging. We consider three operation scenarios for a representative microservice-based system through the utilization of stress testing and fault injection as a manner to potentiate aging in the services composing the system to evaluate the capacity of Kubernetes mechanisms to detect it. The results demonstrate that even though some services tend to accumulate aging effects, with increasing resource consumption, Kubernetes does not detect them nor acts on them, which indicates that the probe mechanisms may be insufficient for aging scenarios. This factor may indicate the necessity for more effective mechanisms, capable of detecting aging early on and act on it in a more proactive manner without requiring the services to become unresponsive.
@inproceedings{flora2021my,
  title={My Services Got Old! Can Kubernetes Handle the Aging of Microservices?},
  author={Flora, Jos{\'e} and Gon{\c{c}}alves, Paulo and Teixeira, Miguel and Antunes, Nuno},
  booktitle={2021 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW)},
  pages={40--47},
  year={2021},
  organization={IEEE}
}
Lukas Bierlieb, Lukas Iffläender, Aleksandar Milenkoski, Alberto Avritzer, Nuno Antunes and Samuel Kounev. Software Testing Strategies for Detecting Hypercall Handlers' Aging-related Bugs. In Proceedings of the 13th International Workshop on Software Aging and Rejuvenation (WOSAR 2021).
[ bibtex | abstract ]
Keywords: software testing; virtual machine monitors; computer bugs; reverse engineering; manuals, documentation, aging.

With the continuing rise of cloud technology hypervisors play a vital role in the performance and reliability of current services. As long-running applications, they are susceptible to software aging. Hypervisors offer so-called hypercall interfaces for communication with the hosted virtual machines. These interfaces require thorough testing to ensure their long-term reliability. Existing research deals with the aging properties of hypervisors in general without considering the hypercalls. In this work, we share our experience that we collected during trying to understand hypercalls and their parameters and use them to construct test cases for hypervisor aging of Microsoft Hyper-V. We present a bug that we detected, which was reported and acknowledged by Microsoft. Further, based on our manual binary code analysis, we propose the idea of automating the analysis process to detect valid parameter ranges and execution conditions of hypercalls without manual effort.
@inproceedings{DBLP:conf/issre/BeierliebAIAMK21,
  author    = {Lukas Beierlieb and
               Alberto Avritzer and
               Lukas Iffl{\"{a}}nder and
               Nuno Antunes and
               Aleksandar Milenkoski and
               Samuel Kounev},
  title     = {Software Testing Strategies for Detecting Hypercall Handlers' Aging-related
               Bugs},
  booktitle = {{IEEE} International Symposium on Software Reliability Engineering,
               {ISSRE} 2021 - Workshops, Wuhan, China, October 25-28, 2021},
  pages     = {48--55},
  publisher = {{IEEE}},
  year      = {2021},
  url       = {https://doi.org/10.1109/ISSREW53611.2021.00043},
  doi       = {10.1109/ISSREW53611.2021.00043},
}
Gonçalves, C. F., & Antunes, N. (2020, October). Vulnerability Analysis as Trustworthiness Evidence in Security Benchmarking: A Case Study on Xen. In 2020 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW) (pp. 231-236). IEEE.
[ bibtex | abstract ]
Keywords: security benchmark; vulnerabilities; hypervisor; virtualization.

Hypervisors govern the resources of virtualized systems and are a crucial component of many cloud solutions. As a critical component, cloud providers should assess the hypervisor’s security to mitigate risk before adoption. Ideally, a benchmark should be applied to compare the security of different systems objectively, but security benchmarking is still an open problem. Notwithstanding, the evaluation of the system’s trustworthiness has been adopted as a promising approach as part of this complex evaluation process. In this work, we present a vulnerability data analysis of the Xen hypervisor. Additionally, we address the problem of how to apply this analysis results as trustworthiness evidence that can be applied in security benchmarks. Our results present an insightful characterization of Xen’s vulnerabilities evaluating their lifespan, distribution, and modeling. We also show that vulnerability data analysis can qualitatively characterize the Xen hypervisor’s trustworthiness and possibly reflect the security development efforts into its codebase.
@inproceedings{gonccalves2020vulnerability,
  title={Vulnerability Analysis as Trustworthiness Evidence in Security Benchmarking: A Case Study on Xen.},
  author={Gon{\c{c}}alves, Charles F and Antunes, Nuno},
  booktitle={2020 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW)},
  pages={231--236},
  year={2020},
  organization={IEEE}
}
Lukas Beierlieb, Lukas Iffländer, Samuel Kounev, and Aleksandar Milenkoski. Towards Testing the Performance Influence of Hypervisor Hypercall Interface Behavior. In Proceedings of the 10th Symposium on Software Performance 2019 (SSP’19).
[ bibtex | abstract ]
Keywords: hypervisors; hypercalls; testing; performance.

With the continuing rise of cloud technology hypervisors play a vital role in the performance and reliability of current services. Hypervisors offer so-called hypercall interfaces for communication with the hosted virtual machines. These interfaces require thorough robustness to assure performance, security, and reliability. Existing research focusses on finding hypercall-related vulnerabilities. In this work, we discuss open challenges regarding hypercall interfaces. To address these challenges, we propose an extensive framework architecture to perform robustness testing on hypercall interfaces. This framework supports test campaigns and modeling of hypercall interfaces.
@inproceedings{Beierlieb\_2019\_TowardsTestingthePerformanceInfluenceofHypervisorHypercallInterfaceBehavior,  author = {Beierlieb, Lukas and Iffländer, Lukas and Kounev, Samuel and Milenkoski, Aleksandar},
  booktitle = {Proceedings of the 10th Symposium on Software Performance 2019 (SSP'19)},
  month = 11,
  title = {Towards Testing the Performance Influence of Hypervisor Hypercall Interface Behavior},
  year = 2019
}
Lukas Beierlieb, Lukas Iffländer, Aleksandar Milenkoski, Charles F. Goncalves, Nuno Antunes, and Samuel Kounev. Towards Testing the Software Aging Behavior of Hypervisor Hypercall Interfaces. In 2019 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW).
[ bibtex | abstract ]
Keywords: aging; hypervisors; hypercalls; testing; challenges; dependability; robustness; security.

With the continuing rise of cloud technology hypervisors play a vital role in the performance and reliability of current services. As long-running applications, they are susceptible to software aging. Hypervisors offer so-called hypercall interfaces for communication with the hosted virtual machines. These interfaces require thorough robustness to assure performance, security, and reliability. Existing research either deals with the aging properties of hypervisors in general without considering the hypercalls or focuses on finding hypercall-related vulnerabilities. In this work, we discuss open challenges regarding hypercall interfaces. To address these challenges, we propose an extensive framework architecture to perform robustness testing on hypercall interfaces. This framework supports extensive test campaigns as well as the modeling of hypercall interfaces.
@inproceedings{Beierlieb\_2019\_TowardsTestingtheSoftwareAgingBehaviorofHypervisorHypercallInterfaces,
  author = {Beierlieb, Lukas and Iffländer, Lukas and Milenkoski, Aleksandar and Goncalves, Charles F. and Antunes, Nuno and Kounev, Samuel},
  booktitle = {2019 {IEEE} International Symposium on Software Reliability Engineering Workshops ({ISSREW})},
  month = 11,
  organization = {{IEEE}},
  title = {Towards Testing the Software Aging Behavior of Hypervisor Hypercall Interfaces},
  year = 2019
}
Lukas Iffländer, Jonathan Stoll, Nishant Rawtani, Veronika Lesch, Klaus-Dieter Lange, and Samuel Kounev. Performance Oriented Dynamic Bypassing for Intrusion Detection Systems (Short Paper). In Proceedings of the 2019 ACM/SPEC International Conference on Performance Engineering, Mumbai, India, 2019, ICPE '19, pages 159--166. ACM, New York, NY, USA. 2019.
[ bibtex | abstract ]
Keywords: intrusion detection, software-deined networking; network function virtualization; adaptive networking.

Attacks on software systems are becoming more and more frequent, aggressive and sophisticated. With the changing threat landscape, in 2018, organizations are looking at when they will be attacked, not if. Intrusion Detection Systems (IDSs) can help in defending against these attacks. The systems that host IDSs require extensive computing resources as IDSs tend to detect attacks under overloaded conditions wrongfully. With the end of Moore's law and the growing adoption of Internet of Things, designers of security systems can no longer expect processing power to keep up the pace with them. This limitation requires ways to increase the performance of these systems without adding additional compute power. In this work, we present two dynamic and a static approach to bypass IDS for traffic deemed benign. We provide its prototype implementation and evaluate our solution. Our evaluation shows promising results. Performance is increased up to the level of a system without an IDS. Attack detection is within the margin of error from the 100% rate. However, our findings show that dynamic approaches perform best when using software switches. The use of a hardware switch reduces the detection rate and performance significantly.

@inproceedings{Ifflaender2019_PerformanceOrientedDynamicBypassingforIntrusionDetectionSystems,
  author = {Iffl{\"a}nder, Lukas and Stoll, Jonathan and Rawtani, Nishant and Lesch, Veronika and Lange, Klaus-Dieter and Kounev, Samuel},
  title = {{Performance Oriented Dynamic Bypassing for Intrusion Detection Systems}},
  titleaddon = {{(Short Paper)}},
  booktitle = {Proceedings of the 2019 ACM/SPEC International Conference on Performance Engineering},
  series = {ICPE '19},
  year = {2019},
  isbn = {978-1-4503-6239-9},
  location = {Mumbai, India},
  pages = {159--166},
  numpages = {8},
  url = {http://doi.acm.org/10.1145/3297663.3310313},
  doi = {10.1145/3297663.3310313},
  acmid = {3310313},
  publisher = {ACM},
  address = {New York, NY, USA},
  keywords = {adaptive networking, intrusion detection, network function virtualization, software-defined networking},
  pdf = {https://se2.informatik.uni-wuerzburg.de/publications/download/paper/1846.pdf},
  slides = {https://se2.informatik.uni-wuerzburg.de/publications/download/slides/1846},
}
Lukas Iffländer, Jürgen Walter, Simon Eismann, and Samuel Kounev. The vision of self-aware reordering of security network function chains (Vision Paper). In Proceedings of the 2018 ACM/SPEC International Conference on Performance Engineering, Berlin, Germany, 2018, ICPE '18, pages 1--4. ACM, New York, NY, USA. 2018.
[ bibtex | abstract ]
Keywords: network security, software-deined networking; network function virtualization; adaptive networking.

Services provided online are subject to various types of attacks. Security appliances can be chained to protect a system against multiple types of network attacks. The sequence of appliances has a significant impact on the efficiency of the whole chain. While the operation of security appliance chains is currently based on a static order, traffic-aware reordering of security appliances may significantly improve efficiency and accuracy. In this paper, we present the vision of a self-aware system to automatically reorder security appliances according to incoming traffic. To achieve this, we propose to apply a model-based learning, reasoning, and acting (LRA-M) loop. To this end, we describe a corresponding system architecture and explain its building blocks.

@inproceedings{IfWaEiKo2018-ICPE-SSFC-Vision,
  author = {Iffl{\"a}nder, Lukas and Walter, J{\"u}rgen and Eismann, Simon and Kounev, Samuel},
  title = {The Vision of Self-aware Reordering of Security Network Function Chains},
  booktitle = {Proceedings of the 2018 ACM/SPEC International Conference on Performance Engineering},
  series = {ICPE '18},
  year = {2018},
  isbn = {978-1-4503-5629-9},
  location = {Berlin, Germany},
  pages = {1--4},
  numpages = {4},
  doi = {10.1145/3185768.3186309},
  acmid = {3186309},
  publisher = {ACM},
  address = {New York, NY, USA},
  keywords = {models at run-time, network function virtualization, service function chaining, software-defined networking},
  pdf = {https://se2.informatik.uni-wuerzburg.de/publications/download/paper/1564.pdf},
  titleaddon = {{(Vision Paper)}},
  slides = {https://se2.informatik.uni-wuerzburg.de/publications/download/slides/1564},
}
Aleksandar Milenkoski, Bryan D. Payne, Nuno Antunes, Marco Vieira, Samuel Kounev, Alberto Avritzer, and Matthias Luft, "Evaluation of Intrusion Detection Systems in Virtualized Environments Using Attack Injection", In Proceedings of The 18th International Symposium on Research in Attacks, Intrusions, and Defenses (RAID 2015), 2015
[ bibtex | abstract ]
Keywords: Cloud computing; Virtualization; Hypercalls; Intrusion detection systems; Workloads; Attack injection.

The evaluation of intrusion detection systems (IDSes) is an active research area with many open challenges, one of which is the generation of representative workloads that contain attacks. In this paper, we propose a novel approach for the rigorous evaluation of IDSes in virtualized environments, with a focus on IDSes designed to detect attacks leveraging or targeting the hypervisor via its hypercall interface. We present hInjector, a tool for generating IDS evaluation workloads by injecting such attacks during regular operation of a virtualized environment. We demonstrate the application of our approach and show its practical usefulness by evaluating a representative IDS designed to operate in virtualized environments. The virtualized environment of the industry-standard benchmark SPECvirt_sc2013 is used as a testbed, whose drivers generate workloads representative of workloads seen in production environments. This work enables for the first time the injection of attacks in virtualized environments for the purpose of generating representative IDS evaluation workloads.

@inproceedings{MiPaAnViKoAvLu2015-RAID-Challenges,
author = {Aleksandar Milenkoski and Bryan D. Payne and Nuno Antunes and Marco Vieira and Samuel Kounev and Alberto Avritzer and Matthias Luft},
booktitle = {The 18th International Symposium on Research in Attacks, Intrusions, and Defenses (RAID 2015)},
publisher = {{Springer}},
location = {Kyoto, Japan},
title = {{Evaluation of Intrusion Detection Systems in Virtualized Environments Using Attack Injection}},
year = {2015},
month = {{November}},
url = {http://link.springer.com/chapter/10.1007/978-3-319-26362-5_22},
}
Aleksandar Milenkoski, Marco Vieira, Samuel Kounev, Alberto Avrtizer, and Bryan D. Payne, "Evaluating Computer Intrusion Detection Systems: A Survey of Common Practices", ACM Computing Surveys, 48(1):12:1-12:41, September 2015, ACM, New York, NY, USA. 5-year Impact Factor (2014): 5.949.
[ bibtex | abstract ]
Keywords: Survey; Intrusion detection systems; Evaluation; Experimentation.

The evaluation of computer intrusion detection systems (which we refer to as intrusion detection systems) is an active research area. In this paper, we survey and systematize common practices in the area of evaluation of intrusion detection systems. For this purpose, we define a design space structured into three parts: workload, metrics, and measurement methodology. We then provide an overview of the common practices in evaluation of intrusion detection systems by surveying evaluation approaches and methods related to each part of the design space. Finally, we discuss open issues and challenges focusing on evaluation methodologies for novel intrusion detection systems.
@article{MiViKoAvPa2015-CSUR-IDSEval,
author = {Aleksandar Milenkoski and Marco Vieira and Samuel Kounev and Alberto Avrtizer and Bryan D. Payne},
title = {{Evaluating Computer Intrusion Detection Systems: A Survey of Common Practices}},
journal = {{ACM Computing Surveys}},
publisher = {ACM},
address = {New York, NY, USA},
year = {2015},
month = {{September}},
volume = {48},
number = {1},
pages = {12:1--12:41},
url = {http://dl.acm.org/authorize?N06203},
}
Aleksandar Milenkoski, Bryan D. Payne, Nuno Antunes, Marco Vieira, and Samuel Kounev, "Experience Report: An Analysis of Hypercall Handler Vulnerabilities", In Proceedings of The 25th IEEE International Symposium on Software Reliability Engineering (ISSRE 2014) - Research Track, 2014
[ bibtex | abstract ]
Keywords: Cloud computing; Virtualization; Hypercalls; Vulnerability analysis.

Hypervisors are becoming increasingly ubiquitous with the growing proliferation of virtualized data centers. As a result, attackers are exploring vectors to attack hypervisors, against which an attack may be executed via several attack vectors such as device drivers, virtual machine exit events, or hypercalls. Hypercalls enable intrusions in hypervisors through their hypercall interfaces. Despite the importance, there is very limited publicly available information on vulnerabilities of hypercall handlers and attacks triggering them, which significantly hinders advances towards monitoring and securing these interfaces. In this paper, we characterize the hypercall attack surface based on analyzing a set of vulnerabilities of hypercall handlers. We systematize and discuss the errors that caused the considered vulnerabilities, and activities for executing attacks triggering them. We also demonstrate attacks triggering the considered vulnerabilities and analyze their effects. Finally, we suggest an action plan for improving the security of hypercall interfaces.
@inproceedings{MiPaAnViKo2014-ISSRE-AnAnalHypHanVulns,
author = {Aleksandar Milenkoski and Bryan D. Payne and Nuno Antunes and Marco Vieira and Samuel Kounev},
title = {{Experience Report: An Analysis of Hypercall Handler Vulnerabilities}},
location = {Naples, Italy},
month = {November},
booktitle = {{Proceedings of The 25th IEEE International Symposium on Software Reliability Engineering (ISSRE 2014) --- Research Track}},
organization = {IEEE},
publisher = {IEEE Computer Society},
address = {{Washington DC, USA}},
year = {2014},
}
Aleksandar Milenkoski, Bryan D. Payne, Nuno Antunes, Marco Vieira, and Samuel Kounev, "HInjector: Injecting Hypercall Attacks for Evaluating VMI-based Intrusion Detection Systems", In The 2013 Annual Computer Security Applications Conference (ACSAC 2013) (Poster paper), 2013
[ bibtex ]
Keywords: Cloud computing; Virtualization; Intrusion detection system benchmarking.

@inproceedings{MiPaAnViKo2013-ACSAC-HInjector,
  address = {Maryland, USA},
  author = {Aleksandar Milenkoski and Bryan D. Payne and Nuno Antunes and Marco Vieira and Samuel Kounev},
  booktitle = {The 2013 Annual Computer Security Applications Conference (ACSAC 2013)},
  publisher = {{Applied Computer Security Associates (ACSA)}},
  location = {New Orleans, Louisiana, USA},
  title = {{HInjector: Injecting Hypercall Attacks for Evaluating VMI-based Intrusion Detection Systems (Poster Paper)}},
  year = {2013}
}

Jose Fonseca, Marco Vieira, and Henrique Madeira, "Vulnerability & Attack Injection for Web Applications", in 39th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN 2009), pp. 93-102, 2009
[ bibtex | abstract ]
Keywords: Intrusion Detection; Performance Evaluation; Probes; Testing.

In this paper we propose a methodology to inject realistic attacks in Web applications. The methodology is based on the idea that by injecting realistic vulnerabilities in a Web application and attacking them automatically we can assess existing security mechanisms. To provide true to life results, this methodology relies on field studies of a large number of vulnerabilities in Web applications. The paper also describes a set of tools implementing the proposed methodology. They allow the automation of the entire process, including gathering results and analysis. We used these tools to conduct a set of experiments to demonstrate the feasibility and effectiveness of the proposed methodology. The experiments include the evaluation of coverage and false positives of an intrusion detection system for SQL injection and the assessment of the effectiveness of two Web application vulnerability scanners. Results show that the injection of vulnerabilities and attacks is an effective way to evaluate security mechanisms and tools.
@INPROCEEDINGS{5270349,
	author={Fonseca, J. and Vieira, M. and Madeira, H.},
	booktitle={IEEE/IFIP International Conference on Dependable Systems Networks, 2009. DSN '09.}, 
	title={Vulnerability #x00026; attack injection for web applications},
	year={2009},
	month={29 2009-july 2},
	pages={93 -102},
	keywords={Application software;Automation;Counting circuits;Information security;Inspection;Instruments;Intrusion detection;Performance evaluation;Probes;Testing;Internet;SQL;program debugging;program testing;security of data;SQL injection;Web application;Web application vulnerability scanner;intrusion detection system;realistic attack injection tool;realistic test bed;realistic vulnerability injection;security mechanism;software bug;},
	doi={10.1109/DSN.2009.5270349},
}
						
Ivano A. Elia, Jose Fonseca, and Marco Vieira, "Comparing SQL Injection Detection Tools Using Attack Injection: An Experimental Study", in 21st Annual International Symposium on Software Reliability Engineering, pp. 289-298, 2010
[ bibtex | abstract ]
Keywords: Databases; Intrusion Detection; Monitoring; SQL Injection Detection.

System administrators frequently rely on intrusion detection tools to protect their systems against SQL Injection, one of the most dangerous security threats in database-centric web applications. However, the real effectiveness of those tools is usually unknown, which may lead administrators to put an unjustifiable level of trust in the tools they use. In this paper we present an experimental evaluation of the effectiveness of five SQL Injection detection tools that operate at different system levels: Application, Database and Network. To test the tools in a realistic scenario, Vulnerability and Attack Injection is applied in a setup based on three web applications of different sizes and complexities. Results show that the assessed tools have a very low effectiveness and only perform well under specific circumstances, which highlight the limitations of current intrusion detection tools in detecting SQL Injection attacks. Based on experimental observations we underline the strengths and weaknesses of the tools assessed.
@INPROCEEDINGS{5635053,
	author={Elia, I.A. and Fonseca, J. and Vieira, M.},
	booktitle={Software Reliability Engineering (ISSRE), 2010 IEEE 21st International Symposium on}, 
	title={Comparing SQL Injection Detection Tools Using Attack Injection: An Experimental Study},
	year={2010},
	month={nov.},
	pages={289 -298},
	keywords={Databases;Intrusion detection;Monitoring;Payloads;Scalp;Web server;Internet;SQL;security of data;SQL injection detection tools;attack injection;database-centric Web application;intrusion detection tool;security threat;system administration;vulnerability;Fault Injection;Intrusion Detection;SQL Injection;Security;Web applications;},
	doi={10.1109/ISSRE.2010.32},
	ISSN={1071-9458},
}
						
Nuno Antunes and Marco Vieira, "Benchmarking Vulnerability Detection Tools for Web Services", in IEEE International Conference on Web Services (ICWS 2010), pp. 203-210, 2010
[ bibtex | abstract ]
Keywords: Vulnerability Benchmarking; Web Services.

Vulnerability detection tools are frequently considered the silver-bullet for detecting vulnerabilities in web services. However, research shows that the effectiveness of most of those tools is very low and that using the wrong tool may lead to the deployment of services with undetected vulnerabilities. In this paper we propose a benchmarking approach to assess and compare the effectiveness of vulnerability detection tools in web services environments. This approach was used to define a concrete benchmark for SQL Injection vulnerability detection tools. This benchmark is demonstrated by a real example of benchmarking several widely used tools, including four penetration-testers, three static code analyzers, and one anomaly detector. Results show that the benchmark accurately portrays the effectiveness of vulnerability detection tools and suggest that the proposed approach can be applied in the field.
@article{10.1109/ICWS.2010.76,
	author = {Nuno Antunes and Marco Vieira},
	title = {Benchmarking Vulnerability Detection Tools for Web Services},
	journal ={2012 IEEE 19th International Conference on Web Services},
	volume = {0},
	isbn = {978-0-7695-4128-0},
	year = {2010},
	pages = {203-210},
	doi = {http://doi.ieeecomputersociety.org/10.1109/ICWS.2010.76},
	publisher = {IEEE Computer Society},
	address = {Los Alamitos, CA, USA},
}
						
Alberto Avritzer, Andre Bondi, and Elaine Weyuker, "Ensuring stable performance for systems that degrade", in Proceedings of the 5th international workshop on software and performance, pp. 43-51, 2005
[ bibtex | abstract ]

A new approach that is useful in identifying and eliminating performance degradation occurring in aging software is proposed. A customer-affecting metric is used to initiate the restoration of such a system to full capacity. A case study is described in which, by simulating an industrial software system, we are able to show that by monitoring a customer-affecting metric and frequently comparing its degradation to the performance objective, we can ensure system stability at a very low cost.
@inproceedings{Avritzer:2005:ESP:1071021.1071026,
 author = {Avritzer, Alberto and Bondi, Andre and Weyuker, Elaine J.},
 title = {Ensuring stable performance for systems that degrade},
 booktitle = {Proceedings of the 5th international workshop on Software and performance},
 series = {WOSP '05},
 year = {2005},
 isbn = {1-59593-087-6},
 location = {Palma, Illes Balears, Spain},
 pages = {43--51},
 numpages = {9},
 url = {http://doi.acm.org/10.1145/1071021.1071026},
 doi = {10.1145/1071021.1071026},
 acmid = {1071026},
 publisher = {ACM},
 address = {New York, NY, USA},
}
						
Alberto Avritzer, Rajanikanth Tanikella, Kiran James, Robert G. Cole, and Elaine Weyuker, "Monitoring for security intrusion using performance signatures", in Proceedings of the first joint WOSP/SIPEW international conference on Performance engineering, pp. 93-104, 2010
[ bibtex | abstract ]
Keywords: Measurement; Monitoring; Performance signatures; Security.

A new approach for detecting security attacks on software systems by monitoring the software system performance signatures is introduced. We present a proposed architecture for security intrusion detection using off-the-shelf security monitoring tools and performance signatures. Our approach relies on the assumption that the performance signature of the well-behaved system can be measured and that the performance signature of several types of attacks can be identified. This assumption has been validated for operations support systems that are used to monitor large infrastructures and receive aggregated traffic that is periodic in nature. Examples of such infrastructures include telecommunications systems, transportation systems and power generation systems. In addition, significant deviation from well-behaved system performance signatures can be used to trigger alerts about new types of security attacks. We used a custom performance benchmark and five types of security attacks to derive performance signatures for the normal mode of operation and the security attack mode of operation. We observed that one of the types of the security attacks went undetected by the off-the-shelf security monitoring tools but was detected by our approach of monitoring performance signatures. We conclude that an architecture for security intrusion detection can be effectively complemented by monitoring of performance signatures.
                @inproceedings{Avritzer:2010:MSI:1712605.1712623,
                author = {Avritzer, Alberto and Tanikella, Rajanikanth and James, Kiran and Cole, Robert G. and Weyuker, Elaine},
                title = {Monitoring for security intrusion using performance signatures},
                booktitle = {Proceedings of the first joint WOSP/SIPEW international conference on Performance engineering},
                series = {WOSP/SIPEW '10},
                year = {2010},
                isbn = {978-1-60558-563-5},
                location = {San Jose, California, USA},
                pages = {93--104},
                numpages = {12},
                url = {http://doi.acm.org/10.1145/1712605.1712623},
                doi = {10.1145/1712605.1712623},
                acmid = {1712623},
                publisher = {ACM},
                address = {New York, NY, USA},
                keywords = {measurement, monitoring, performance signatures, security},
                }
        
Katinka Wolter, Alberto Avritzer, Marco Vieira, and Aad van Moorsel (Eds.), "Resilience Assessment and Evaluation of Computing Systems", XVIII, 489 p. 95 illus, ISBN: 978-3-642-29031-2, Springer, 2012
Keywords: Benchmarking; Dependability; Networks; Performance; Reliability; Robustness; Statistical methods; Stochastic modeling; System evaluation; Testing.

Aleksandar Milenkoski and Samuel Kounev, "Towards benchmarking intrusion detection systems for virtualized cloud environments", in Proceedings of the 2012 International Conference For Internet Technology And Secured Transactions (Work-in-progress paper), pp. 562-563, 2012
[ bibtex | abstract ]
Keywords: Cloud computing; Security of data; Virtualisation; Intrusion detection system benchmarking.

Many recent research works propose novel architectures of intrusion detection systems specifically designed to operate in virtualized environments. However, little attention has been given to the evaluation and benchmarking of such architectures with respect to their performance and dependability. In this paper, we present a research roadmap towards developing a framework for benchmarking intrusion detection systems for cloud environments in a scientifically rigorous and a representative manner.
                @INPROCEEDINGS{6470873,
                author={Milenkoski, A. and Kounev, S.},
                booktitle={2012 International Conference For Internet Technology And Secured Transactions}, title={Towards benchmarking intrusion detection systems for virtualized cloud environments},
                year={2012},
                pages={562-563},
                keywords={cloud computing;security of data;virtualization;intrusion detection system benchmarking;virtualized cloud environments;Benchmark testing;Cloud computing;Hardware;Intrusion detection;Measurement;Monitoring;Virtual machine monitors;benchmark testing;intrusion detection},}
        
Alberto Avritzer, Robert G. Cole, and Elaine Weyuker, "Using performance signatures and software rejuvenation for worm mitigation in tactical MANETs", in Proceedings of the 6th international workshop on Software and performance (WOSP '07), pp. 172-180, 2007
[ bibtex | abstract ]
Keywords: Network monitoring; Protection Mechanisms; Performance signatures; Computer worms; Mitigation.

In this paper, we propose a new approach for mitigation of worm propagation through tactical Mobile Ad-Hoc Networks (MANETs) which is based upon performance signatures and software rejuvenation. Three application performance signature and software rejuvenation algorithms are proposed and analyzed. These algorithms monitor critical applications' responsiveness and trigger actions for software rejuvenation when host resources degrade due to a co-resident worm competing for host resources. We analyze the effectiveness of our algorithms through analytic modeling and detailed, extensive simulation studies. The key performance metrics investigated are application response time, mean time between rejuvenations and the steady state probability of host infection. We also use simulation models to investigate several design and parameter tuning issues. We investigate the relationship between the rate at which the application performance monitors can detect out-of-specification applications and the rate of worm propagation in the network.
                @inproceedings{Avritzer:2007:UPS:1216993.1217023,
                author = {Avritzer, Alberto and Cole, Robert G. and Weyuker, Elaine J.},
                title = {Using performance signatures and software rejuvenation for worm mitigation in tactical MANETs},
                booktitle = {Proceedings of the 6th international workshop on Software and performance},
                series = {WOSP '07},
                year = {2007},
                isbn = {1-59593-297-6},
                location = {Buenes Aires, Argentina},
                pages = {172--180},
                numpages = {9},
                url = {http://doi.acm.org/10.1145/1216993.1217023},
                doi = {10.1145/1216993.1217023},
                acmid = {1217023},
                publisher = {ACM},
                address = {New York, NY, USA},
                keywords = {computer worms, mitigation, mobile ad hoc networks (MANETS), software monitoring},
                }