Saturday, 23 September 2017

Group Publications

The publications endorsed by SPEC RG are available at the publication page.

Affiliated Publications

In the following, we list a selection of relevant publications by members of the IDS Benchmarking Working Group, which are not formally endorsed by SPEC.
Aleksandar Milenkoski, Bryan D. Payne, Nuno Antunes, Marco Vieira, Samuel Kounev, Alberto Avritzer, and Matthias Luft, "Evaluation of Intrusion Detection Systems in Virtualized Environments Using Attack Injection", In Proceedings of The 18th International Symposium on Research in Attacks, Intrusions, and Defenses (RAID 2015), 2015
[ bibtex | abstract ]
Keywords: Cloud computing; Virtualization; Hypercalls; Intrusion detection systems; Workloads; Attack injection.

The evaluation of intrusion detection systems (IDSes) is an active research area with many open challenges, one of which is the generation of representative workloads that contain attacks. In this paper, we propose a novel approach for the rigorous evaluation of IDSes in virtualized environments, with a focus on IDSes designed to detect attacks leveraging or targeting the hypervisor via its hypercall interface. We present hInjector, a tool for generating IDS evaluation workloads by injecting such attacks during regular operation of a virtualized environment. We demonstrate the application of our approach and show its practical usefulness by evaluating a representative IDS designed to operate in virtualized environments. The virtualized environment of the industry-standard benchmark SPECvirt_sc2013 is used as a testbed, whose drivers generate workloads representative of workloads seen in production environments. This work enables for the first time the injection of attacks in virtualized environments for the purpose of generating representative IDS evaluation workloads.

@inproceedings{MiPaAnViKoAvLu2015-RAID-Challenges,
author = {Aleksandar Milenkoski and Bryan D. Payne and Nuno Antunes and Marco Vieira and Samuel Kounev and Alberto Avritzer and Matthias Luft},
booktitle = {The 18th International Symposium on Research in Attacks, Intrusions, and Defenses (RAID 2015)},
publisher = {{Springer}},
location = {Kyoto, Japan},
title = {{Evaluation of Intrusion Detection Systems in Virtualized Environments Using Attack Injection}},
year = {2015},
month = {{November}},
url = {http://link.springer.com/chapter/10.1007/978-3-319-26362-5_22},
}
Aleksandar Milenkoski, Marco Vieira, Samuel Kounev, Alberto Avrtizer, and Bryan D. Payne, "Evaluating Computer Intrusion Detection Systems: A Survey of Common Practices", ACM Computing Surveys, 48(1):12:1-12:41, September 2015, ACM, New York, NY, USA. 5-year Impact Factor (2014): 5.949.
[ bibtex | abstract ]
Keywords: Survey; Intrusion detection systems; Evaluation; Experimentation.

The evaluation of computer intrusion detection systems (which we refer to as intrusion detection systems) is an active research area. In this paper, we survey and systematize common practices in the area of evaluation of intrusion detection systems. For this purpose, we define a design space structured into three parts: workload, metrics, and measurement methodology. We then provide an overview of the common practices in evaluation of intrusion detection systems by surveying evaluation approaches and methods related to each part of the design space. Finally, we discuss open issues and challenges focusing on evaluation methodologies for novel intrusion detection systems.
@article{MiViKoAvPa2015-CSUR-IDSEval,
author = {Aleksandar Milenkoski and Marco Vieira and Samuel Kounev and Alberto Avrtizer and Bryan D. Payne},
title = {{Evaluating Computer Intrusion Detection Systems: A Survey of Common Practices}},
journal = {{ACM Computing Surveys}},
publisher = {ACM},
address = {New York, NY, USA},
year = {2015},
month = {{September}},
volume = {48},
number = {1},
pages = {12:1--12:41},
url = {http://dl.acm.org/authorize?N06203},
}
Aleksandar Milenkoski, Bryan D. Payne, Nuno Antunes, Marco Vieira, and Samuel Kounev, "Experience Report: An Analysis of Hypercall Handler Vulnerabilities", In Proceedings of The 25th IEEE International Symposium on Software Reliability Engineering (ISSRE 2014) - Research Track, 2014
[ bibtex | abstract ]
Keywords: Cloud computing; Virtualization; Hypercalls; Vulnerability analysis.

Hypervisors are becoming increasingly ubiquitous with the growing proliferation of virtualized data centers. As a result, attackers are exploring vectors to attack hypervisors, against which an attack may be executed via several attack vectors such as device drivers, virtual machine exit events, or hypercalls. Hypercalls enable intrusions in hypervisors through their hypercall interfaces. Despite the importance, there is very limited publicly available information on vulnerabilities of hypercall handlers and attacks triggering them, which significantly hinders advances towards monitoring and securing these interfaces. In this paper, we characterize the hypercall attack surface based on analyzing a set of vulnerabilities of hypercall handlers. We systematize and discuss the errors that caused the considered vulnerabilities, and activities for executing attacks triggering them. We also demonstrate attacks triggering the considered vulnerabilities and analyze their effects. Finally, we suggest an action plan for improving the security of hypercall interfaces.
@inproceedings{MiPaAnViKo2014-ISSRE-AnAnalHypHanVulns,
author = {Aleksandar Milenkoski and Bryan D. Payne and Nuno Antunes and Marco Vieira and Samuel Kounev},
title = {{Experience Report: An Analysis of Hypercall Handler Vulnerabilities}},
location = {Naples, Italy},
month = {November},
booktitle = {{Proceedings of The 25th IEEE International Symposium on Software Reliability Engineering (ISSRE 2014) --- Research Track}},
organization = {IEEE},
publisher = {IEEE Computer Society},
address = {{Washington DC, USA}},
year = {2014},
}
Aleksandar Milenkoski, Bryan D. Payne, Nuno Antunes, Marco Vieira, and Samuel Kounev, "HInjector: Injecting Hypercall Attacks for Evaluating VMI-based Intrusion Detection Systems", In The 2013 Annual Computer Security Applications Conference (ACSAC 2013) (Poster paper), 2013
[ bibtex ]
Keywords: Cloud computing; Virtualization; Intrusion detection system benchmarking.

@inproceedings{MiPaAnViKo2013-ACSAC-HInjector,
  address = {Maryland, USA},
  author = {Aleksandar Milenkoski and Bryan D. Payne and Nuno Antunes and Marco Vieira and Samuel Kounev},
  booktitle = {The 2013 Annual Computer Security Applications Conference (ACSAC 2013)},
  publisher = {{Applied Computer Security Associates (ACSA)}},
  location = {New Orleans, Louisiana, USA},
  title = {{HInjector: Injecting Hypercall Attacks for Evaluating VMI-based Intrusion Detection Systems (Poster Paper)}},
  year = {2013}
}

Jose Fonseca, Marco Vieira, and Henrique Madeira, "Vulnerability & Attack Injection for Web Applications", in 39th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN 2009), pp. 93-102, 2009
[ bibtex | abstract ]
Keywords: Intrusion Detection; Performance Evaluation; Probes; Testing.

In this paper we propose a methodology to inject realistic attacks in Web applications. The methodology is based on the idea that by injecting realistic vulnerabilities in a Web application and attacking them automatically we can assess existing security mechanisms. To provide true to life results, this methodology relies on field studies of a large number of vulnerabilities in Web applications. The paper also describes a set of tools implementing the proposed methodology. They allow the automation of the entire process, including gathering results and analysis. We used these tools to conduct a set of experiments to demonstrate the feasibility and effectiveness of the proposed methodology. The experiments include the evaluation of coverage and false positives of an intrusion detection system for SQL injection and the assessment of the effectiveness of two Web application vulnerability scanners. Results show that the injection of vulnerabilities and attacks is an effective way to evaluate security mechanisms and tools.
@INPROCEEDINGS{5270349,
	author={Fonseca, J. and Vieira, M. and Madeira, H.},
	booktitle={IEEE/IFIP International Conference on Dependable Systems Networks, 2009. DSN '09.}, 
	title={Vulnerability #x00026; attack injection for web applications},
	year={2009},
	month={29 2009-july 2},
	pages={93 -102},
	keywords={Application software;Automation;Counting circuits;Information security;Inspection;Instruments;Intrusion detection;Performance evaluation;Probes;Testing;Internet;SQL;program debugging;program testing;security of data;SQL injection;Web application;Web application vulnerability scanner;intrusion detection system;realistic attack injection tool;realistic test bed;realistic vulnerability injection;security mechanism;software bug;},
	doi={10.1109/DSN.2009.5270349},
}
						
Ivano A. Elia, Jose Fonseca, and Marco Vieira, "Comparing SQL Injection Detection Tools Using Attack Injection: An Experimental Study", in 21st Annual International Symposium on Software Reliability Engineering, pp. 289-298, 2010
[ bibtex | abstract ]
Keywords: Databases; Intrusion Detection; Monitoring; SQL Injection Detection.

System administrators frequently rely on intrusion detection tools to protect their systems against SQL Injection, one of the most dangerous security threats in database-centric web applications. However, the real effectiveness of those tools is usually unknown, which may lead administrators to put an unjustifiable level of trust in the tools they use. In this paper we present an experimental evaluation of the effectiveness of five SQL Injection detection tools that operate at different system levels: Application, Database and Network. To test the tools in a realistic scenario, Vulnerability and Attack Injection is applied in a setup based on three web applications of different sizes and complexities. Results show that the assessed tools have a very low effectiveness and only perform well under specific circumstances, which highlight the limitations of current intrusion detection tools in detecting SQL Injection attacks. Based on experimental observations we underline the strengths and weaknesses of the tools assessed.
@INPROCEEDINGS{5635053,
	author={Elia, I.A. and Fonseca, J. and Vieira, M.},
	booktitle={Software Reliability Engineering (ISSRE), 2010 IEEE 21st International Symposium on}, 
	title={Comparing SQL Injection Detection Tools Using Attack Injection: An Experimental Study},
	year={2010},
	month={nov.},
	pages={289 -298},
	keywords={Databases;Intrusion detection;Monitoring;Payloads;Scalp;Web server;Internet;SQL;security of data;SQL injection detection tools;attack injection;database-centric Web application;intrusion detection tool;security threat;system administration;vulnerability;Fault Injection;Intrusion Detection;SQL Injection;Security;Web applications;},
	doi={10.1109/ISSRE.2010.32},
	ISSN={1071-9458},
}
						
Nuno Antunes and Marco Vieira, "Benchmarking Vulnerability Detection Tools for Web Services", in IEEE International Conference on Web Services (ICWS 2010), pp. 203-210, 2010
[ bibtex | abstract ]
Keywords: Vulnerability Benchmarking; Web Services.

Vulnerability detection tools are frequently considered the silver-bullet for detecting vulnerabilities in web services. However, research shows that the effectiveness of most of those tools is very low and that using the wrong tool may lead to the deployment of services with undetected vulnerabilities. In this paper we propose a benchmarking approach to assess and compare the effectiveness of vulnerability detection tools in web services environments. This approach was used to define a concrete benchmark for SQL Injection vulnerability detection tools. This benchmark is demonstrated by a real example of benchmarking several widely used tools, including four penetration-testers, three static code analyzers, and one anomaly detector. Results show that the benchmark accurately portrays the effectiveness of vulnerability detection tools and suggest that the proposed approach can be applied in the field.
@article{10.1109/ICWS.2010.76,
	author = {Nuno Antunes and Marco Vieira},
	title = {Benchmarking Vulnerability Detection Tools for Web Services},
	journal ={2012 IEEE 19th International Conference on Web Services},
	volume = {0},
	isbn = {978-0-7695-4128-0},
	year = {2010},
	pages = {203-210},
	doi = {http://doi.ieeecomputersociety.org/10.1109/ICWS.2010.76},
	publisher = {IEEE Computer Society},
	address = {Los Alamitos, CA, USA},
}
						
Alberto Avritzer, Andre Bondi, and Elaine Weyuker, "Ensuring stable performance for systems that degrade", in Proceedings of the 5th international workshop on software and performance, pp. 43-51, 2005
[ bibtex | abstract ]

A new approach that is useful in identifying and eliminating performance degradation occurring in aging software is proposed. A customer-affecting metric is used to initiate the restoration of such a system to full capacity. A case study is described in which, by simulating an industrial software system, we are able to show that by monitoring a customer-affecting metric and frequently comparing its degradation to the performance objective, we can ensure system stability at a very low cost.
@inproceedings{Avritzer:2005:ESP:1071021.1071026,
 author = {Avritzer, Alberto and Bondi, Andre and Weyuker, Elaine J.},
 title = {Ensuring stable performance for systems that degrade},
 booktitle = {Proceedings of the 5th international workshop on Software and performance},
 series = {WOSP '05},
 year = {2005},
 isbn = {1-59593-087-6},
 location = {Palma, Illes Balears, Spain},
 pages = {43--51},
 numpages = {9},
 url = {http://doi.acm.org/10.1145/1071021.1071026},
 doi = {10.1145/1071021.1071026},
 acmid = {1071026},
 publisher = {ACM},
 address = {New York, NY, USA},
}
						
Alberto Avritzer, Rajanikanth Tanikella, Kiran James, Robert G. Cole, and Elaine Weyuker, "Monitoring for security intrusion using performance signatures", in Proceedings of the first joint WOSP/SIPEW international conference on Performance engineering, pp. 93-104, 2010
[ bibtex | abstract ]
Keywords: Measurement; Monitoring; Performance signatures; Security.

A new approach for detecting security attacks on software systems by monitoring the software system performance signatures is introduced. We present a proposed architecture for security intrusion detection using off-the-shelf security monitoring tools and performance signatures. Our approach relies on the assumption that the performance signature of the well-behaved system can be measured and that the performance signature of several types of attacks can be identified. This assumption has been validated for operations support systems that are used to monitor large infrastructures and receive aggregated traffic that is periodic in nature. Examples of such infrastructures include telecommunications systems, transportation systems and power generation systems. In addition, significant deviation from well-behaved system performance signatures can be used to trigger alerts about new types of security attacks. We used a custom performance benchmark and five types of security attacks to derive performance signatures for the normal mode of operation and the security attack mode of operation. We observed that one of the types of the security attacks went undetected by the off-the-shelf security monitoring tools but was detected by our approach of monitoring performance signatures. We conclude that an architecture for security intrusion detection can be effectively complemented by monitoring of performance signatures.
@inproceedings{Avritzer:2010:MSI:1712605.1712623,
 author = {Avritzer, Alberto and Tanikella, Rajanikanth and James, Kiran and Cole, Robert G. and Weyuker, Elaine},
 title = {Monitoring for security intrusion using performance signatures},
 booktitle = {Proceedings of the first joint WOSP/SIPEW international conference on Performance engineering},
 series = {WOSP/SIPEW '10},
 year = {2010},
 isbn = {978-1-60558-563-5},
 location = {San Jose, California, USA},
 pages = {93--104},
 numpages = {12},
 url = {http://doi.acm.org/10.1145/1712605.1712623},
 doi = {10.1145/1712605.1712623},
 acmid = {1712623},
 publisher = {ACM},
 address = {New York, NY, USA},
 keywords = {measurement, monitoring, performance signatures, security},
}

		
Katinka Wolter, Alberto Avritzer, Marco Vieira, and Aad van Moorsel (Eds.), "Resilience Assessment and Evaluation of Computing Systems", XVIII, 489 p. 95 illus, ISBN: 978-3-642-29031-2, Springer, 2012
Keywords: Benchmarking; Dependability; Networks; Performance; Reliability; Robustness; Statistical methods; Stochastic modeling; System evaluation; Testing.

Aleksandar Milenkoski and Samuel Kounev, "Towards benchmarking intrusion detection systems for virtualized cloud environments", in Proceedings of the 2012 International Conference For Internet Technology And Secured Transactions (Work-in-progress paper), pp. 562-563, 2012
[ bibtex | abstract ]
Keywords: Cloud computing; Security of data; Virtualisation; Intrusion detection system benchmarking.

Many recent research works propose novel architectures of intrusion detection systems specifically designed to operate in virtualized environments. However, little attention has been given to the evaluation and benchmarking of such architectures with respect to their performance and dependability. In this paper, we present a research roadmap towards developing a framework for benchmarking intrusion detection systems for cloud environments in a scientifically rigorous and a representative manner.
@INPROCEEDINGS{6470873,
author={Milenkoski, A. and Kounev, S.},
booktitle={2012 International Conference For Internet Technology And Secured Transactions}, title={Towards benchmarking intrusion detection systems for virtualized cloud environments},
year={2012},
pages={562-563},
keywords={cloud computing;security of data;virtualization;intrusion detection system benchmarking;virtualized cloud environments;Benchmark testing;Cloud computing;Hardware;Intrusion detection;Measurement;Monitoring;Virtual machine monitors;benchmark testing;intrusion detection},}
		
		
Alberto Avritzer, Robert G. Cole, and Elaine Weyuker, "Using performance signatures and software rejuvenation for worm mitigation in tactical MANETs", in Proceedings of the 6th international workshop on Software and performance (WOSP '07), pp. 172-180, 2007
[ bibtex | abstract ]
Keywords: Network monitoring; Protection Mechanisms; Performance signatures; Computer worms; Mitigation.

In this paper, we propose a new approach for mitigation of worm propagation through tactical Mobile Ad-Hoc Networks (MANETs) which is based upon performance signatures and software rejuvenation. Three application performance signature and software rejuvenation algorithms are proposed and analyzed. These algorithms monitor critical applications' responsiveness and trigger actions for software rejuvenation when host resources degrade due to a co-resident worm competing for host resources. We analyze the effectiveness of our algorithms through analytic modeling and detailed, extensive simulation studies. The key performance metrics investigated are application response time, mean time between rejuvenations and the steady state probability of host infection. We also use simulation models to investigate several design and parameter tuning issues. We investigate the relationship between the rate at which the application performance monitors can detect out-of-specification applications and the rate of worm propagation in the network.
@inproceedings{Avritzer:2007:UPS:1216993.1217023,
 author = {Avritzer, Alberto and Cole, Robert G. and Weyuker, Elaine J.},
 title = {Using performance signatures and software rejuvenation for worm mitigation in tactical MANETs},
 booktitle = {Proceedings of the 6th international workshop on Software and performance},
 series = {WOSP '07},
 year = {2007},
 isbn = {1-59593-297-6},
 location = {Buenes Aires, Argentina},
 pages = {172--180},
 numpages = {9},
 url = {http://doi.acm.org/10.1145/1216993.1217023},
 doi = {10.1145/1216993.1217023},
 acmid = {1217023},
 publisher = {ACM},
 address = {New York, NY, USA},
 keywords = {computer worms, mitigation, mobile ad hoc networks (MANETS), software monitoring},
}