EMFASE (Empirical Framework for Security Design and Economic Trade-Off) is funded by SESAR Joint Undertaking (WPE Call for Tender) and is managed by Eurocontrol.

Evaluation and validation methodologies are integral parts of Air Traffic Management (ATM). They are well understood for safety, environmental and other business cases – for which operational validation guidelines exist which are well defined and widely used. In contrast, the effectiveness of risk assessment practices for security, as well as comparative evaluation of such practices, is largely uncharted territory. We don't know to what degree the practices and their activities provide security and whether or not they give return on investment. Furthermore, we currently don't know how to evaluate or compare security practices; there are no accepted metrics to decide that activity X works better than activity Y in a given setting. This becomes even more true in an uncertain and rapidly changing environment with changing demands by users and changing threats.

The question is: How can SESAR stakeholders know that their methods for ensuring security in the complex ATM domain really work? Would additional expensive security analysis and measures be worth the cost?

One cannot simply use proven techniques from safety and just replace “safety” with “security”: safety risk analysis assumes a game against Nature (including involuntary human errors), while security risks are a game against Man. Nature might not deliberately trigger two faults; while Man can. On the opposite side, Nature is never running short of budget or motivation, while Man does. The purpose of this project is to provide ways of evaluating and comparing risk assessment methods for security in ATM, especially in relation to human factors. The goal is to provide relevant stakeholders with the means to select the risk assessment methods are best suited for the task at hand, for example security assessment in relation to introduction of a particular new system by taking into account the specific aspect of security.

The only way to know the actual effectiveness of a risk assessment activity is to empirically investigate it. In this project we will therefore subject risk assessment methods to scientific empirical methods. It is obviously unfeasible to investigate all existing methods, so a selection of methods to investigate will be made. While the project will evaluate this selection of existing methods, the overall framework (concepts, terminology, study designs and metrics) that must be developed to do this evaluation will be of a general nature so as to enable later replications and comparable studies.

University of Trento (Coordinator, Italy), SINTEF, DeepBlue and University of Southampton.

Please check SVN Repository (Restricted Access)

EMFASE Poster presented at SID 2013

Below is the list of experiments and related activities.

Trento's results are also reported in Empirical Validation of Risk and Security Methodologies.

1. SESAR Jamboree Nov 2013
   • Participants: ATM experts
   • Feedback: questionnaire, focus group interview

1. UNITN Security Engineering course 2013-14:
   • Participants: 29 MSc students enrolled to Security Engineering course at the University of Trento
   • Method: CORAS vs Eurocontrol SECRAM (*)
   • Case Study: SmartGrid
   • Final result: excel file with threats and controls, presentations, report
   • Feedback: questionnaire, interview
2. First International Week with Italian Post on Cyber Security in Complex Information Systems 2014 (Rome, Italy):
   • Participants: students - around 60 sort of controlled participants
   • Method: CORAS vs SESAR SECRAM (*)
   • Case Study: Online Banking
   • Final result: excel file with threats and controls, report
   • Feedback: questionnaire
3. UNITN Security Engineering course 2014-15:
   • Participants: MSc students - around 30 sort of controlled participants
   • Method: CORAS vs SESAR SecRAM (*)
   • Case Study: Remotely Operated Tower (ATM) (*)
   • Final result: excel file with threats and controls, presentations, report
   • Feedback: questionnaire, focus groups interview
4. UNITN Security Engineering course 2015-16:
   • Participants: MSc students - around 50 sort of controlled participants
   • Method: CORAS vs SESAR SecRAM (*)
   • Case Study: Unmanned Aerial System Traffic Management (UTM)
   • Final result: excel file with threats and controls, presentations, report
   • Feedback: questionnaire, focus groups interview

1. EIT Winter School 2014:
   • Participants: students around 20 sort of controlled participants
   • Method: SESAR SecRAM (*) + [ BSI Catalog vs SECRAM Catalog (*) ]
   • Case Study: Remotely Operated Tower (*)
   • Final result: excel file with requirements, hand-drawn poster for result presentation, report
   • Feedback: questionnaire
2. EMFASE SecRAM Evaluation Workshop 2014:
   • Participants: professionals around 15 sort of controlled participants
   • Method: SESAR SecRAM (*) + [ BSI catalogue vs SECRAM catalogue (*) vs No catalogue (control group)]
   • Case Study: Remotely Operated Tower (*)
   • Final result: excel file with requirements, report
   • Feedback: questionnaire, focus groups interview

1. UNITN Security Engineering course 2014-15:
   • Participants: 35 MSc students - controlled participants
   • Representation: Graphical (CORAS) vs Tabular (NIST)
   • Scenario: Online Banking and Health Care Network
   • Final result: responses to the online comprehensibility task
   • Feedback: post-task questionnaire
2. University of Oslo Model Engineering course 2014-2015:
   • Participants: 11 MSc students - controlled participants
   • Representation: Graphical (CORAS) vs Tabular (NIST)
   • Scenario: Online Banking
   • Final result: responses to the online comprehensibility task
   • Feedback: post-task questionnaire
3. PUCRS Information Systems course 2014-15:
   • Participants: 27 MSc and 13 BSc students - controlled participants
   • Representation: Graphical (CORAS) vs Tabular (NIST)
   • Scenario: Online Banking and Health Care Network
   • Final result: responses to the online comprehensibility task
   • Feedback: post-task questionnaire
4. University of Calabria Cybersecurity professional master course - September 2015:
   • Participants: 52 MSc students - controlled participants
   • Representation: Graphical (CORAS) vs Tabular (NIST)
   • Scenario: Online Banking and Health Care Network
   • Final result: responses to the online comprehensibility task
   • Feedback: post-task questionnaire
5. UNITN Security Engineering course 2015-16:
   • Participants: 51 MSc students - controlled participants
   • Representation: Graphical (CORAS) vs Tabular (NIST)
   • Scenario: Online Banking and Health Care Network
   • Final result: responses to the online comprehensibility task
   • Feedback: post-task questionnaire
6. EMFASE - Security Risk Assessment Tutorial at SESAR Innovation Days 2015 (Bologna, Italy):
   • Participants: 14 professionals - sort of controlled participants
   • Representation: Graphical (CORAS) vs Tabular (SESAR SecRAM)
   • Scenario: Online Banking
   • Final result: responses to the paper-based comprehensibility task
   • Feedback: post-task questionnaire
7. EMFASE Online Study on Comprehensibility of Risk Models:
   • Participants: 60 professionals
   • Representation: Graphical (CORAS) vs Tabular (NIST)
   • Scenario: Online Banking
   • Final result: responses to the online comprehensibility task
   • Feedback: post-task questionnaire

In part (*) means confidential documents are distributed

1. D1.1 Selection of risk assessment methods object of study
2. D1.2 First Empirical Evaluation Framework
3. D1.3 Refined Empirical Evaluation Framework
4. D2.1 Scenario Descriptions
5. D2.2 First Evaluation Report
6. D3.1 Draft Causal Explanations

• K. Labunets, Y. Li, F. Massacci, F. Paci, M. Ragosta, B. Solhaug, K. Stølen, A. Tedeschi. Preliminary Experiments on the Relative Comprehensibility of Tabular and Graphical Risk Models, In the Proceedings of 5th SESAR Innovation Days (SIDs'15). PDF
• K. Labunets, F. Paci, F. Massacci. Which Security Catalogue Is Better for Novices? In Proc. of EmpiRE Workshop at IEEE RE'15. PDF (preprint)
• M. de Gramatica, K. Labunets, F. Massacci, F. Paci, and A. Tedeschi. The Role of Catalogues of Threats and Security Controls in Security Risk Assessment: An Empirical Study with ATM Professionals. In Proc. of REFSQ'15. PDF
• K. Labunets, F. Massacci, F. Paci, M. Ragosta, B. Solhaug, K. Stølen, A. Tedeschi. A First Empirical Evaluation Framework for Security Risk Assessment Methods in the ATM Domain, In the Proceedings of 4th SESAR Innovation Days (SIDs'14). PDF
• M. Giacalone, R. Mammoliti, F. Massacci, F. Paci, R. Perugino, and C. Selli. Security Triage: A Report of a Lean Security Requirements Methodology for Cost-Effective Security Analysis. A short summary appears In Proc. of EmpiRE Workshop at IEEE RE'14. 3 pages PDF. A longer Industry report appears in Proc. of ESEM'2014. PDF (preprint)
• K. Labunets, F. Paci, F. Massacci, and R. Ruprai. An Experiment on Comparing Textual vs. Visual Industrial Methods for Security Risk Assessment. In Proc. of EmpiRE Workshop at IEEE RE'14 PDF