This shows you the differences between two versions of the page.
emfase [2016/03/22 09:19] elisa.chiarani@unitn.it |
emfase [2021/01/29 10:58] |
||
---|---|---|---|
Line 1: | Line 1: | ||
- | ====== The EMFASE Project ====== | ||
- | |||
- | EMFASE (//Empirical Framework for Security Design and Economic Trade-Off//) is funded by **SESAR Joint Undertaking** (WPE Call for Tender) and is managed by Eurocontrol. | ||
- | |||
- | ===== Topic ===== | ||
- | |||
- | Evaluation and validation methodologies are integral parts of Air Traffic Management (ATM). They are | ||
- | well understood for safety, environmental and other business cases – for which operational validation | ||
- | guidelines exist which are well defined and widely used. In contrast, the effectiveness of risk | ||
- | assessment practices for security, as well as comparative evaluation of such practices, is largely | ||
- | uncharted territory. We don't know to what degree the practices and their activities provide security | ||
- | and whether or not they give return on investment. Furthermore, we currently don't know how to | ||
- | evaluate or compare security practices; there are no accepted metrics to decide that activity X works | ||
- | better than activity Y in a given setting. This becomes even more true in an uncertain and rapidly | ||
- | changing environment with changing demands by users and changing threats. | ||
- | |||
- | The question is: //How can SESAR stakeholders know that their methods for ensuring security in | ||
- | the complex ATM domain really work? Would additional expensive security analysis and | ||
- | measures be worth the cost?// | ||
- | |||
- | One cannot simply use proven techniques from safety and just replace "safety" with “security”: safety | ||
- | risk analysis assumes a game against Nature (including involuntary human errors), while security | ||
- | risks are a game against Man. Nature might not deliberately trigger two faults; while Man can. On the | ||
- | opposite side, Nature is never running short of budget or motivation, while Man does. | ||
- | The purpose of this project is to provide ways of evaluating and comparing risk assessment methods | ||
- | for security in ATM, especially in relation to human factors. The goal is to provide relevant | ||
- | stakeholders with the means to select the risk assessment methods are best suited for the task at hand, | ||
- | for example security assessment in relation to introduction of a particular new system by taking into | ||
- | account the specific aspect of security. | ||
- | |||
- | The only way to know the actual effectiveness of a risk assessment activity is to empirically investigate | ||
- | it. In this project we will therefore subject risk assessment methods to scientific empirical methods. It | ||
- | is obviously unfeasible to investigate all existing methods, so a selection of methods to investigate will | ||
- | be made. While the project will evaluate this selection of existing methods, the overall framework | ||
- | (concepts, terminology, study designs and metrics) that must be developed to do this evaluation will | ||
- | be of a general nature so as to enable later replications and comparable studies. | ||
- | |||
- | ===== Partners ===== | ||
- | |||
- | University of Trento (Coordinator, Italy), SINTEF, DeepBlue and University of Southampton. | ||
- | |||
- | ===== Project Internal Information ===== | ||
- | |||
- | Please check [[https://trinity.disi.unitn.it/emfase/|SVN Repository]] (Restricted Access) | ||
- | |||
- | ===== Project presentation ===== | ||
- | |||
- | |||
- | ===== Current Activities ===== | ||
- | |||
- | Below is the list of experiments and related activities. | ||
- | |||
- | Trento's results are also reported in [[validation_of_risk_and_security_requirements_methodologies|Empirical Validation of Risk and Security Methodologies]]. | ||
- | |||
- | |||
- | |||
- | ==== Criteria identification and validation ==== | ||
- | |||
- | - SESAR Jamboree Nov 2013 | ||
- | * Participants: ATM experts | ||
- | * Feedback: questionnaire, focus group interview | ||
- | |||
- | ==== Experiments ==== | ||
- | |||
- | === Comparison of Security Risk Assessment methods === | ||
- | - UNITN Security Engineering course 2013-14: | ||
- | * Participants: 29 MSc students enrolled to Security Engineering course at the University of Trento | ||
- | * Method: CORAS vs Eurocontrol SECRAM (*) | ||
- | * Case Study: SmartGrid | ||
- | * Final result: excel file with threats and controls, presentations, report | ||
- | * Feedback: questionnaire, interview | ||
- | - First International Week with Italian Post on Cyber Security in Complex Information Systems 2014 (Rome, Italy): | ||
- | * Participants: students - around 60 sort of controlled participants | ||
- | * Method: CORAS vs SESAR SECRAM (*) | ||
- | * Case Study: Online Banking | ||
- | * Final result: excel file with threats and controls, report | ||
- | * Feedback: questionnaire | ||
- | - UNITN Security Engineering course 2014-15: | ||
- | * Participants: MSc students - around 30 sort of controlled participants | ||
- | * Method: CORAS vs SESAR SecRAM (*) | ||
- | * Case Study: Remotely Operated Tower (ATM) (*) | ||
- | * Final result: excel file with threats and controls, presentations, report | ||
- | * Feedback: questionnaire, focus groups interview | ||
- | - UNITN Security Engineering course 2015-16: | ||
- | * Participants: MSc students - around 50 sort of controlled participants | ||
- | * Method: CORAS vs SESAR SecRAM (*) | ||
- | * Case Study: Unmanned Aerial System Traffic Management (UTM) | ||
- | * Final result: excel file with threats and controls, presentations, report | ||
- | * Feedback: questionnaire, focus groups interview | ||
- | |||
- | === Effectiveness of Catalogues of Threats and Security Controls in Security Risk Assessment === | ||
- | - EIT Winter School 2014: | ||
- | * Participants: students around 20 sort of controlled participants | ||
- | * Method: SESAR SecRAM (*) + [ BSI Catalog vs SECRAM Catalog (*) ] | ||
- | * Case Study: Remotely Operated Tower (*) | ||
- | * Final result: excel file with requirements, hand-drawn poster for result presentation, report | ||
- | * Feedback: questionnaire | ||
- | - EMFASE SecRAM Evaluation Workshop 2014: | ||
- | * Participants: professionals around 15 sort of controlled participants | ||
- | * Method: SESAR SecRAM (*) + [ BSI catalogue vs SECRAM catalogue (*) vs No catalogue (control group)] | ||
- | * Case Study: Remotely Operated Tower (*) | ||
- | * Final result: excel file with requirements, report | ||
- | * Feedback: questionnaire, focus groups interview | ||
- | |||
- | === An Empirical Comparison of Tabular vs. Graphical Risk Model Representations === | ||
- | - UNITN Security Engineering course 2014-15: | ||
- | * Participants: 35 MSc students - controlled participants | ||
- | * Representation: Graphical (CORAS) vs Tabular (NIST) | ||
- | * Scenario: Online Banking and Health Care Network | ||
- | * Final result: responses to the online comprehensibility task | ||
- | * Feedback: post-task questionnaire | ||
- | - University of Oslo Model Engineering course 2014-2015: | ||
- | * Participants: 11 MSc students - controlled participants | ||
- | * Representation: Graphical (CORAS) vs Tabular (NIST) | ||
- | * Scenario: Online Banking | ||
- | * Final result: responses to the online comprehensibility task | ||
- | * Feedback: post-task questionnaire | ||
- | - PUCRS Information Systems course 2014-15: | ||
- | * Participants: 27 MSc and 13 BSc students - controlled participants | ||
- | * Representation: Graphical (CORAS) vs Tabular (NIST) | ||
- | * Scenario: Online Banking and Health Care Network | ||
- | * Final result: responses to the online comprehensibility task | ||
- | * Feedback: post-task questionnaire | ||
- | - University of Calabria Cybersecurity professional master course - September 2015: | ||
- | * Participants: 52 MSc students - controlled participants | ||
- | * Representation: Graphical (CORAS) vs Tabular (NIST) | ||
- | * Scenario: Online Banking and Health Care Network | ||
- | * Final result: responses to the online comprehensibility task | ||
- | * Feedback: post-task questionnaire | ||
- | - UNITN Security Engineering course 2015-16: | ||
- | * Participants: 51 MSc students - controlled participants | ||
- | * Representation: Graphical (CORAS) vs Tabular (NIST) | ||
- | * Scenario: Online Banking and Health Care Network | ||
- | * Final result: responses to the online comprehensibility task | ||
- | * Feedback: post-task questionnaire | ||
- | - EMFASE - Security Risk Assessment Tutorial at SESAR Innovation Days 2015 (Bologna, Italy): | ||
- | * Participants: 14 professionals - sort of controlled participants | ||
- | * Representation: Graphical (CORAS) vs Tabular (SESAR SecRAM) | ||
- | * Scenario: Online Banking | ||
- | * Final result: responses to the paper-based comprehensibility task | ||
- | * Feedback: post-task questionnaire | ||
- | - EMFASE Online Study on Comprehensibility of Risk Models: | ||
- | * Participants: 60 professionals | ||
- | * Representation: Graphical (CORAS) vs Tabular (NIST) | ||
- | * Scenario: Online Banking | ||
- | * Final result: responses to the online comprehensibility task | ||
- | * Feedback: post-task questionnaire | ||
- | In part (*) means confidential documents are distributed | ||
- | |||
- | ===== Deliverables ===== | ||
- | - {{:projects:emfase:e.02.32_d1.1_selection_of_risk_assessment_methods_object_of_study_00.01.03.pdf|D1.1 Selection of risk assessment methods object of study}} | ||
- | - {{:projects:emfase:deliverable:d1-2_firstempiricalevaluationframework_v000102.pdf|D1.2 First Empirical Evaluation Framework}} | ||
- | - {{:projects:emfase:deliverable:e.02.32_d1.3_refinedempiricalevaluationframework_v000100.pdf|D1.3 Refined Empirical Evaluation Framework}} | ||
- | - {{:projects:emfase:deliverable:d2_1_scenariodescriptions_v00_01_03.pdf|D2.1 Scenario Descriptions}} | ||
- | - {{:projects:emfase:deliverable:e.02.32_-_emfase_-_d2.2_-_first_evaluation_report_ed.00.01.00.pdf|D2.2 First Evaluation Report}} | ||
- | - {{:projects:emfase:deliverable:e_02_32_-_emfase_-_d3_1_-_draft_causal_explanations-ed.00.01.00.pdf|D3.1 Draft Causal Explanations}} | ||
- | |||
- | ===== Publications ===== | ||
- | * K. Labunets, Y. Li, F. Massacci, F. Paci, M. Ragosta, B. Solhaug, K. Stølen, A. Tedeschi. **Preliminary Experiments on the Relative Comprehensibility of Tabular and Graphical Risk Models**, In //the Proceedings of 5th SESAR Innovation Days (SIDs'15).// {{:research_activities:experiments:2014-comprehensibility:labunets-etal-sids_2015_paper_32.pdf|PDF}} | ||
- | * K. Labunets, F. Paci, F. Massacci. **Which Security Catalogue Is Better for Novices?** In //Proc. of EmpiRE Workshop at IEEE RE'15.// {{:research_activities:experiments:2014-winter-school:labunets-etal-empire-re15-preprint.pdf|PDF (preprint)}} | ||
- | * M. de Gramatica, K. Labunets, F. Massacci, F. Paci, and A. Tedeschi. **The Role of Catalogues of Threats and Security Controls in Security Risk Assessment: An Empirical Study with ATM Professionals.** In //Proc. of REFSQ'15//. {{:research_activities:experiments:2014-rome-deepblue:gramatica-etal-refsq2015.pdf|PDF}} | ||
- | * K. Labunets, F. Massacci, F. Paci, M. Ragosta, B. Solhaug, K. Stølen, A. Tedeschi. **A First Empirical Evaluation Framework for Security Risk Assessment Methods in the ATM Domain**, In //the Proceedings of 4th SESAR Innovation Days (SIDs'14).// {{:research_activities:experiments:2014-seceng:labunets-etal-sids_2014_paper_40.pdf|PDF}} | ||
- | * M. Giacalone, R. Mammoliti, F. Massacci, F. Paci, R. Perugino, and C. Selli. **Security Triage: A Report of a Lean Security Requirements Methodology for Cost-Effective Security Analysis.** A short summary appears In //Proc. of EmpiRE Workshop at IEEE RE'14//. {{:research_activities:experiments:giacalone-etal-re14-preprint.pdf|3 pages PDF}}. A longer Industry report appears in //Proc. of ESEM'2014//. {{:research_activities:security_requirements_engineering:paper-207-esem-2014.pdf|PDF (preprint)}} | ||
- | * K. Labunets, F. Paci, F. Massacci, and R. Ruprai. **An Experiment on Comparing Textual vs. Visual Industrial Methods for Security Risk Assessment.** In //Proc. of EmpiRE Workshop at IEEE RE'14// {{:research_activities:experiments:labunets-etal-empire-re14-preprint.pdf|PDF}} | ||
- | |||