User Tools

Site Tools


seceng-course-exp-2012

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
seceng-course-exp-2012 [2014/07/08 15:47]
martina.degramatica@unitn.it [Task]
seceng-course-exp-2012 [2021/01/29 10:58] (current)
Line 3: Line 3:
 An experiment by Katsyarina Labunets, Fabio Massacci, Federica Paci, Le Minh Sang Tran. An experiment by Katsyarina Labunets, Fabio Massacci, Federica Paci, Le Minh Sang Tran.
  
-This page provides additional resources that enable replication of our work published at {{:​research_activities:​experiments:​2013-seceng:​labunets-esem-2013-accepted.pdf|ESEM 2013}}.+This page provides additional resources that enable replication of our work published at {{:​research_activities:​experiments:​2013-seceng:​labunets-esem-2013-accepted.pdf|ESEM 2013}}. See the [[validation_of_risk_and_security_requirements_methodologies|main page]] for our work on empirical validation of security risk assessment methods and other experiments.
 ===== Goals ===== ===== Goals =====
-The goal of the experiment was to evaluate and compare two types of risk-driven methods, namely, visual methods (CORAS) and textual methods (SREP) with respect to their effectiveness in identifying threats and security requirements,​ and the participants’ perception of the two methods.+The goal of the experiment was to evaluate and compare two types of risk-driven methods, namely, visual methods (CORAS) and textual methods (SREP) with respect to their //effectiveness// in identifying threats and security requirements,​ and the //participants’ perception// of the two methods.
 ===== Context of the Experiment ===== ===== Context of the Experiment =====
  
Line 12: Line 12:
  
 ==== Methods ==== ==== Methods ====
-Method experts presented to the participants a {{:​research_activities:​experiments:​2013-seceng:​lecture-06-riskwithcoras.pptx|CORAS ​tutorial}} and {{:​research_activities:​experiments:​2013-seceng:​srep_tutorial.pdf|SREP ​tutorial}}. +The methods evaluated were {{:​research_activities:​experiments:​2013-seceng:​lecture-06-riskwithcoras.pptx|CORAS}} ​(visual method) ​and {{:​research_activities:​experiments:​2013-seceng:​srep_tutorial.pdf|SREP}} 
-The methods are briefly introduced as follows: +(textual ​method).
- +
-CORAS is a visual ​method ​which consists of three tightly integrated parts: a method for risk analysis, a language for risk modeling, and a tool to support the risk analysis process. The risk analysis in CORAS is a structured and systematic process which use diagrams to document the result of the execution of each step. The steps are based on the international standard ISO 31000 for risk management: context establishment,​ risk analysis (that identifies assets, unwanted incidents, threats and vulnerabilities), and risk treatments. +
- +
-SREP is an asset-based and risk-driven method for the establishment of security requirements of secure Information Systems. SREP supports a micro-process,​ consisting of nine steps: agree on definitions,​ identify critical assets, identify security objectives, identify threats and develop artifacts, risk assessment, elicit security requirements,​ categorize and prioritize security requirements,​ requirements inspection, and repository improvement. The result of the execution of each step of the process is represented using tables or natural language. SREP is compliant with international standards ISO/IEC 27002 and ISO/IEC 15408 within the scope of requirements engineering and security management.+
  
  
 ==== Case Study ==== ==== Case Study ====
-A domain expert introduced to the participants the {{:​research_activities:​experiments:​2013-seceng:​lecture-03-casestudy.pptx|Smart Grid application scenario}}.  +The participants ​applied ​the methods to a {{:​research_activities:​experiments:​2013-seceng:​lecture-03-casestudy.pptx|Smart Grid application scenario}}. ​
-The Smart Grid is an electricity network that can integrate in a cost-efficient manner the behavior and actions of all users connected to it like generators, and consumers. They use information and communication technologies to optimize the transmission and distribution of electricity from suppliers to consumers.+
  
  
Line 28: Line 23:
 The experiment was conducted as part of the Security Engineering course. Here, you can find the summary of the {{:​research_activities:​experiments:​2013-seceng:​experiment-agenda.pdf|Tasks}} to be accomplished in the experiment. ​ The experiment was conducted as part of the Security Engineering course. Here, you can find the summary of the {{:​research_activities:​experiments:​2013-seceng:​experiment-agenda.pdf|Tasks}} to be accomplished in the experiment. ​
 ===== Measurements ===== ===== Measurements =====
-During the experiment two type of questionnaires have been distributed to the participants:​ a Background Questionnaire ({{:​research_activities:​experiments:​2013-seceng:​q1_-_background.docx|Q1}}) and a Post-Task Questionnaire ({{:​research_activities:​experiments:​2013-seceng:​seceng2013-post-task-questionnaire.pdf|Q2}}) to assess ​the difference in the participants’ perception of visual and textual methods.  + 
-Students also evaluated the methods through an {{:​research_activities:​experiments:​2013-seceng:​interview_guide.docx|Interview ​Session}} giving their overall ​opinion ​and listing ​advantages and disadvantages of the considered methodologies+ 
-Lastly, the participants had to summarize the results they achieved in a {{:​research_activities:​experiments:​2013-seceng:​securityengineering2013.docx|Final Report}}. ​+  * {{:​research_activities:​experiments:​2013-seceng:​q1_-_background.docx|Background Questionnaire}} - collect participants demographic data. 
 +  * {{:​research_activities:​experiments:​2013-seceng:​seceng2013-post-task-questionnaire.pdf|Post-Task Questionnaire}} assess participants’ perception of visual and textual methods.  
 +  ​* ​{{:​research_activities:​experiments:​2013-seceng:​interview_guide.docx|Interview ​Guide}} - collect participants' ​opinion ​on advantages and disadvantages of visual and textual methods
 +  ​* ​{{:​research_activities:​experiments:​2013-seceng:​securityengineering2013.docx|Final Report}} ​- document methods'​ application
  
  
 ===== Results ===== ===== Results =====
-The main findings are that the visual method ​yields to identify ​more threats than textual ​one, while the textual one is slightly better to identify security requirementsThe difference in the number of threats identified with the two methods is statistically significant ​and participants’ interviews suggests that this is due to the difference in the artifacts used to model threats.  +  * //​Methods'​ effectiveness//​ 
-The visual method ​uses diagrams ​to represent threats while the textual method uses tables: diagrams help brainstorming on threats ​and thus yield participants to identify more threatsOn the contrary, the difference in the number of security requirements identified ​with the two methods ​is not statistically significant +Results show that visual method ​is more effective in identifying ​threats than textual ​methodThis is confirmed if we consider ​the //number of threats// identified with visual ​and textual methods across the task assigned ​to the groups. ​ Instead, with respect ​to //number of security requirements//,​ textual method is slightly more effective than 
-The textual ​method ​identified a slightly higher number ​of security requirements but this is not statistically significant. A possible explanation emerging from the interviews ​is that process supported ​by the textual method +the visual one in identifying security requirements
-offers ​systematic approach to identify security requirements+  * //​Methods'​ perception//​ 
-In addition, ​the visual method’s overall perception and intention ​to use are higher than for the textual ​method.+Participants’ //overall preference//​ is higher for visual ​than for textual ​method, while regarding ​to the perceived ease of use and the usefulness no statistically significant difference is proven by the experimentMoreover, in respect to the intention to use, the difference in participants’ perception is statistically significant in favour of the visual method.  
 +  * //​Qualitative Explanation//​ 
 +The different ​number of threats and security requirements identified ​can be likely explained by the differences between ​the two methods ​indicated by the participants during the interviews//Diagrams in visual ​method ​help brainstorming on the threats//, giving an overview ​of the possible threats, the threat scenarios and the assets, while the identification of threats in textual method ​is not facilitated by the use of tables as it is more difficult to link assets and threats. As suggested ​by the participants then, the identification of threats in textual method ​could be made easier if catalog of common threats was available.  
 +On the other side, //textual method is slightly more effective in eliciting security requirements//​ than visual ​approach because the order of steps in textual ​method ​process guides the analyst, while the same it seems not to hold for the visual ​method’s process.
 ===== Additional Material ===== ===== Additional Material =====
   * For additional information on the experimental design please see the {{:​research_activities:​experiments:​2013-seceng:​experiment-description.pdf|Experimental Protocol}}.   * For additional information on the experimental design please see the {{:​research_activities:​experiments:​2013-seceng:​experiment-description.pdf|Experimental Protocol}}.
   * For privacy reasons, at the beginning of the experiment a {{:​research_activities:​experiments:​2013-seceng:​consent-form-security-engineering.docx|Consent Form}} was administered to participants.  ​   * For privacy reasons, at the beginning of the experiment a {{:​research_activities:​experiments:​2013-seceng:​consent-form-security-engineering.docx|Consent Form}} was administered to participants.  ​
   * Participants'​ results have been assessed by methods and domain experts (see {{:​research_activities:​experiments:​2013-seceng:​evaluation_sheet.xlsx|Evaluation Score Sheet}}). ​   * Participants'​ results have been assessed by methods and domain experts (see {{:​research_activities:​experiments:​2013-seceng:​evaluation_sheet.xlsx|Evaluation Score Sheet}}). ​
 +  * 
 +
 +
 +**Data collected during the experiment are available upon request.**
   ​   ​
seceng-course-exp-2012.1404827260.txt.gz · Last modified: 2021/01/29 10:58 (external edit)