User Tools

Site Tools


seceng-course-exp-2013

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
seceng-course-exp-2013 [2014/03/25 13:35]
katsiaryna.labunets@unitn.it [Supplement Materials]
seceng-course-exp-2013 [2021/01/29 10:58] (current)
Line 1: Line 1:
  
  
-====== ​A Replicated ​Experiment on Comparing Textual vs Visual Methods for Security Risk Assessment ======+====== ​An Experiment on Comparing Textual vs Visual ​Industrial ​Methods for Security Risk Assessment ======
  
-The goal of the experiment ​we want to conduct following the goal/​question/​metric (GQM) template was to compare visual and textual methods for security risk assessment methods with respect to how successful they are in identifying threats and security controls. The context of the experiment consists of security requirements engineers who identify a list of security requirements for an advanced metering infrastructure scenario from the Smart Grid domain+An experiment ​by Katsyarina Labunets, Fabio Massacci, Federica Paci.
  
-==== Supplement Materials ​==== +This page provides additional resources that enable replication of our work published at {{:​research_activities:​experiments:​2014-seceng:​empire-2014.pdf|EmpiRE2014}}. See the [[validation_of_risk_and_security_requirements_methodologies|main page]] for our work on empirical validation of security risk assessment methods and other experiments. 
-During ​the experiment ​we distributed among participants two type of questionnaires:  ​ + 
-  * Pre-task questionnaire to collect some information about participants ​and thier background: {{:​research_activities:​experiments:​2013-2014-se-exp:seceng2013-2014-background.pdf|Q1}}. +===== Goals ===== 
-  * Post-task questionnaire to collect ​participants' ​perception of methods{{:​research_activities:​experiments:​2013-2014-se-exp:seceng2013-2014-post-task-questionnaire.pdf|Q2}}. + 
-For individual interview we used the following interview guide: {{:​research_activities:​experiments:​2013-2014-se-exp:​seceng2013-2014-interviewguide.pdf|guide}}.+The goal of the experiment ​was to compare the actual effectiveness,​ and perception of visual (CORAS) and textual (EUROCONTROL SecRAM) methods for security risk assessment with respect to their //​effectiveness//​ in identifying threats and security requirements,​ and the //participants’ perception//​ of the two methods. 
 + 
 + 
 +===== Context ​of the Experiment ===== 
 + 
 +==== Subjects ==== 
 + 
 +We involved 29 participants: ​15 students of the MSc in Computer Science and 14 students of the EIT ICT LAB MSc in Security and Privacy of the University of Trento. In this within-subject design, each participant applied both methods.  
 + 
 +==== Methods ==== 
 + 
 +The methods evaluated were {{:​research_activities:​experiments:​2014-seceng:​lecture-06-risk_with_coras.pptx|CORAS}} (visual method) and {{:​research_activities:​experiments:​2014-seceng:​lecture-07-risk_assessment-secram.pptx|EUROCONTROL SecRAM }} (textual method).  
 + 
 +==== Case study ==== 
 + 
 +The participants applied the methods to a {{:research_activities:​experiments:​2014-seceng:​lecture-02-smartmetering-casestudy.pptx|Smart Grid application scenario}}.  
 + 
 +===== Measurements ===== 
 +  * {{:​research_activities:​experiments:​2014-seceng:q1.docx|Background Questionnaire}} - collect participants demographic data
 +  * {{:​research_activities:​experiments:​2014-seceng:​q2.docx|Post-Task Questionnaire}} - assess ​participants’ perception of visual and textual ​methods
 +  * {{:​research_activities:​experiments:​2014-seceng:​SecEng2013-2014-interviewguide.docx|Interview Guide}} ​collect participants'​ opinion on advantages and disadvantages of visual and textual methods. 
 +  * {{:​research_activities:​experiments:2014-seceng:​security_engineering_report.docx|Final Report}} ​document methods'​ application.  
 + 
 +===== Results ===== 
 +  * //​Methods'​ effectiveness//​ 
 +Results show that there is //no difference in the number of threats and controls identified//​ with CORAS and EUROCONTROL SecRAM, differently from the results we achieved in the previous experiment where the visual method (CORAS) performed better in threats identification rather than the textual method (SREP). This difference may be due to the change of the textual method: SecRAM could perform better than SREP, or due to the difference in the experimental design. In the first experiments indeed participants applied each method twice, while in the present experiment there was only one application of the method. The participants of the first experiment might have learnt methods better and produced significant results. 
 + 
 +  * //Methods perception//​  
 +Participants’ //overall perception//​ is higher for visual than for textual method with statistical significance for all participants. The same result holds for the perceived usefulness and the intention to use; however, regarding the perceived ease of use no statistically significant difference is proven by the experiment. Similar results were found in the first experiment.  
 + 
 +  * //​Qualitative Explanation//​ 
 +Qualitative explanations from the interviews illustrate that the visual method is perceived better than textual one. //Diagrams in visual method help participants to model the system and help in identifying threats and security controls// because they give an overview of the possible threats, the threat scenarios and the assets, while the identification of threats in textual method is not facilitated by the use of tables because it is difficult to keep the link between assets and threats and the process 
 +is unclear. Also, lower perception of textual method can be explained by a poor worked example illustrating method application,​ and the unavailability of the software that would help to generate a bulk of tables. 
 + 
 + 
 + 
 + 
 + 
 + 
 + 
 +===== Additional Material ===== 
 +  * For additional information on the experimental design please see the {{:​research_activities:​experiments:​2014-seceng:​experiment-description.pdf|Experimental Protocol}}. 
 +  ​* ​For privacy reasons, at the beginning of the experiment a  {{:research_activities:​experiments:​2014-seceng:​consent-form-security-engineering.pdf|Consent Form }} was administered to participants. 
 +  * To protect the confidentiality of the training materials provided, participants were asked to sign a {{:​research_activities:​experiments:​2014-seceng:2014-trento-securityengineering-nda.docx|Non-Disclosure Agreement 
 + }}. 
 +  * Participants'​ results have been assessed by methods and domain experts ({{:​research_activities:​experiments:​2014-seceng:​evaluation_score_template.xlsx|Evaluation Score Sheet}}) 
 + 
 + 
 + 
 +**Data collected during the experiment are available upon request.**
seceng-course-exp-2013.1395750905.txt.gz · Last modified: 2021/01/29 10:58 (external edit)