User Tools

Site Tools


seceng-course-exp-2013

This is an old revision of the document!


An Experiment on Comparing Textual vs Visual Industrial Methods for Security Risk Assessment

An experiment by Katsyarina Labunets, Fabio Massacci, Federica Paci.

This page provides additional resources that enable replication of our work published at EMPIRE2014.

Goals

The goal of the experiment was to compare the actual effectiveness, and perception of visual (CORAS) and textual (EUROCONTROL SecRAM) methods for security risk assessment with respect to their effectiveness in identifying threats and security requirements, and the participants’ perception of the two methods.

Context of the Experiment

Subjects

We involved 29 participants: 15 students of the MSc in Computer Science and 14 students of the EIT ICT LAB MSc in Security and Privacy of the University of Trento. In this within-subject design, each participant applied both methods.

Methods

The methods evaluated were CORAS (visual method) and EUROCONTROL SecRAM (textual method).

Case study

The participants applied the methods to a Smart Grid application scenario.

Measurements

Results

  • Methods' effectiveness

Results show that there is no difference in the number of threats and controls identified with CORAS and EUROCONTROL SecRAM, differently from the results we achieved in the previous experiment where the visual method (CORAS) performed better in threats identification rather than the textual method (SREP). This difference may be due to the change of the textual method: SecRAM could perform better than SREP, or due to the difference in the experimental design. In the first experiments indeed participants applied each method twice, while in the present experiment there was only one application of the method. The participants of the first experiment might have learnt methods better and produced significant results.

  • Methods perception

Participants’ overall perception is higher for visual than for textual method with statistical significance for all participants. The same result holds for the perceived usefulness and the intention to use; however, regarding the perceived ease of use no statistically significant difference is proven by the experiment. Similar results were found in the first experiment.

  • Qualitative Explanation

Qualitative explanations from the interviews illustrate that the visual method is perceived better than textual one. Diagrams in visual method help participants to model the system and help in identifying threats and security controls because they give an overview of the possible threats, the threat scenarios and the assets, while the identification of threats in textual method is not facilitated by the use of tables because it is difficult to keep the link between assets and threats and the process is unclear. Also, lower perception of textual method can be explained by a poor worked example illustrating method application, and the unavailability of the software that would help to generate a bulk of tables.

Supplement Materials

During the experiment we distributed among participants two type of questionnaires:

  • Pre-task questionnaire to collect some information about participants and thier background: Q1.
  • Post-task questionnaire to collect participants' perception of methods: Q2.

For individual interview we used the following interview guide: guide.

seceng-course-exp-2013.1405516514.txt.gz · Last modified: 2021/01/29 10:58 (external edit)