An Experiment on Comparing Textual vs Visual Industrial Methods for Security Risk Assessment

An experiment by Katsyarina Labunets, Fabio Massacci, Federica Paci.

This page provides additional resources that enable replication of our work published at EmpiRE2014. See the main page for our work on empirical validation of security risk assessment methods and other experiments.

The goal of the experiment was to compare the actual effectiveness, and perception of visual (CORAS) and textual (EUROCONTROL SecRAM) methods for security risk assessment with respect to their effectiveness in identifying threats and security requirements, and the participants’ perception of the two methods.

We involved 29 participants: 15 students of the MSc in Computer Science and 14 students of the EIT ICT LAB MSc in Security and Privacy of the University of Trento. In this within-subject design, each participant applied both methods.

The methods evaluated were CORAS (visual method) and EUROCONTROL SecRAM (textual method).

The participants applied the methods to a Smart Grid application scenario.

• Background Questionnaire - collect participants demographic data.
• Post-Task Questionnaire - assess participants’ perception of visual and textual methods.
• Interview Guide - collect participants' opinion on advantages and disadvantages of visual and textual methods.
• Final Report - document methods' application.

• Methods' effectiveness

Results show that there is no difference in the number of threats and controls identified with CORAS and EUROCONTROL SecRAM, differently from the results we achieved in the previous experiment where the visual method (CORAS) performed better in threats identification rather than the textual method (SREP). This difference may be due to the change of the textual method: SecRAM could perform better than SREP, or due to the difference in the experimental design. In the first experiments indeed participants applied each method twice, while in the present experiment there was only one application of the method. The participants of the first experiment might have learnt methods better and produced significant results.

• Methods perception

Participants’ overall perception is higher for visual than for textual method with statistical significance for all participants. The same result holds for the perceived usefulness and the intention to use; however, regarding the perceived ease of use no statistically significant difference is proven by the experiment. Similar results were found in the first experiment.

• Qualitative Explanation

Qualitative explanations from the interviews illustrate that the visual method is perceived better than textual one. Diagrams in visual method help participants to model the system and help in identifying threats and security controls because they give an overview of the possible threats, the threat scenarios and the assets, while the identification of threats in textual method is not facilitated by the use of tables because it is difficult to keep the link between assets and threats and the process is unclear. Also, lower perception of textual method can be explained by a poor worked example illustrating method application, and the unavailability of the software that would help to generate a bulk of tables.

• For additional information on the experimental design please see the Experimental Protocol.
• For privacy reasons, at the beginning of the experiment a Consent Form was administered to participants.
• To protect the confidentiality of the training materials provided, participants were asked to sign a Non-Disclosure Agreement .
• Participants' results have been assessed by methods and domain experts (Evaluation Score Sheet).

Data collected during the experiment are available upon request.