An Experimental Comparison of Two Risk-Based Security Methods

An experiment by Katsyarina Labunets, Fabio Massacci, Federica Paci, Le Minh Sang Tran.

This page provides additional resources that enable replication of our work published at ESEM 2013. See the main page for our work on empirical validation of security risk assessment methods and other experiments.

The goal of the experiment was to evaluate and compare two types of risk-driven methods, namely, visual methods (CORAS) and textual methods (SREP) with respect to their effectiveness in identifying threats and security requirements, and the participants’ perception of the two methods.

The experiment involved 28 participants: 16 students of the master in Computer Science and 12 students of the EIT ICT LAB master in Security and Privacy. They were divided into 16 groups using a randomized block design.

The methods evaluated were CORAS (visual method) and SREP (textual method).

The participants applied the methods to a Smart Grid application scenario.

The experiment was conducted as part of the Security Engineering course. Here, you can find the summary of the Tasks to be accomplished in the experiment.

• Background Questionnaire - collect participants demographic data.
• Post-Task Questionnaire - assess participants’ perception of visual and textual methods.
• Interview Guide - collect participants' opinion on advantages and disadvantages of visual and textual methods.
• Final Report - document methods' application.

• Methods' effectiveness

Results show that visual method is more effective in identifying threats than textual method. This is confirmed if we consider the number of threats identified with visual and textual methods across the task assigned to the groups. Instead, with respect to number of security requirements, textual method is slightly more effective than the visual one in identifying security requirements.

• Methods' perception

Participants’ overall preference is higher for visual than for textual method, while regarding to the perceived ease of use and the usefulness no statistically significant difference is proven by the experiment. Moreover, in respect to the intention to use, the difference in participants’ perception is statistically significant in favour of the visual method.

• Qualitative Explanation

The different number of threats and security requirements identified can be likely explained by the differences between the two methods indicated by the participants during the interviews. Diagrams in visual method help brainstorming on the threats, giving an overview of the possible threats, the threat scenarios and the assets, while the identification of threats in textual method is not facilitated by the use of tables as it is more difficult to link assets and threats. As suggested by the participants then, the identification of threats in textual method could be made easier if a catalog of common threats was available. On the other side, textual method is slightly more effective in eliciting security requirements than visual approach because the order of steps in textual method process guides the analyst, while the same it seems not to hold for the visual method’s process.

• For additional information on the experimental design please see the Experimental Protocol.
• For privacy reasons, at the beginning of the experiment a Consent Form was administered to participants.
• Participants' results have been assessed by methods and domain experts (see Evaluation Score Sheet).

Data collected during the experiment are available upon request.