The Role of Catalogues of Threats and Security Controls in Security Risk Assessment: An Empirical Study with ATM Professionals

This page provides additional resources that enable replication of our work published at REFSQ2015. See the main page for our work on empirical validation of security risk assessment methods and other experiments.

The goal of the experiment is to compare the effect of using domain-general versus domain-specific catalogs of threats and security controls on security risk assessment's actual effectiveness and perception.

The participants of the experiment were 15 professionals from severalATMItalian companies. The participants worked individually. The participants were divided into three groups and assigned to three different treatments. They were asked to apply individually the same method, namely SESAR SecRAM, with the support of domain-specific catalogues (EUROCONTROL ATM), domain-general catalogues (BSI IT-Grundschutz) or without any catalogues.

To conduct security risk assessment the groups used SESAR ATM Security Risk Assessment Method (SecRAM). As an instance of domain-specific catalogues we used the security catalogue developed by EUROCONTROL which come with SecRAM method. As an instance of domain-general catalogues we chose the threats and security controls catalogues of the BSI IT-Grundschutz standard.

The materials related to SecRAM method and EUROCONTROL EATM catalogues are confidential.

As application scenario to be used by the participants, we chose a new operational concept which is emerging in the ATM named Remotely Operated Tower (ROT).

The experiment was conducted as part of the SESAR SecRAM workshop organized by EMFASE project. The participants were given a tutorial on the method of the duration of 8 hours spanned over 2 days. The tutorial was divided into different parts. Each part consisted of 30 minutes of introduction of a couple of steps of the method, followed by 45 minutes of application of the steps and 15 minutes of presentation and discussion of the results with the expert. After the application phase participants delivered their final reports documenting the conducted security risk assessment of the ROT.

During the experiment we distributed among participants two type of questionnaires:

• Pre-task questionnaire to collect some information about participants and their background:Background Questionnaire.
• Post-task questionnaire to collect participants' perception of the method and/or catalogues: versions for domain-specific catalogues, domain-general catalogues, and method only.

Data collected during the experiment are available upon request.