An experiment on the effect of using a catalog of security requirements on elicitation effectiveness

The goal of the experiment we want to conduct following the goal/question/metric (GQM) template is investigating the use of a catalog of security requirements with the purpose of understanding if the use of a catalog has an effect on the effectiveness of eliciting security requirements from the point of view of security requirements engineers. The context of the experiment consists of security requirements engineers who identify a list of security requirements for an advanced metering infrastructure scenario from the Smart Grid domain.

The main research questions of the experiment are the following:

• RQ1: Does the use of a catalog of security requirements lead to identify more security requirements?

• RQ2: Is participants' perception of using a catalog of security requirements in security requirements elicitation better than not using it?

RQ1 concerns the effectiveness of security requirements elicitation. To answer the research question we will measure effectiveness by counting the number of security requirements identified by the participants. The data analysis for RQ1 will be done using the Mann-Whitney test. Only security requirements specific for the scenario analyzed by the participants will be considered for statistical analysis.

RQ2 is related to the participants' perception of using the catalog of security requirements in eliciting security requirements. Participants' perception will be measured through a post-task questionnaire inspired to the Technology Acceptance Model (TAM). The data analysis for RQ2 will be done using the Mann-Whitney test.

The null hypotheses coming from the research questions are the following:

• H1: There will be no difference in the number of security requirements found with a catalog of security requirements and the one found without.

• H2: There will be no difference in the participants' perception of eliciting security requirement with a catalog of security requirements and without.

Participants to our experiment should have a background in security and at least two years experience in security requirements elicitation. A between-subject design will be used where the participants will be randomly assigned to two groups denoted as G1 and G2. The participants in G1 will identify security requirements for an advanced metering infrastructure scenario with the use of catalog of security requirements. The participants in G2 will have to identify security requirements for the same scenario but without the support of the catalog. The scenario focuses on a private household where a smart meter is installed which records consumption of electric energy and communicates this information daily back to the utility for monitoring and billing purposes.

• Training. All participants are provided with a four pages tutorial on the advanced metering infrastructure scenario and are administered a demographic questionnaire about their education level, security requirements engineering experience, and industry experience.
• Application. All participants are provided with a table listing assets and possible threats for the advanced metering infrastructure scenario. Participants in group G1 receive a printed copy of the SANS Institute Catalog of Critical Security Controls for Cyber Defence. Each participant in group G1 fill in the table with security requirements protecting the assets from identified threats with the support of the SANS Institute Catalog of Critical Security Controls for Cyber Defence. Each participants in group G2 perform the same task without the catalog support. All participants perform the work individually.
• Evaluation. Participants fill in the post-task questionnaire about their perception of the use of a security requirements catalog.

We will allocate 20 minutes for the Training phase, 50 minutes for the Application phase, and 10 minutes for the Evaluation phase. The experiment should be conducted in a closed room with no Internet connection to not disturb the participants. Printing facilities for the researchers conducting the experiment should be provided at the conference.