The goal of the experiment we want to conduct following the goal/question/metric (GQM) template is investigating the use of a catalog of security requirements with the purpose of understanding if the use of a catalog has an effect on the effectiveness of eliciting security requirements from the point of view of security requirements engineers. The context of the experiment consists of security requirements engineers who identify a list of security requirements for an advanced metering infrastructure scenario from the Smart Grid domain.
The main research questions of the experiment are the following:
RQ1 concerns the effectiveness of security requirements elicitation. To answer the research question we will measure effectiveness by counting the number of security requirements identified by the participants. The data analysis for RQ1 will be done using the Mann-Whitney test. Only security requirements specific for the scenario analyzed by the participants will be considered for statistical analysis.
RQ2 is related to the participants' perception of using the catalog of security requirements in eliciting security requirements. Participants' perception will be measured through a post-task questionnaire inspired to the Technology Acceptance Model (TAM). The data analysis for RQ2 will be done using the Mann-Whitney test.
The null hypotheses coming from the research questions are the following:
Participants to our experiment should have a background in security and at least two years experience in security requirements elicitation. A between-subject design will be used where the participants will be randomly assigned to two groups denoted as G1 and G2. The participants in G1 will identify security requirements for an advanced metering infrastructure scenario with the use of catalog of security requirements. The participants in G2 will have to identify security requirements for the same scenario but without the support of the catalog. The scenario focuses on a private household where a smart meter is installed which records consumption of electric energy and communicates this information daily back to the utility for monitoring and billing purposes.
We will allocate 20 minutes for the Training phase, 50 minutes for the Application phase, and 10 minutes for the Evaluation phase. The experiment should be conducted in a closed room with no Internet connection to not disturb the participants. Printing facilities for the researchers conducting the experiment should be provided at the conference.