The eRISE (engineering RIsks and SEcurity Requirements) challenge is a series of empirical studies that aim to compare security engineering methods sponsored by NESSoS European Project and EIT ICT Labs. Three editions of eRISE challenge has been held eRISE 2011, eRISE 2012, and eRISE 2013. See the main page for our work on empirical validation of security risk assessment methods and other experiments.

The idea of eRISE challenge is to bring together researchers, young students and practitioners to understand if security methods are effective and what features determine their effectiveness.

With eRISE we want to be able to tell whether “it is not a method to find security recommendations..”, or at least “it helps us to represent the model but does not help in finding solution”, or hopefully “it helps to find out specific security requirement.” (quoting some of the participants of our experiments).

eRISE provides method designer with:

• Empirical evaluation and benchmarking of security engineering methods;
• Understand if the proposed method works in practice
• Feedback to improve a security method by investigating its strengths, weakness and limitations.

eRISE aims to provide the participants with the benefit of:

• Knowledge about various state-of-the art methods in the research field, on analyzing security risks and requirements of a system;
• Opportunity to participate and work in international collaborative project.

• RQ1 Are security requirements and risk methods effective when applied by someone different than their own inventor?

• RQ2 Why are the methods effective? Why they are not?

A method is effective when it assists the analyst to produce high quality security requirements with less time and less effort.

Since our research questions are exploratory in nature, we applied a mix-method experimental methodology combining both qualitative and quantitative data collection and analysis techniques. We evaluate methods' effectiveness based on the reports delivered by the participants, while we investigate the whys methods are effective by means of questionnaires, focus group interviews and post-it notes (RQ2).

One of our goals is to investigate whether the methods under evaluation could be used effectively by users who have no prior knowledge of the methods. Therefore we have designed a protocol to conduct comparative empirical studies in this setting. The protocol consists of three main phases:

• Training. First, participants are administered a questionnaire to collect information about their level of expertise in requirement engineering, security and on other methods they may know. Then, they are divided in groups where each group is composed of one master students and two professionals. The groups are assigned to a security requirements or a risk analysis method and to an industrial case to be analyzed using the method. The participants have to attend lectures about the method and on the industrial application scenario. At the end of the Training phase, the participants are administered a questionnaire to determine their level of understanding of the methods and of the industrial applications scenario.

• Application. Participants work in groups and apply the method to analyze the application scenario.Group collaboration takes place both face-to-face and remotely by using multiple communication channels (e.g. mail, chat, video conferencing facilities). At the end of this phase, participants are involved in focus groups interviews, and they are requested to fill in post-it notes and a questionnaire about their impressions on the method. To document the application of the methods, the groups are audio-video recorded. In addition, groups have to deliver a final report.

• Evaluation. On one side, participants assess the methods' effectiveness: they are involved in focus groups interviews, and they are requested to fill in post-it notes and a questionnaire about their impressions on the method. On the other side, method designers assess if the participants have a followed the method while customers, instead, evaluate if the groups have identified a set of security requirements or countermeasures that are specific for the application scenario, and if they are able to justify their results based on the method's application.

The eRISE Experimental Protocol involves five types of actors:

1. Method Designer is the researcher who has proposed one of the method under evaluation. His main responsibility is to train participants in the method and to answer participants' questions during the Application phase. S/he also contributes to the assessment of the methods'effectiveness by analyzing groups' reports.
2. Customer is an industrial partner who introduces the industrial application scenario to the participants. S/he also has to be available during the Application phase to answers all possible questions that participants may raise during analysis.
3. Observer plays an important role during the Application phase because they supplement audio-video recording with information about the behavior of participants e.g (if the Participants work in group vs work alone) and the difficulties that they face during the application of the method. The observer also interviews the groups and leads the post-it notes sessions.
4. Researcher takes care of the organization, sets the research questions, selects the participants, invites the method designers and the customers, and analyzes the data collected during the study.
5. Participant is the most important role. Participants work in group and apply a method provided by one of the method designers to analyze the risk and security issues of the scenario provided by the customer.