The eRISE 2012 challenge was conducted to empirically evaluate security engineering and risk analysis methods. The event was carried out in May and June 2012. The first part of experiment took place at the University of Trento, Italy, the second at Dauphine University, Paris, France.

To have a glimpse of eRISE 2012 you can look a video on YouTube eRISE 2012. See the main page for our work on empirical validation of security risk assessment methods and other experiments.

In eRISE 2012 were involved the following participants:

• Customers
  • Marina Egea (Atos Research);
  • Jorge Cuellar (Siemens AG);
• Method Designers:
  • Le Minh Sang Tran - SINTEF/University of Trento (CORAS)
  • Kim Wuyts, Riccardo Scandariato - Katholieke Universiteit Leuven (LINDDUN)
  • Thein Than Tun - Open University (Security Argumentation)
  • Michalis Pavlidis - University of East London (Secure Tropos)
  • Daniel G. Mellado - University of Castilla La Mancha (SREP)
• Observers:
  • Sarila Rana
  • Martina Degramatica
  • Deepa Nagaraj
  • Elda Paja
  • Jennifer Horkoff
• Participants:
  • 15 students were enrolled in the Master in Computer Science at the University of Trento and had a background in Security Engineering and Information Systems
  • 27 professionals were attending a Master Course in Audit for Information System in Enterprises at Dauphine University. This master has an admission requirement of a minimum of five years of working experience in the field of Auditing in Information Systems

The selection of the security requirements methods to be evaluated was driven by three main factors: the number of citations, the fact that research on the method is still ongoing, and availability of the methods designers.

Five methods have been evaluated and compared during eRISE 2012:

• CORAS is a model-driven method for risk analysis proposed by SINTEF, Norway. Materials: book chapter, tutorial.
• LINDDUN is a methodology to elicit the privacy requirements of software-intensive systems and select privacy enhancing technologies designed by Distrinet Research Group at Katholieke Universiteit Leuven, Belgium. Materials: paper, case study, tutorial.
• SECURITY ARGUMENTATION is a framework for security requirements elicitation and analysis developed at Open University, Buckinghamshire, United Kingdom. Materials: paper, tutorial.
• SECURE TROPOS is a methodology designed at University of East London, United Kingdom; the methodology supports capturing, analysis and reasoning of security requirements from the early stages of the development process. Materials: paper, paper, tutorial.
• SREP is an asset-based and risk-driven method developed at University of Castilla-La Mancha, Spain for the establishment of security requirements in the development of secure Information Systems. Materials: paper, case study, tutorial.

In eRISE 2012 two industrial application scenarios were proposed to the participant for analysis.

The Health Care scenario was proposed by Siemens. This scenario was related to the management of electronic healthcare records. The scenario focused on registering new patients in a clinic including assigning the clinicians (doctors, nurses, etc.), reading and updating a record, retrieving patient information from external sources, and providing the results of examinations and treatment to authorized externals clinical entities.

The materials about this scenario are available online: scenario description and presentation.

The Smart Grids scenario was proposed by Atos Research. Smart Grid is an electricity network using information and communication technology (ICT) to optimize the transmission and distribution of electricity from suppliers to consumers. In particular, the scenario was focused on the smart meter which records consumption of electric energy and communicates this information daily back to the utility for monitoring and billing purposes.

The materials about this scenario are available online: scenario description and presentation.

eRISE 2012 was conducted in three main phases:

• Training Phase where participants attended tutorials on the methods under evaluation and on the eHealth and Smart Grid industrial cases:
  1. May 7-9, 2012 at the University of Trento, Italy
• Application Phases, where participants applied the methods to analyse security issues of the eHealth and Smart Grid industrial cases:
  1. May 10-11, 2012 at the University of Trento, Italy
  2. June 14-15, 2012 at Dauphine University, Paris, France
• Evaluation Phase, where participants evaluated the methods through focused group interviews and post-it notes sessions while method designers and customers evaluated the final reports. The goal is to assess the correctness of the methods application and the quality of the security requirements identified by the participants.
  1. June 15, 2012 Focus Groups and Post-it notes sessions with participants, at Dauphine University, Paris, France
  2. June 30- July 15, 2012 Reports Assessment by method designers and customers

We have collected different kinds of data:

• Questionnaires include questions on subjects' knowledge of IT security, risk assessment, and requirements engineering and their evaluation of the methods' aspects. The participants were administered five questionnaires during the execution of the eRISE 2012:
  • Q1 was administered at the beginning of the Training phase to collect participants' background (Q1);
  • Q2 was distributed at the end of the Training phase (Q2);
  • Q3 and Q4 were administered during the two Application phases (Q3 and Q4);
  • Q5 was administered at the end of the Application phase to compare the method applied by the participants with other methods they may already knew (Q5);
• Audio/Video Recordings* capture the application of the methods by subjects and the focus groups interviews;
• Post-it Notes* list positive and negative aspects about the methods and the study itself;
• Focus Group Transcripts* report the discussion on the methods'application between participants and observers
• Group Presentations* by participants summarize the results of method's application;
• Final Reports* describe in detail how participants have identified the security requirements following the method.

* These materials are available upon e-mail request.

Questionnaires have been analyzed using statistical analysis. For post-it notes we have used affinity analysis in order to group similar feedback on positive and negative aspects of the methods. The transcripts of the focus groups discussions have been analyzed using coding, a content analysis technique used in grounded theory. Coding helped us to discover text patterns that are relevant to what makes methods effective in identifying security requirements and why. We have performed a qualitative analysis of the final reports.