EMFASE Internal Activities

This wiki page describes the main decision points of the EMFASE project.

Emfase conf call – 07/11/2014 Participants: UNITN: FP, MDG, FM SINTEF: BS, KS DBL: MR

Points to be discussed:

1. Discussion for comprehension questionnaire
2. Proposal of DBL
3. Contribution to D2.1

Main hypothesis:

• There is no difference between the comprehension levels of graphical and textual risk models

Sub-hypothesis:

• Visual graphs are better than tables for understanding how separate paths/rows are related
• Visual graphs are better than tables for calculation and consistency checking
• Tables are better than visual graphs filtering out specific information

Discussion about the definition of graphical model

• Nodes and edges definition is not exhaustive; a working definition which specifies which are the elements of the graphical models is needed. We need to understand which are the features of the graphic model, in order to identify if these features work or not.
• The scenario identified in Oslo was Poste Italiane Home Banking.

• MR explains how they defined criteria (according to MEM), sub-categories (according to success criteria), categories (as defined in Oslo) and indicators (defined by DBL)that can be measured and direct observed.
• A brief description of the experimental protocol is needed: how are you going to do the direct observation? How are you going to measure what you are observing? Consider that only partial observability will be available, how are you going to solve this problem?

• FP: share paper about comprehensibility by 1th July
• UNITN, SINTEF: provide semi-final Comprehension Questionnaire by 8th August
• DBL: provide a short description on the added-value for the project and for the participants and the protocol for the observation by End of July
• UNITN: provide 1 page with protocol for Section 4 of D2.1

Some interesting article for the experiments Zip files

• UNITN: FP, FM
• SINTEF: BS, KS, AR
• DBL: AT, MR

Everybody should submit their slides in the SVN.

Day 1:

• FP (Slides): Empirical results of the experiments (9:00-11:30) Long discussion of clarification of which are actual statistical significant results and relation with the MEM model
• BS (slides): Framework presentation: discussion on the framework (see the slides) (13:30-15:00)
• AT (slides): Description of experiments and past experiments (15:30-17:00)

Day 2:

• AT: Discussion of future experiments (9:00-10:00)
• FM: Discussion of Framework Drivers (10:00-11:30)
• FM+KS: Discussion of Revised Model (13:00-14:30)
• FM (excel file+powerpoint): Revised Experiment Plan for Comprhensibility Driver(14:30-15:30). Comprehensibility is not really well covered
• BS (draft ToC with schedule and relevant): Discussion of D1.2 ToC

Timeline

• June 27 ToC – sent to RK
• July 4 - Skeleton for SINTEF's chapters
• August 1 - Input by DBL+UNITN chapters
• August 15 - First version of deliverable – sent to RK
• August 21 - review by FM of deliverable
• August 30 - Final version to EC to sent to RK

Hypothesis to be tested is Comprehensibility Tentative experiments

• ModelEng (Oct) 2014 – UiO’s course
• SecEng (Oct) 2014 – UNITN’s course
• DBL as test case (3-4)

Case study

• Poste Italiane (CORAS vs SESAR already done)

Schedule

• July 8 – First draft of “Actual” questions
• July 11 – Conf. call to agree on final draft
• End August – First draft of Table + Graph + Perception Questions
• End Septembet – Final version

Proposal is to perform the observation of the SESAR assessment

• July 8 – First draft of observation protocol
• July 11 – Conf. call to agree on final draft

Decisions

FM contacted RK who said that he will talk to IANS tutorial manager to see whether he will agree.

Oct 2014 – Jan 2015 - Interviews with Security Experts (Raminder Ruprai - NGRID, Ivonne Herrera – SINTEF, Birgit Goelz – DFS, Lorenzo Falciani – PwC, Gianluca Gargiulo - NAIS Solutions, etc.)

• UNITN: FP, FM, EC
• SINTEF: BS
• DBL: AT
• EUROCONTROL: RC (only 11-12)

D1.1 (State of the Art). No major issue internally.

RK's Comment: there is a presentation problem: we discuss the existence of the standards but don't discuss whether they complete or not and wat is missing and why tey are different, section 3 describe the criteria, and 4 are well done but the end of chapter 4 explain about verification and we instantiate immediately instantiated to an empirical verification of a subset, why the oter ave not been selected. The revision should address the one to assess of what is missing in the state of the art, how the criteria linked back to SOA or how they help us selecting the case studies).

D2.1 (Case study) The case study deliverable is essentially rephrased from SESAR deliverables. Question of access has been raised with RK (see later).

RK's comment: First 3 chapters ok, with section 4 have some yellow markers, conclusions are empty. migt be a problem of right version of the document.

D1.1 RK to send comment so far. BS to address them and We send it back by Friday 28 March.

D2.1. EC to send back the right version immediately and wait for feedback

RK comment: a SJU Memo is in preparation for granting access to SESAR documents (at least without IPR claims as foreground). We will foll tat procedure when it is ready.

In the cases where official access is not possible we will use a disclaimer about expert opinion and use an agreed procedure to collect confidential material.UNITN has already used it for the SECRAM list of pre- and post- controls at the WINTER experiment.

Disclaimer to be used in case study descriptions:

The case study description is based from interviews of experts familiar the Remote Virtual Tower development at SESAR. Its content should not be taken in any way as officially endorsed by SESAR SJU or SESAR's participants

The procedure for handling confidential material for the experiments is the following:

1. Ask for a confidentiality agreement to be signed by the subjects
2. watermark their copy of the document that is only given in physical form
3. collect back the copy at the end of the experiment
4. archive the signed document and its personal copy

In part (*) means confidential document is distributed according to the process above.

1. Experiments during UNITN Course:
   • Participants: students around 60 sort of controlled participants
   • Method: Coras vs Eurocontrol SECRAM (*)
   • Case Study: SmartGrid
   • Final result: excel file with threats and controls, presentations, report
   • Feedback: questionnaire, interview
2. Experiments during EIT Winter School:
   • Participants: students around 20 sort of controlled participants
   • Method: SESAR SecRAM (*) + [ BSI Catalog vs SECRAM Catalog (*) ]
   • Case Study: Remotely Operated Tower (*)
   • Final result: excel file with requirements, hand-drawn poster for result presentation, report
   • Feedback: questionnaire

RK reported a good feedback from Martin Hawley.

Only three groups produced a report in te SESAR format. The other have not done it. We will only evaluate the excel file.

FP to save the excel file produced by the students in the repository and then share the google doc wit Hans De Han, Martin Hawley, AT and BS. Evaluation to be done by them. Internal deadline for feedback to be aligned wit Hans availability.

The next experiment is Rome

1. Poste Italiane Experiment (13-14 in Rome in Tor Vergata University):
   • Participants: large scale students+practitioners around 100 too large to distribute confidential documents
   • Method: SESAR SecRAM (*) + CORAS
   • Process: step-wise (training+execution)
   • Case Study: Credit card security at Poste
   • Final result: excel file with threats and controls for SECRAM, powerpoint with details for threats and controls for CORAS plus final table summary with threats and controls
   • Feedback: questionnaire
   • Eurocontrol input: ex-ante validation of the training material
2. DBL Experiment (15-16 May in Rome location to be determined)
   • Participants: DBL people (12) + other “constellation” of companies in the ATM area
   • Method: SESAR SecRAM (*) + [ BSI Catalog vs SECRAM Catalog (*) ]
   • Process: step-wise (training+execution)
   • Case Study: Remotely Operated Tower
   • Final result: excel file with threats and controls,
   • Feedback: questionnaire + feedback gathering (interviews or post-it notes) (?)
   • Eurocontrol input: Rainer Koeller as official Trainers about SESAR SecRAM

MDG will prepare a draft description of these experiments (who, what, when) to be circulated and posted on the SVN (one directory per experiment).

BS to prepare a draft of the CORAS Material to follow te step by step process. FP to prepare draft of the SeCRAM material from Martin Hawley's material from Winter Experiment

EC to set a conference call one for organizing the May experiment

Presentation at SESAR Jamborre (20-22 May) of results to EMFASE activities. RK suggests a slot on Monday.

1. little show of the students who participated in the winter experiment,
2. presentation of success criterias of previous Jamboree's interview
3. presentation of preliminary results of winter experiment

RK suggest to prepare a White paper to be distributed in advance.

• AT to extract the D1.1 summary of results of interviews (2-3)
• FP to write a 1 page highlight of winter experiments
• BS to finalize the whole stuf

Deadline end of April.