This page provides additional resources that enable replication of our experiment with MSc students. See the main page for our work on empirical validation of security risk assessment methods and other experiments.

The goal of our study is to investigate the discrepancy between CVSS Environmental metrics for environmental vulnerability assessment and their dificulty in practice. We find that vulnerability assessment using CVSS Environmental metrics do not scale well with complexity in spite of the fact that we only considered security requirements and let alone any technical configuration. Specifically, assessments in a semgmented network scenario are characterized by significantly higher error rate than assessments performed in a flat network scenario.

The experiment was conducted at the University of Trento in November 2016 as part of the Cyber Security Risk Assessment course. The participants were 29 MSc students in Computer Science. The experiment took place in a single computer laboratory. The experiment was presented as a laboratory activity and only the high-level goal of the experiment was mentioned. The experimental hypotheses were not revealed so as not to influence the participants, but they were informed about the procedure.

To test the effectiveness of the CVSS guidance we considered two scenarios (flat and segmented networks) and how their environmental metrics should change after security metrics are deployed. In our study we used two scenarios described in the “PCI Compliance: Understand and Implement Effective PCI Data Security Standard Compliance” book. First we provided partecipants with the flat network scenario then the segmented where the critical appliances are segregated from public parts of the network.

Here are the materials that our participants received:

• Flat Network Description PDF.
• Segmented Network Description PDF.