User Tools
tutorial
Differences
This shows you the differences between two versions of the page.
| Both sides previous revision Previous revision Next revision | Previous revision | ||
|
tutorial [2014/04/09 13:50] federica.paci@unitn.it |
tutorial [2021/01/29 10:58] (current) |
||
|---|---|---|---|
| Line 1: | Line 1: | ||
| ====== How to Empirically Evaluate your pet (Security but not only) Requirements Engineering Method ====== | ====== How to Empirically Evaluate your pet (Security but not only) Requirements Engineering Method ====== | ||
| + | Running empirical experiments is not so simple. It takes months to prepare, conduct and analyze experiments properly. In this tutorial we will provide hands-on insights on how to run an empirical experiments for comparing RE methods based on our experience gained over the past 4 years in evaluating and comparing security requirements engineering methods during the E-RISE challenge. | ||
| + | |||
| + | In the first part of the tutorial, we will introduce a general experimental protocol to perform empirical evaluations of (security) RE methods. Then, we will illustrate it with examples of solutions that worked out either in our controlled experiments and in other experiments from the key literature on evaluating RE methods. We will also share lessons learned during the execution of our experiments, as there are many pitfalls and traps and they are simply not written in research papers reporting empirical experiments. | ||
| + | |||
| + | In the second part, in order to show participants how the guidelines can be put into practice, we will engage them in a short experiment to evaluate the effectiveness of a security requirement elicitation method. At the end of the experiment we will look back at the experiment just run and reflect with the audience on what really happened behind the scenes and how different choices could have lead to different experiments or outright failure. | ||
| + | |||
| + | ===Important Links==== | ||
| + | Please visit [[http://webhotel.bth.se/re14/pages/conference/tutorials/#T11|our page]] on the RE'14 official website. You can also download our [[http://webhotel.bth.se/re14/pdf/RE14tutorialflyerT11A4.pdf|flyer]].\\ | ||
| + | [[http://webhotel.bth.se/re14/pages/conference/registration/|Early-bird registration is open!]] \\ | ||
| + | Watch our video about previous experiments: [[http://youtu.be/hyxg56RTQsw|eRISE 2012]] | ||
| ====Relevance for the RE Conference and Intended Audience of the Tutorial==== | ====Relevance for the RE Conference and Intended Audience of the Tutorial==== | ||
| Line 11: | Line 21: | ||
| Over the last four years, we have run the E-RISE challenge (http://securitylab.disi.unitn.it/doku.php?id=erise), a series of empirical experiments in which different researchers working in security requirements engineering across the world have seen their methodology applied by a number of students and practitioners on a number of case studies provided by industry. | Over the last four years, we have run the E-RISE challenge (http://securitylab.disi.unitn.it/doku.php?id=erise), a series of empirical experiments in which different researchers working in security requirements engineering across the world have seen their methodology applied by a number of students and practitioners on a number of case studies provided by industry. | ||
| - | What we will illustrate in the tutorial is a distilled, concrete protocol that try to validate whether \emph{a method works in practice}, i.e. whether it can be used | + | What we will illustrate in the tutorial is a distilled, concrete protocol that try to validate whether //a method works in practice//, i.e. whether it can be used |
| * //effectively// and | * //effectively// and | ||
| * //efficiently// | * //efficiently// | ||
| Line 22: | Line 32: | ||
| This tutorial should then be of interest of graduate students, academic and industrial researchers who want to learn how to conduct controlled experiments to evaluate their favorite RE methods and techniques. | This tutorial should then be of interest of graduate students, academic and industrial researchers who want to learn how to conduct controlled experiments to evaluate their favorite RE methods and techniques. | ||
| - | \section{Structure of the Tutorial} | + | ====Structure of the Tutorial==== |
| The tutorial will be divided into four parts: | The tutorial will be divided into four parts: | ||
| - | \begin{enumerate} | + | - **Experimental Protocol** (2 hrs) We will introduce you a general experimental protocol that we developed and refined during a series of controlled experiments. The protocol is specifically targeted to assess the performance of elicitation methods. We will discuss its organization along |
| - | \item {\bf Experimental Protocol} (2 hrs) We will introduce you a general experimental protocol that we developed and refined during a series of controlled experiments. The protocol is specifically targeted to assess the performance of elicitation methods. We will discuss its organization along | + | * the temporal development (training, execution, evaluation) |
| - | \begin{itemize} | + | * the conceptual dimension (execution of the experiment, and measurements of the results) |
| - | \item the temporal development (training, execution, evaluation) | + | * the data dimensions (quantitative vs qualitative feedback, perceived vs actual results) |
| - | \item the conceptual dimension (execution of the experiment, and measurements of the results) | + | * the analysis dimensions (some simple statistical tests). |
| - | \item the data dimensions (quantitative vs qualitative feedback, perceived vs actual results) | + | - **Experimental Instantiation** (1.5 hrs) We will discuss at this stage how each step can be instantiated in order to achieve your desired results with a series of examples (a semester long, a two weeks long, a two days long and a two hours long experiments). |
| - | \item the analysis dimensions (some simple statistical tests). | + | - **Experiment** (2,5 hrs). We will show participants how guidelines can be put into practice. We will involve them into an experiment to evaluate the effectiveness of a security requirementselicitation method. |
| - | \end{itemize} | + | - **Reflection** (1 hr) After the practical exercise, we will dissect the experiment to explain and discuss |
| - | \item {\bf Experimental Instantiation} (1.5 hrs) We will discuss at this stage how each step can be instantiated in order to achieve your desired results with a series of examples (a semester long, a two weeks long, a two days long and a two hours long experiments). | + | * how it complies with presented theoretical aspects |
| + | * some of the tricky issues behind its organization | ||
| + | * what seemed work but didn't | ||
| + | * the issues raised during the execution and | ||
| + | * what was really necessary during the preparation. | ||
| - | \item {\bf Experiment} (2,5 hrs). We will show participants how guidelines can be put into practice. We will involve them into an experiment to evaluate the effectiveness of a security requirements | ||
| - | elicitation method. | ||
| - | \item {\bf Reflection} (1 hr) After the practical exercise, we will dissect the experiment to explain and discuss | ||
| - | \begin{itemize} | ||
| - | \item how it complies with presented theoretical aspects | ||
| - | \item some of the tricky issues behind its organization | ||
| - | \item what seemed work but didn't | ||
| - | \item the issues raised during the execution and | ||
| - | \item what was really necessary during the preparation. | ||
| - | \end{itemize} | ||
| - | \end{enumerate} | ||
tutorial.1397044241.txt.gz · Last modified: 2021/01/29 10:58 (external edit)
Page Tools
