Photo of Hanan Hibshi

Hanan Hibshi

Office: Wean Hall - WEH 4125
Phone: 412-268-4885
Email: hhibshi [at] cmu [dot] edu

Information for the RE16 Tutorial T03
Effective User Survey Design and Data Analysis

Software engineers find designing user surveys not as simple as compiling a list of questions that seem reasonable to the investigator. Survey design should leverage the wealth of theory that informs whether a proposed survey measures what is claimed to be measured. For example, engineering researchers studying methods often default to measuring the time of completion or the number of tasks achieved to evaluate a solution. Alternatively, the engineering problems are rich with human-subject-relevant phenomena that can advance our knowledge in requirements engineering. Furthermore, researchers in psychology and the social sciences have discovered foundational theories that can be used as guidelines to create experiments to access that knowledge.

In this tutorial, we introduce the audience to relevant social science theories and show how they can be applied in survey design. Students will learn this application using a sample survey in class where they apply what they have learned. We aim to teach the community about the challenges in user survey design and how to address these challenges and reduce bias. We also explain the different scales and metrics in surveys, and we discuss theories from the psychometrics field behind choosing scales for the construct of interest in a survey. We base our survey design techniques on well-known methods in the social science community aimed at increasing conclusion reliability. In addition, the tutorial will explain analysis techniques for survey data. We will explain different types of statistical tests their differences, and how we choose the appropriate test. We will explore topics like: sampling, test conditions, assumptions, and statistical power. We will also explain how to present findings from user surveys in research papers and how to report the statistics.

For time and location information please check the RE16 program here

Tutorial Sessions

We broke our tutorial into 4 sessions. In each session, we will introduce concepts and definitions supported by examples, exercises, and recommendations for do's and dont's.