Acting on Student Evaluations: SET Results Flawed by Low Response Rates?

Abstract

This presentation addresses the issue of the usefulness of data derived from student evaluations of teaching (SET) as a basis for decision making in the revision of courses. What confidence can be placed in these results given that the response rates to student online surveys is very low, typically around 40%? Does a higher response rate automatically guarantee more comprehensive results, offering a more solid base for decision-making? These questions frequently arise about SET from both instructors and administrators of degree programmes in the University of Teacher Education Vaud (HEP Vaud) in Lausanne.

Drawing on survey methodology literature, and with the help of examples from the HEP Vaud database, we provide useful tips for assessing the impact of response rates, distinguishing between (a) the bias and (b) the precision of the results. Indeed, a valid SET (i.e. appropriate interpretation from the survey instrument) is subject to two necessary conditions regarding the available sample of student evaluations: this sample must be (a) representative and (b) of sufficient size. In a first step, the question is whether the students who did respond to an item adequately represent those who did not: it requires empirical analysis to answer this. In the case where the answer of the former differs from that of the latter, the results are biased by non-response error. A higher response rate (e.g. 80%) would still leave the 20% group of dissent opinions uncovered. In a second step, the question is how confidently the sample result can be attributed to all students enrolled in the course: it requires the calculation of the sampling error that arises from the specific size of the course sample (online calculators are available). The common feature of these two steps is that they aim to consider the result of respondents as a measure of the evaluation of the course as a whole. But answers can also be considered per se, as a unique source of information stemming from a group of students who have completed the course.