Évaluation dans l'enseignement supérieur : perspectives offertes par l'application des cycles de qualité aux trois premiers niveaux du modèle de Kirkpatrick et leur instrumentation numérique

Abstract

In our opinion, a quality approach in assessment development in higher education is inseparable from a reflection on assessment quality criteria. In this area we were inspired by the recommendations contained in the Standards for Educational and Psychological Testing (AERA, APA, NCME, 1999). In order to put these reflections within the reach of field practitioners, we propose to meet quality criteria such as: validity, reliability, sensitivity, diagnosticity, equity, practicability, communicability, authenticity and acceptability (we will give a brief definition for each of them during this conference).

In addition to these criteria, it is also essential to define a step-by-step process with quality controls to create robust assessments. Given the issues and the amounts invested, training assessment methods in higher education must be of high quality and provide reliable results in order to improve education efficiency. In this area we were influenced by Edward Deming's work considered by many to be the father of the quality management movement. The way he envisions management processes improvement by adopting appropriate principles in order to increase quality and simultaneously reduce costs for organizations (by reducing waste, rework, staff attrition, etc.) was a source of inspiration for us. His key idea for a quality approach is to practice continual improvement and think the processes as systems (Deming, 1986, 1993). This is what we have transposed to the field of higher education assessments construction with in mind his famous quote “If you cannot describe what you are doing as a process, you do not know what you're doing”. Another key idea we have taken up and adapted from Deming is that of the "Plan, Do, Check, Act" (PDCA) - he always referred to the "Shewhart cycle" - and in fact, at least partially, based on the scientific method (Best & Neuhauser, 2006).

During this conference, we will explain the quality cycle model we propose for higher education effective assessments that includes following steps: (1) Analysis of what we want to evaluate; (2) Design plan concerning how it will be evaluated ; (3) Elaboration of the questions / items / evaluation devices; (4) Training / Information of the people concerned by our evaluation system; (5) Conducting the assessment and collecting the data; (6) Correction, treatment and analysis of results; (7) Feedbacks to those involved in the evaluation and to stakeholders; (8) Regulation for continuous improvement of subsequent assessments. This kind of cyclic process includes quality control procedures. The idea is to provide an easy way to build effective assessments for practitioners who want to ensure a high level of quality in their evaluations at the four levels of Kirkpatrick’s model.

We will present this quality cycle approach applied to 3 of Kirkpatrick's 4 levels of evaluation. We will show how we have applied this quality approach in the context of trainee satisfaction assessments (level 1) during their practical training at the University Hospital of Lausanne. We will also illustrate the application of the quality cycle in learning assessments (level 2) in the context of certification examinations carried out at the University of Teacher Education of State of Vaud in Switzerland. Finally, the use of the quality cycle for the evaluation of the transfer of learning (level 3) will be illustrated by applications in continuing education offered by the Training Center of the University Hospital of Lausanne.

Finally, we will show the facilities offered by certain functionalities of an online Assessment Management System (Docimo) that instruments the quality cycle applied to level 2. We will conclude by outlining the perspectives offered by an online platform yet to be built, which would allow the realization of assessment systems combining the quality cycles at the first three levels of Kirkpatrick's model in order to contribute to the continuous improvement of the quality of learning and teaching in higher education.