Ordered partition model for confidence marking modeling

Abstract

Confidence markingis increasingly used in multiple choice testing situations, but when a Rasch measurement model is applied to the data, only the binary data is used, discarding all the information given by the confidence marking. This study shows how Wilson’s Ordered Partition Model (OPM), a Rasch family model, can be used to model the confidence information. The main goal of this new approach is to produce a modeling of the use of confidence marking. In addition to showing how confidence marking is used, the new model provides a powerful and intuitive diagnostic tool to assess the state or detect problems of a testing situation for both the items and the test-takers. It will produce a set of item parameters that map the probability of using each confidence level of a confidence marking scoring scheme in relation to the test-taker’s latent ability trait: with the associated category characteristic curves for each item, it will be possible to grasp at a glance, for a given test-taker and his ability level, which confidence level he will most likely choose and his probability of a correct answer. The graphic diagnostic tool allows assessing the item difficulty, the test-taker’s over-and under-confidence or the misuse of confidence levels, but also the fact that a question is particularly tricky or creates a lot of doubt. Moreover, the model will show the differences that may arise in the use of the confidence levels. For instance, some items show confidence level bundles, which may mean that test-takers don’t totally differentiate those levels during the confidence expression process. The new model is in strict relation to the Rasch model: the probability curves for a correct/incorrect answer under the Rasch model (the ICC’s) are “split” into a set of curves, each representing aconfidence level of the confidence marking scoring scheme. Hence, the Rasch item difficulty can be easily retrieved by the OPM model.