How to score a multiple choice test?

Permanent short link to this article: https://bit.ly/2xpxRK0

Alexis Direr

Multiple-choice tests are a popular form of assessment in education. They have several advantages, including quick and easy scoring, wide sampling of content, and grading exempt from marker bias. A major drawback is the difficulty of dealing with candidates guessing the answers. Those not sure which answer is correct may still select one and reap a point if they make a lucky guess. Guessing adds an error component to scores. Omission, the possibility of leaving some items blank, may improve the assessment task. Omission suppresses the uncertainty of the chance factor but introduces another type of measurement error, which stems from the impossibility of distinguishing among candidates with varying degrees of partial knowledge.

In this article, Alexis Direr investigates which marking system gives the best estimate of candidate ability. Scoring wrong answers and omissions differs from a standard mean estimation procedure. The efficient scoring rule turns out to be sensitive to the length of the test. When candidates have to answer a limited number of questions, answers by the less able do not allow an accurate estimation of their capacities. In this configuration, the efficient mark for omission is strictly positive, to prompt test-takers not to answer if they are unsure. Conversely, when the test has a large sample of questions, ability is better estimated by inducing candidates to answer, including the less able. In this case, the mark for omission is negative, to dissuade omission.
The model sheds new light on the statistical properties of the two most used scoring methods, number-right scoring and formula scoring. Number-right scoring counts the number of right answers. Omitted items and wrong answers count for zero. Formula scoring imposes a penalty for incorrect selection. The two methods estimate candidate ability with similar measurement errors. Formula scoring induces more omission by penalizing mistakes, which reduces the error component from guessing. Number-right scoring encourages answers, which is only efficient with a very large number of items (more than 80 according to simulations). Both methods set the mark for omission inefficiently and underperform compared with an efficient scoring rule. In a simulated model of scoring, a test-maker using formula scoring or number-right scoring would have to increase the number of items by an average of 30% to obtain the same estimation accuracy as an efficient scoring rule.

..................

Original title of the article: Efficient Scoring of Multiple-Choice Tests

Published in: Working paper LEO and PSE

Available at: http://gdre.leo-univ-orleans.fr/direr/paper/qcm_v10.pdf