What is Distractor Analysis?
What is a distractor?
In Item Response Theory (IRT), a distractor is an incorrect answer option that is included in a multiple-choice question. The purpose of the distractor is to make the question more challenging by providing a plausible answer option for people who don't fully understand the question.
The distractors are chosen to be plausible answers, but not the best answer. They help to identify the test takers who have a good understanding of the subject matter and those who may have guessed the correct answer or have a poor understanding of the subject matter
What is a good distractor?
A good distractor should meet several requirements:
- Good distractors are chosen by a reasonable percentage of the test takers, as this indicates that the answer option is plausible. For questions with a high item difficulty, the percentage of answers per distractor should be higher than for easy questions.
- Good distractors are selected about as often as the other distractors, but not as often as the correct answer.
- Good distractors are selected on average by test takers with a lower IQ than those who select the correct answer. If a distractor is selected by test takers with a higher (or similar) IQ to those who select the correct answer, the distractor may be too similar to the actual answer, which could result in low item discrimination for the question.
How are distractors evaluated at brght?
At brght, we evaluate distractors by looking at the percentage of test takers who select each answer option (including both the correct answer and the distractors) and the average IQ of the test takers who selected the answer option.
For every question we automatically evaluate the following graphs. The two graphs below belong to a question with good distractors.
The correct answer is selected far more often than the distractors. All the other distractors are selected sometimes, not too often and all in the same range. Option D could be looked at, as only a small percentage of test takers selects this option.
The correct answer is chosen by test takers with a higher IQ than those who select one of the distractors. This is a strong indication that this question will have a high level of
item discrimination and will provide a significant amount of information about the IQ of the test taker.
What are simple examples of good and bad distractors?
Question: Who was the President of the United States by the end of 2022?
Good Distractors:
- A) Joe Biden (correct answer, so technically not a distractor)
- B) Donald Trump (a plausible alternative answer, as he was the previous President of the United States)
- C) Barack Obama (a plausible alternative answer, as he was the President of the United States before Donald Trump)
Bad Distractors:
- D) Vladimir Putin (not a plausible alternative answer, as he is not a citizen of the United States and is not eligible to hold the office of President)
- E) Joseph Biden (too similar to the correct answer and might even be seen as a correct answer)
- F) Vice President (too easy to eliminate as it is not a person but a function)