Therefore a combination of open and closed questions in one test is an excellent alternative. But what does this mean for the distribution of questions between these two forms? How do you set cut scores? And how do you determine test reliability? These four steps will get you started.
Step 1: balance the test with these four rules
When you combine open and closed questions in one test, it is important to balance the composition of those. To do this, use these four rules:
- For combining multiple-choice and open-ended questions, appoint about 40-60 score points. This way your test is likely to produce a reasonably reliable result (see this tip for an explanation).
- Use multiple-choice questions to produce spreading of the material. Then use half the number of score points for that, for example. Use the rest of the score points for in-depth open questions. For example, 20 multiple choice questions of 1 score point each and 5 open questions of 4 score points each (= total 40 points).
- Divide the score points so that important topics and skills have more of them, so that you collect more information from the important topics to base the outcome on. For example, if there are four topics in the course, one of which is clearly more important than the other three, you assign 20 score points to that important topic, for example, and a total of 20 score points between the other three topics.
- Finally, should, for example, the one important topic not be adequately reflected in the test (for example, because they are difficult to construct) then you could apply a larger ‘weight’ to these score points to compensate a little more in the final assessment. But preferably you don't.
Step 2: avoid these mistakes
When combining open-ended and closed-ended questions in a test, you will encounter several pitfalls. Be careful to avoid these mistakes:
- Pitfall: no spread of scoring points for open-ended questions. Make sure there is a good spread of points when scoring open-ended questions. An example: an open-ended question where a student can earn 4 points, but assigning scores to student, students only are awarded 0 or 4 points (dichotomous scoring). This compromises both the reliability of the assessment and the passing rate. Ensure that such a question are awarded indeed 0, 1, 2, 3 or 4 points (polytomous scoring).
- Pitfall: giving more score points to difficult questions. Do not intentionally give more score points to the difficult questions. If you do, you will get a distorted picture of what your students can really do. The student with a higher skill level already scores points on all questions and will automatically get a higher final score. There is no need to artificially increase that score even more.
Step 3: assigning grades
To assign grades, you can proceed by combining the data: you add up all the score points. The cut-score is then determined on the basis of (according to Dutch tradition) discounting with the random guess score of the multiple-choice part. The random guess score is subtracted from the total number of points to be obtained from the test and students must then obtain half or more of the remaining portion of the score range for a pass grade.
When you combine different test forms, do students have to score a pass for both the multiple-choice part and the open part? Or do you let the parts compensate for each other? Usually you test the students on skills and knowledge that belongs together, which is why compensation is the rule. If there are parts in the material that the students must complete separately with a pass, then these are considered two different educational components that require two separate tests. The reliability of the two components is examined separately.
Step 4: calculate reliability
The reliability of combination tests with multiple-choice and open-ended questions, is simple to calculate. Cronbach alpha can be calculated from a score table that lists both the scores of the multiple-choice questions (dichotomous: 0-1) and the open questions (polytomous: 0, 1, 2, 3). This is because Cronbach alpha can handle both types of data (unlike KR20 which can only handle dichotomous data). Many assessment systems perform these analyses by default including VU Amsterdam’s TestVision platform.