This is my question every time I teach Generalizability Theory : Can we run a parametric test in non normal distribution data ? As far as I remember, all of them answer NO.
Which is NOT TRUE.
“You do NOT have to have normally distributed data. Anyone who says so is revealing his ignorance of basic statistical theory” , (Bloch & Norman , 2012)
In case of G-Theory, which is based on ANOVA, it is often that they questioning the non parametric data from the exam result whether we are able to run it in G-Theory analysis. Well, Bloch and Norman answered it. If you are not satisfied with the argument, please write comment to the respectful journal. Even I did analyze binary data, and gave beautiful result as I expected, and it called G-Kappa.
There are several things came to my mind when somebody said that the exam score result is not normal distribution (in MCQ or OSCE exam)
Firstly, you have to understand the nature of the exam. In OSCE, it is kind of exam that actually set up that the student have to pass a minimum requirement skills. For example, assess the capability of the student to do CPR.
Let say score (scale 0-100), in OSCE it is often the score mean around 60-70. Why ? Because the majority of the students should be able (or pass) the exam, otherwise the education process is failed. It is being setup in order to make the student able to pass certain level of skills.
It will be different with an MCQ test, we want to whether the student is able to understand the material or not. Which one is the smart students, which students are below the average. We have to understand and reflect back to the Classical Test Theory (CTT), then we will understand why the result should be in (around) the middle.
In MCQ, if the mean is way on the right side, it will raise you question, is it to easy ? or did it breach ? or did you gave the same question from year to year ? This could give you non normal distribution.
Secondly, if you have 100 students, are you sure that they are in the same level of knowledge ? As if you run a test in 90 students with the same question, to 30 year two, 30 year four and 30 year six of elementary school. Or in our exam, that administered in 3 rooms, had a breach in of the room. The homogeneity test will say that is not equal. And I guess that the score of the test could end up not be normal distribution.
There could be more than that, usually. I am by no mean a statistic expert, but that is how I try to understand an assessment. I try to figure where could be the error happened in a test. And if all possible preparation to create a good exam has been taken seriously, than a non normal distribution of the exam result is something worth to explore. But most of the time, there are something wrong in the exam it self. And I know, it is difficult to make a good exam. Therefore, the result, should be in normal distribution, but because there are some error in the exam, it crate different result.
Bloch, R. & Norman, G. (2012) Generalizability theory for the perplexed: A practical introduction and guide: AMEE Guide No. 68. Medical Teacher. [Online] 34 (11), 960–992. Available from: doi:10.3109/0142159X.2012.703791.