Item Analysis using R for Moodle output of Computer Based Test

This R script was created in order to analyze the result of Computer Based Test (CBT) that is using Moodle as the platform. Moodle it self actually provide item analysis, but we need more that.

In our Faculty, there are 4 schools : School of Medicine, School of Dentistry, Nursing School, and Pharmacy. All of the school are using block system as the education process. Each block have twice assessment (knowledge base test), middle and at the end. Each block have different block manager (BM). And each semester, average of 3 blocks in each semester in each school. The exams format are multiple choice.

In order to give feedback for BM, this script was created. The catch is, not all BM are familiar with item analysis, therefore this script’s main goal is : easy to interpret and easy to understand.

In order to use this script, I suggest you to install R studio along with R stat. The library use in this script are psychometric, multilevel, xlsx, and broom. If you use ubuntu, installing xlsx library is quite a problem it self, but for windows user I believe there is no problem at all. In our institution, the calculation have been done by administrative staff and the results are distributed to each BM.

First, before you start, this script was made to read the data import from moodle with English setting. It has “first name” and “surename” in the column title. If you set the moodle in other language, please change the column title accordingly.

It is important to note that you only need to input 4 parameter in this script, at the beginning.

  1. Nama Ujian and Nama blok : you could type what ever name you want. This field is use for output file name
  2. Jumlah Soal : type the total number of the questions
  3. cutoff : this is the minimum score for the exam

After typing those 4 parameters, you are good to go. Just run the whole script and you will find an output file in your active directory. Right in the beginning, you will prompt to choose file to analyze. The output of this script is in xls file format, so it is easy to read or even manipulate after that. Remember, that this script is designed to read the output file (xls) from moodle from the result of CBT with multiple choice format.

The xls output from this script have 7 sheets, as seen below

The calculation result of item analysis from this script is in CTT sheet (CTT stand for classical test theory), as shown below.

I’ve found interesting fact that the majority of the BM don’t really interested to read this result. The goal of giving the result to BM in order to improve the quality of the items will not achieved. Therefore, I created new sheet “warnings”. BM only need to read the resume of the calculation. If they need to know deeper then they should review back to CTT sheet.

As you can seen, in graph above, I gave the BM the solution. Just read the conclusion, no need to read numbers. Medschool doesnt like numbers 🙂
Any empty row, representing each question, means that there should be no problem with the items. Starting script v 6.5.2 (May 6th 2017), I change little bit the warning sheet, to make easier interpretation. In the first column, ideal, if you found “ok” means the question is good. The calculation did not count problem in distractor, at this moment. so if your distractor is have warning, I still count as OK. But you have to aware of this problems. I did this since I cannot find the good evidence of the amount of the distractor that make it problem. 30% is the ideal world of the distractor amount. But what about 35% or 20% … ?

Actually, I put on the note sheet how to read the ctt table. The parameter I use was from AMEE 2015 in workshop session about item analysis. If you want to change the parameter, please feel free to change it.

In the alpha and grade sheet, it is shows the Cronbach Alpha , the mean, and also min-max score. Both sheets can be written into one sheet, but I am just to lazy to re-write the code. The boxplot sheet describe it self.

Failed students are shown in standard setting sheet.

In addition to the item analysis, include in the calculation in this script is the list of failed student using different method of cutting score, both static and dynamic. Together with the alpha, we can have better sense on the exam.

 

Preparation for the xls file of Moodle export :
The file will show 1 or 0 (one or zero). But sometimes 0.5 or 1.6, it is no problem at all, just replace the number with 1 (one). And if the students did not answer the question, in the result will be shown as “-” , therefor you need to replace it with blank space.

What if you dont use moodle ? As long as the data saved in as seen below, it would be fine

 

The criteria of the data structure are :

  1. Correct answer is 1, and wrong answer is 0
  2. The students goes down (row), and the exam questions are to the right (column)
  3. At least 1 column of student id. ID number, First Name, and Last Name columnt (total of 3 columns) are acceptable too.

The script will calculate the total score. There is no need put total score here.
I hope that this script can help little bit for anybody who need to do item analysis. Enjoy

Please feel free to download (click here) the script. Rename the file extension, from .txt into .R
And Run the script.
This script is version 6.5.2 . last updated in May 6th 2017

References :
The variables in my scripts based on this and this. Both are workshop in psychometric in the AMEE conference 2015 in Glasgow.

Leave a Reply

Your email address will not be published. Required fields are marked *