In medical and health sciences, OSCE is one of main important tools to assess student. OSCE stands for Objective Structure Clinical Examination. This tools is intended to assess the student skills. But all OSCE administrator also having headache in managing the paper forms from the stations rubrics.
If you have 100 students, and the OSCE circuit is consist of 12 stations, means you will have 1200 forms to manage in one time OSCE alone. Further deeper to the problem are : imagine if each rubrics have 10 items and each items is a weighted items. The administration staff have to do hand calculate 10 times 1200 forms, 12.000 items. And then transcript into spread sheet software. The probability of error is big.
And in our exam, Raters also have guideline about “how-to mark”.
It is shown that each item is a weighted item. Whilst each rubrics in each stations has different weighted items score. In every items, there are check-list criteria of skills set that have to be done by the student. This heck-list is then translated become the standard score for the student in each item score.
Now, into the technical solutions. Replace the paper rubrics using electronic form. Some are familiar with Google Form. Yes you can use that. But, in order to do that, you have to provide internet connection at all time during exam. Many of us may not have cable connection port to connect computer/laptop in the OSCE stations. Then WiFi come to the rescue! Or Not ? Yes you can use WiFi, but according to my experience, providing stable connection for large area using WiFi is not an easy task.
SO, the key point are : providing paperless form and avoiding the problem of internet connection. Hence, we can use OpenDataKit (ODK) collect, to capture the data in off-line condition. FYI, ODK collect also capable to transmit the data on-line too. This system was created to solve the problem in rural place where internet connection is rare or none.
What is the GOAL ? To reduce the manpower time to administered an OSCE and to reduce the probability of error.
Now, we will start the technical things. You want paper form/rubrics like this one below become electronic ? This rubric is mark with number zero to three (4 scale of likert scale), to mark student ability. Zero for not doing it, and Three is excellent. Every raters mark the student using the marking sheet like this (on the left). And we want to make the marking sheet like on the right. This e-form is in Android base tablet/smartphone.
Let’s review the necessary step to use this system in OSCE.
Basically, there are three steps needed.
In the scenario above, I decided to make this system work with minimal internet connection. The internet connection only needed to download the software and the app. To aggregate the data we can use the software in computer, or local server, or upload. Later we will explain this.
The purpose of this design are, to limit the IT support needed in the process. Whilst you have to remember that other scenario can be apply, depends on your needs.
To create the form, you will need :
- Form in the format of xlsform. This is an excel format file in which have special rules that if you convert, become a form
- ODK XLSForm software. This is a software to convert xls file above become a form which is ready to use in Android based tablet/smartphone.
- Form in the xml format. This is the output of XLSForm software.
The full howto use it is in this xlsform.org.
But rather than creating from a scratch, it is easier to modified files that I created. Attach is my xls files, in xlsform format, which is in my opinion is easier to edit and matching with your own rubrics. This is the link to download.
In this xls form, there are :
- Fields that you have to change in yellow color (if your rubrics is similar with ours)
- I put weighting scale for each item in yellow, which mean you could change according to your own settings.
- The Item name is in “select_one likert” row, label column, yellow color.
- The marking of the item is using scoring number, from Zero to Three.
Rater do not need to calculate the score time weighted item. All is automatically calculated by the system (form)
- This form is included the calculation of the weighted items, total score of all items and also total score in percent (calculated row, calculation column, yellow color)
- The Global Rating / Global Performance marking also incorporated in this form, with the scale : Failed, Borderline, Pass, and Excellent.
- The form sample is 6 items. If you need more items, just copy and paste one group of row (green or purple)
After creating xls file, we need to convert it into xml file, which is later on use in the Android tablet. We will use ODK XLSForm. A windows software from ODK to make sure that the excel file do not have mistake. This will check the correctness of the form format. And then the ODK XLSForm will create a xml file as the output.
Just FYI, the other option after creating xls file is uploading to aggregate server, which is not in my choice in this scenario. We will discuss about it later in different article, since we have many options regarding the usage of server. So, to make it simple, you don’t need server at this point.
ODK XLSForm in available in here.
As you can see above, you only need to point where is the xls file that you need to check and convert into xml format. Note that the file name cannot have blank space.
Once you run, you will see the result in the comment. It says the output in the location that you choose in xml. But if your xls file contain error, there will be error message saying in what line the error is/are.
After you have the xml form ready, now we should prepare the data collection.
First, you will need the ODK collect which you can download from Google PlayStore or directly from ODK collect. If you choose download from PlayStore, it has the advantage of getting automatic update.
Now we have to add manually the form to the ODK Collect. Copy the xml file into your android phone by connecting into computer. Choose the ODK folder and copy to forms folder.
In this sample, I am using emulator to write this “how-to” just for the simplicity creating image. And this emulator works just like your Android phone.
You will do this step for all of your tablet use in the exam.
Then the ODK collect is ready to use. Please click to open.
Click “Fill Blank Form” to open the rubrics.
That is the name of the Form. Click to open.
FYI, in one ODK Collect, you can have more than one forms. Therefore, you can create different form for different purposes and deploy that form just using one device. Suppose after marking the student, you need to fill out the form about the standardize patient or feed back for the exam it self.
Now you are ready to mark the participants. Mark it according the rules. In this form, the rules are : all of the marking are mandatory, you are not allow to leave each one of it empty; rater only need to mark by choosing number 0 to 3, and the weighted item will be automatically calculated; rater cannot save and close the rubrics before all the marking are complete; after completing the form, save – close – and open the new blank form again to mark another students.
Remember, that you do not need internet connection during entering the data. And theoretically, you can fill the form for hundreds time or more, as long as the memory available.
After you are done marking all the participant, all the data from all the stations is ready be aggregated.
Note : since the scenario design of this project, we are not using internet connection for collecting/aggregating the data, therefor, there is no need of setting the ODK Collect. If you decide using aggregate server, what ever you use, then you need to setup the ODK Collect in order to pull the blank form and then push the complete form to server.
This is the step after we are done with the exam or in technical term is collecting data. All raters are done with the participant, all circuit is finished.
Then the next step is combining all the data into one. If you have 12 stations means you have 12 tablet. And even if you have parallel circuits in one or more of the sessions, means you will have 24 tablet.
Well, actually, if you have laptop or desktop, you can run android simulator. You are running an Android inside your pc. For this How-to making, I am using Leapdroid. There are many more simulator out there. Leapdroid itself is not active updating the software anymore since Google bought them. But the old version is still works smoothly, and free.
Ok, back to business. In this scenario, we use ODK Briefcase, which is available here.
What is ODK Briefcase ? This software has the capability of collecting and aggregating the data from ODK Collect. This is windows based software.
All you need is :
- run the software,
- connect all the tablet to the PC one at a time,
- pull the data, and
- export the collected data. ODK Briefcase will automatically recognize same type of forms from different tablet/source.
First, run the ODK Briefcase
Click on the “pull data from”, and choose custom path. When you connect the PC to the tablet, you can browse it. Then go to ODK folder inside the tablet. This software will automatically recognize whether the folder is ODK folder or not.
After that, pull the data. And it will prompt SUCCESS when the process had no problem.
Repeat the data pull from all of tablet.
And go to Export tab
Choose the form you want to export. And you only have option to export into .csv format. Select your folder to put the result. Hit Export.
Do for all of the forms. Suppose you have 12 stations, then you have to export all of 12 station’s forms. Remember, the file output type is in .csv format, which is you can open it using excel or other spreadsheet software.
This is the xls file after I convert the csv file.
As you can see here, this is sample of one station form. Later you will need to combine the files from other stations.
The setting of the XLSForm in the design is consist of weighting item (Ujian-weight column) and also total score item (Ujian-hitung column). The form also auto calculate the Total Score and Total Score in Percent (Ujian-ScorePCT column). Therefore, the administration staff do not need to hand calculated the form. Alll they have to do is matching the students by shorting the student id (Ujian-NIPP column).
The Global Rating (Global Performance) is on the Ujian-GP column. With the 0, 1, 2, and 3 value representing Failed, Borderline, Pass, and Excellent respectively.
I hope, this article is useful, especially for those who do not have the budget to buy an OSCE system or install complicated server to manage OSCE. Reducing the need of IT support staff is my main goal design here.
AND, by reducing the human intervention / hand calculation in paper rubrics, will reduce the error probability. Of course, it will safe a lot of time and cost.
Further development will be creating the R script that can do all the calculation and analysis automatically, based on the ODK output.
That’s all you need. Hope that you can utilize this ODK to improve the education process.
If you have any further question, please write on the comment below.