Academic Writing tips 5 : making a review paper

What I meant here is making a paper journal review, with the aim of good quality journal. Not just any journal, nor “only” a class assignment to review journal. I wrote this based on my experience published a systematic review in high impact journal in my field and supervising research thesis in a form of journal review.

The Story
Currently (December 19, 2018) I have research projects to make a systematic review with my master students and finished 2 draft of mini review project. I will try to share my experience in making it, supervising it, and rough road of submit-publish it.

During my experience teaching as a lecture in master program of hospital administration, so many question that I have or many questions from my student that I could not find definitive answer based on literature. Many time it is a random answer from a paper or two. This raise me a question, what is actually the true condition out there. And I’d like to make a review project, hence I can answer based on real evidence or not only based on somebody experience.

The Real Condition
Ok .. this is quite interesting. Making a real good review paper is super duper time consuming, and quite complicated. And I would say, if you aim high impact journal, this gonna be difficult.
But the catch is, if you are making review paper, you will have citation alot at no time … kind of.

This is not something that you can create easily during your class or as a class assignment. For my colleague, for you the one that think that creating one is easy, and make is as student assignment. Think twice … or three times … or even more.
Unless you only ask them to review several paper with no intention at all to published it. It is quite simple. With this, they can have basic knowledge for their chapter 2 (usually the Literature Review part)

In my opinion, well you can agree or not, before you ask your student(s) to publish a review paper, I would suggest that you write one. Then you can feel the real condition of making a good one. This is important so that, you can guide the direction of the review and you know when to stop.

Back to the topic, the real situation. If you read review paper from a reputable journal, you can immediately tell that usually they have alot of information and some of them discuss very deep theory.
Some of them make you headache when you try to read it. And some wrote it in excellent language style and you can understand it easily.
The one that easy to understand, will make a false impression by the reader, that make a good review paper is super duper easy. Even you can make it over the weekend. Big mistake, trust me.

If you pay attention to the review paper that you read, usually, the creator is the one that already long-time researcher in the field. And even he/she is the guru.
The question, is it possible that we (new kid on the block) make and publish review paper ? Quick answer, yes we can. The catch, you need time. A lot …

What is the main difficulties of creating a review paper ? We have to have a good background of knowledge in that field.

Hence if you ask your student to make a review paper, I would suggest that this will be his/her thesis project. And you have to get ready to spent your time too. Yes, we can ask the student to read. We can push them to the limit. But remember, many times, they are lost in the middle of the jungle. 
We have to guide them, and also read along with them. So trust me, if they have to read alot, we have to do it too. And double the energy, since we have understand and guide them.

In my opinion, before making are view paper, we have to read another review paper (that related to our goal) between 20-50 papers. This will create a good background knowledge.

So, for my student, get ready to read …read …and read. Make a technical notes to compile and cluster your information into topics. You can find how to make technical notes here. And do not forget to ask your supervisor to read too 😛
Well, I have agreement with my students. Whenever I ask them to read alot, I will do too. So then we can have a good discussion. And they have right to “get angry at me”  if I don’t do my part. Fair condition.

Type of the review
This will be arguable, please consider this as rough guideline.

For those who is in the healthcare related fields, a systematic review is familiar. Especially the chocrane. I would say it is chocrane style :p , don’t be offense. Since there are many systematic review that do not follow the chocrane rules. I personally will never say that other style is not good. Plenty of them are really good. I believe that there are many solutions to solve the problems, so chocrane style is not the only one. The simple explanation is, they follow strict rules. If you dont find any paper that already published, then you could wait. Well you might able to start with a few papers that fall into inclusion criteria. Since the chocrane team is last (forever), they could wait for very long time to make the paper done. Actually, they also publish the protocol too, and it is really good. Real good. Well, rather I have to explain it, please read about it by your self in chocrane review. I did a systematic review, but not using this protocol since it doesnot fit with my need (according to my calculation at that time).

If you google, you will find a paper that discuss the type of review in “A typology of reviews: an analysis of 14 review types and associated methodologies”.  (you can find it here)

I did systematic review, and the process in this article was based on my experience, and my experience of supervising a thesis project (of a review paper).

Making the review
In this section, I will tell the step by step going through the making of review paper.

Tools to Work with
Of course, to work in review or writing any academic paper, using bibliographic software is a must. I wrote series of how-to using zotero, and you can click it here.

You could start from part 1, if you never use it before. At least part 1 -2 -3 is a mandatory skills that you will use alot.
Regarding the citation style, find the one that using numbering format, not author date. This is easier to read, and do not use the upper script numbering format to work with.
When you are done with the draft, then you can choose the style later according to the journal that you aim.

In many workshop, I’ve got question always, which one is the good one, zotero – mendeley – or endnote ?
mmm … what is the best mobile phone ? (in terms of texting and calling) the best one is the one tha you have it with you at that moment.
Pick one among those three. You do not need to install three of them. They do have similar function. As long as it works, and you use it … this will be the best.

Do I test them ? Not testing, try to use them. Yes actually I tried 3 of them just to test several function. And I choose zotero. It just because I like it.

Note : if you work with co-author, make sure that all the author have the same software. Otherwise, it will be difficult to share the bibliographic files.

You have to remember that if you are doing systematic review, you have so many things todo. Dont be surprise. It is not like the how to written in many web site.
mmm … may be I will focus into my review project, not systematic review. Since it will be to hard for master thesis. I think. But I will use my logic in making systematic review.

In this step, I always ask my student, please make a diary. A log book. You can make it in a file, or you can make it in a book. Write the step that you made or write the plan for the next step.
This log book will be useful to keep track of what you are doing, and for me when supervising my student, I can see what they are doing. It is like, they become my eyes, I am analyzing the facts.

Dear my student, what do you want to find out ? what do you want to know ?
This is my first question to them. With the aim to know their intention. This is not like a fix research question, but become rough guideline.
It could be our discussion about the topic that I want to know. And I discuss with him/her to find out whether my student interested to take this topic.

Let me pick one example, from my student project.
Me : what do you want to know about ?
Students (S) : I want to know about hospital information system.
Me : Ok, so this is everything about HIS … the electronic medical record (EMR) things in hospital ? 
S : No, I’m interesting in e-prescribing

From this small conversation I suggest him to find a review or possible systematic review paper at least 10 about HIS in general or what ever he could find, and another min 10 review paper about e-prescribing.
I always suggest them find review paper to start with. Sometimes, in the beginning will be difficult. But this is important as the basic knowledge for them.
We will then share the files, and read it. And discuss the papers. At this point they will know more about the topic. And it is useful to sharpen the idea or research question of their own.

From the initial discussion, we have another findings or I can say topic of the field. Which is about the adherence of using CPOE (computerized patient order entry). 

The searching. Now we start to collect the article. Yes this is beginning only. Above we have 3 sub topics actually, HIS – CPOE – adherence.

First and foremost, you have to know how to search. Basic search skills can be found here (at the bottom part of the article).
This including how to choose the keywords. how to build up keywords in first place.

Find the keywords for each sub-topics. Or people say as research variable. I usually use spreadsheet (eg excel or google spreadsheet) to write down the keywords. Put every sub-topics into separate columns. Write down your idea of keywords or strike-out the keyword that you dont need. Do not delete. Use is as a tracking things, so you dont forget of what you already try to use in database searching.

After the first search using those boolean strategies, please read at least the first 10-15 papers that related to your topic.
Usually you will find another keywords ideas. Put those keywords into the the search strategies from the beginning.
Read the result, find ideas to add keywords.
And do all over again

Do those step until you feel that there is no more keywords idea coming from the reading.
Of course searching in one database is never enough. In systematic review, usually we will use at least 3 databases to search. With the same keywords, same strategies, same boolean logic.

Research Topic
Hey …. where is the topic ??? Yup this is kind of unusual. Based on my experience, and also one of workshop I attend during my PhD program, this is something that you cannot make it fix at beginning.

Some of us might think that this research topic has to be fix since the beginning. Nope, that is not how it works (according to my style, I could be wrong).
If your research question is too wide … your “analyzed article” will be too many. If your research question is too specific, you only get few articles. And end up with  nothing to review

In one of workshop in 2012, I asked the professor, how much is too much and how few is not enough.
Well it depends on what are you trying to find out. What do you want to know ?

So basically, the research question will be flexible in the process of building the sub-topics followed by the keywords building.
Read the result. And back to reviewing the research questions.

I used to say to my students, this is the never ending process whenever you want to create a good research question for a review paper. When I did for my paper, it takes 6 months, alone.

Once you satisfied … well, actually when you do systematic review, I will call this stop when you find that the keywords is saturated (just like qualitative research). When you read again … and again, you dont find any important keywords again.

If you pay attention to review paper, they will not (usually) explain the detail of the search strategies. The detail usually will only found in systematic review paper.
The easiest example is ” we search the database with the keywords such as hospital information system, electronic medical records, CPOE and adherence” … 🙂
In systematic review, even the word choice is already gone through testing … and retesting. And you should be able to show the strategies clearly, and repeatable by other researcher.

After you have your search result, it is the time when you choose the article that you want to analyzed. Series of steps that you have to design and decide your self. In my case with my students’ project, I helped them to build it. They become my eyes, they and me … us become one to analyze the papers. 

Build the inclusion and exclusion criteria (which is actually you already have it in brief when you create the keywords). But in this step, you have to write it down, make agreement with you co-author.

Remember, this filtering process, is a long and boring step. There are some steps that you remove the files by reading the title and abstract only. But then if it not enough, you should read the full paper.
From 2000 papers result, may be you should review by reading the 300-ish full paper just for removing the articles.

By the time you reach the total number of “analyzed paper” for the review, actually you already read each paper more than 3 times, at least.
Do you know why ? Because you have to be sure with your choice.

Yes and No
I would say that reviewing article that written in English is enough. Acceptable. Unless for example you will analyze the condition in specific country, then paper that written in that country language is fine.
But you still have to remember, the quality of the journal is taking into account.

You dont review a review paper. So original research paper is the choice. And I am not a big fan of reviewing grey literature, unless very specific and strong reason applied.

Step by step in a rough guide :
1. Removing the duplicate papers
2. Removing paper from reading the title only
3. Removing paper from reading the abstract only

And anything that make you in doubt, you should read the full paper

4. Removing paper after full paper reading

The removing step is based on your inclusion and exclusion criteria.

5. Analyzing the “chosen” paper

Notes :
For you my students, in the beginning, you will feel that you need to read full paper (always) to remove the paper. But, you will learn, that after you build your knowledge, it is gonna be easier to remove the unnecessary papers.

Making a review paper is full of reading …reading …and reading. And because of that nature, you have to be able to read fast.
I wrote a tips to do a speed reading (click here). It might help.

Writing the Result
So many way of reporting the result. I would say, this is your writing style. But actually also related with your main goal of making your review.

First and foremost in writing a paper. Choose your writing style, if you dont know yet, find a good paper that you like it so much in terms of the writing style. Print it, and bring it everywhere you go.

In reporting the result, I will suggest my student or whoever interested in making a review paper for the first time, write it in listing type model. No discussion in deep and full of discussion in theory.
I found some of this very good review paper, is written by the Guru.
Well, to be honest, I m not guru nor you my students. We are just a novice person.

This article, Healthcare Staff Wellbeing, Burnout, and Patient Safety: A Systematic Review, is good example of what I meant above.
Dont be fool by the result that “only” look like that. It takes time. And huge effort. But in my opinion and experience, this type of reporting result is the best choice for us, the novice person.

Choosing the Journal
Just like any other writing an academic paper, choosing the journal is quite intrigue. Many will suggest that before you write, you have to know where you want to publish. Well, it could be right, but not always.

In my case, if I do have an idea, I will write it immediately. Choose the journal later when you are done with the full draft and make adjustment according to the journal requirement.

For you my student, you might have this as the first experience in writing a serious paper, know where to publish before you write is kind of impossible. The easier way is, find the journal from the papers that you read.
For the title of you paper, you can use title that make people interesting to read. Make it so tempting to read. This is like selling product. I will suggest you to use the most interesting findings as the idea of your title.
Even though usually, the writer will use descriptive type of title that explain what it is. As for systematic review, it is unique that the title will be so descriptive and clearly define. But for other review, it is good idea, but not necessarily done like systematic review


That’s it … a simple guide for making a review paper.
I will make editing when I remember another step that I might missed in this article. Good luck with your review paper. And dont forget to share it to the world.