Tips for the first-time academic publication referee

In this blog post I have collected some tips for writing a good referee report in the research methods subject IT3010 that I teach at NTNU together with Elena Parmiggiani. The tips are based on actual review reports that the students submit. So new students will benefit from carefully reading this blog post. Please provide your feedback to teachers if you have questions or suggestions for more tips.

Before I start with the tips, maybe a meta-tip is this: writing a referee report for an academic publication is very different from writing an Amazon review for a book you recently read. This list points out some of the differences.

  • The review you will do will have two purposes. First and foremost, it will help the authors improve their final submission. Second, it will help conference/course organizers to grade the final submission. So focus on how you can help the authors improve their paper first, and then on the organizers.
  • It is important to have an improvement mindset rather than a critique mindset. You are not the enemy of the authors. You are their friend. You should see potential for improvements everywhere, even in the weakest parts of the paper. Your goal is to help authors bring the paper up to a quality level that is good enough for publication.
  • Even a short paper can be improved in a hundred ways. Handle major issues first–e.g. if there are no explicit research questions, if the research strategy does not fit the RQ . Then list minor issues –such as language, errors in numbers, inconsistencies. If you need to cut down because of word limits, then cut down on minor issues first.
  • Be specific both when you say something positive and negative. Remember that your job is not to provide your opinion, but to help authors improve their paper. So instead of saying “I think the paper is very well-written” say rather: “The paper is easy to follow because the authors explicitly write about their RQ in the introduction, show what we know and not know about the RQ, and are clear about their own contribution”.
  • You should justify your critique and your praise. Don’t just say “the method used fits well the purpose of the paper” without also saying “because interview as method gives a detailed insight….”. Don’t say “the article lacks focus” without also saying “because the research question is too wide”.
  • You should focus on what the authors have written, and avoid the temptation to demonstrate how you would have written the paper. You are not the author, you are the reviewer. Point out the potential for improvements of what is already there. Don’t introduce new topics e.g. “the authors have looked into RQ1, but I strongly believe RQ2 would have been a better one to address.”
  • You should avoid mentioning your score –i.e. accept, reject etc. –in your review. You should focus on your suggestions to the authors. Don’t write things like “The lack of references to literature reduces the score of the paper” or “I withdrew one point because of…”. It is important that you are aware of your role as a reviewer, and course organizers’ role as those who will take the last decision on accept/reject.
  • In order to make it easier to address your suggestions for improvement, try to enumerate them or provide them as a bullet list of proposals. A dense paragraph consisting of many proposals and suggestions can be difficult to handle. Be as clear as you can. (E.g. “I have five proposals to improve the methods sections, as listed below:”).
  • Avoid informal language (e.g. “cmm’on guys you can do better”). Note that as a referee you are entering an international academic network where norms of communication can vary a lot from country to country. Some people might misinterpret an informal language. Be concise, but not too informal.
  • Your main audience for the review is the authors. So consider writing as if you are talking to them. This might for example mean you should not talk about them in third person –like “they propose to solve problem X,” or “the authors propose…” –but rather “you propose…”.
  • Avoid exaggerated language such as “there are huge problems with the methods section”. “Huge” in this sentence refers to your personal opinion, and needs to be justified. How do you define huge, and why are the huge problems here?
  • Do not tell the authors about your relevant expertise and authority –or lack of such — inside the review report, e.g. don’t write “I am not familiar with this field, so read my review with a pinch of salt”. There is a specific field in the review form that asks you to rate your own expertise in the field of research that the paper is about.
  • Do not write in your review report that the research or the research field is not interesting for you because you are not e.g a psychologist or anthropologist etc. Your job as a reviewer is not to be interested in the topic but to help improve the paper.
  • Since the course has a tight schedule it is not useful to give comments such as “you should collect more data” or “you should revise your research question”. Your comments should focus on the available material and how it can be improved, e.g. through a better analysis or bringing in a couple of new references.