CBE Research Workshop ReviewingRefereeing Papers

1
CBE Research WorkshopReviewing/Refereeing Papers

• Panel Discussion
• 28 August 2009
• Panel Professors Ujwal Kayande, Martin
  Richardson and Tom Smith
• Chair Dr Walter Fernández

2
Agenda

• Writing Referee Reports
• By Prof Tom Smith, FAS, CBE
• Referee Reports an editors perspective
• By Prof Martin Richardson, Economics, CBE
• Umpiring in the market of ideas
• By Prof Ujwal Kayande, MMIB, CBE
• QA

3
Writing Referee Reports

• Tom SmithFAS, CBE

4
Why be a Referee

• Our Goal as Academics is to do high quality
  research and publish it in the best possible
  place
• Does being a referee help?

5
Pros

• Service to the Academic Community
• Journals need referees to screen papers, you need
  to publish in Journals so this is your
  contribution to the system
• Can lead to Associate Editor/Editor roles which
  are important Esteem Factors in obtaining
  promotion within the University System. These
  roles also give you visibility beyond your own
  University
• Active Referees can influence the development of
  the literature. If you dont like the direction
  others may take the literature, then you need to
  do your bit to keep it on track
• Regarded as part of your job as an Academic

6
Cons

• Dont want to lose sight of your goal of doing
  high quality research and publishing it in the
  best possible place
• Referee reports can take a lot of time and effort
• No real immediate payoff, rewards are more
  indirect and long term in nature

7
When you receive a Review Request

• Need to decide whether you will review the paper.
  In general, you should review it. Possible
  exceptions
• Not your area of expertise
• You have reviewed the paper before for another
  Journal
• You are going on extended leave to a remote
  locale
• The Review request will have a time limit,
  sometimes as short as 4-6 weeks. You should
  review the paper within the time limit.
• Common Misconceptions
• Early Career Researchers often think of the
  review as an evaluation of themselves rather than
  an evaluation of the paper under review and tend
  to put it off.
• Some people think if you get it back on time or
  too early that this will just attract more
  reviews.

8
Writing the Report

• Start with a paragraph setting out what the paper
  does the what, why, how part of the review.
• This shows the author that you understand what
  they have done
• Also very helpful for the Editor who may have
  2000 articles under review
• Next a paragraph about your overall assessment of
  the paper. This might be more appropriate for
  conference discussions rather than Journal
  referee reports

9
Writing the Report

• Next, a list of your main points. These
  essentially fall under the What/Why/How
  headings
• The Topic of the Paper the What
• Theoretical motivation the Why
• Methodological the How
• Data with what
• Empirical the What
• Conclusions are they supported by the analysis
• How the What/Why/How ties together
• Writing/Editorial
• Note that you wouldnt go through all of the
  above, only the ones that needed addressing
• It is a good idea to number your points for ease
  of reference for when the Author is responding

10
Writing the Report

• Next, a list of your minor points. For example
• Typos
• Length
• Tables
• Figures
• References
• Be constructive. Remember the bad referee reports
  that you have received. You dont want to
  perpetuate this process!
• A rejection could be very short. For example
• The study may have a fatal flaw in Theorem 1 on
  page 1
• The paper is not up to the standards of the
  Journal
• A revise and resubmit can be reasonably long --
  2-10 pages

11
Accept/Reject/Revise Resubmit

• Note that you dont give you decision in the
  referee report, you do this in the cover letter
  to the Editor
• Be explicit about your recommendation
• Accept
• Revise and Resubmit
• Reject
• Note that for major Journals an Accept
  recommendation on a first round review would be
  very unusual eg the Journal of Finance in a
  typical year has over 2000 submissions and no
  papers are accepted on a first round review!

12
Final Thoughts

• If you are having trouble with your report, ask
  your Colleagues. You will find a
  sharing/caring/supportive community there to help
  you.
• Final point is that you have a duty not to
  plagiarize. If you are someone who rejects papers
  and then writes a paper yourself using those
  ideas you will be quickly caught out!

13
Final Thoughts

• If you are having trouble with your report, ask
  your Colleagues. You will find a
  sharing/caring/supportive community there to help
  you.
• Final point is that you have a duty not to
  plagiarize. If you are someone who rejects papers
  and then writes a paper yourself using those
  ideas you will be quickly caught out!

14
Classic Referee Reports

• George Ackerlof The Market for Lemons paper.
• American Economic Review report said the paper
  was interesting but that the AER did not publish
  such trivial stuff
• Journal of Political Economics report said that
  the paper was too general to be true
• Review of Economic Studies report said that it
  was too trivial
• Finally the paper was accepted by the Quarterly
  Journal of Economics

15
Classic Referee Reports

• William Sharpe Capital Asset Prices A Theory of
  Equilibrium Under Conditions of Risk. Referee
  said the assumptions that all investors made the
  same predictions were so preposterous that it
  made all the conclusions uninteresting
• For other Classic Referee Reports see
• Gans and Shepherd, How are the Mighty Fallen
  Rejected Classic Articles by Leading Economists,
  The Journal of Empirical Perspectives, 1994,
  165-79

16
Example 1
17
(No Transcript)
18
(No Transcript)
19
(No Transcript)
20
(No Transcript)
21
(No Transcript)
22
Example 1 2nd Round Report
23
Example 2
24
(No Transcript)
25
(No Transcript)
26
Referee Reports an editors perspective

• Martin RichardsonEconomics, CBE

27
Why should you say yes?

• Essential professional service
• Benefits to referee
• Vote of confidence in your expertise
• Keeps you up to date with field
• Warm fuzzies from public service

28
When can you say no?

• Can become a burden
• Quick no better than slow no (or, worse, no
  response or, worst, yes but actually no.)

29
When can you say no?

• No editor expects writing a report to be a top
  priority. Own rule of thumb say no if have
  stock of 3 already. Others say 3-5.
• Obviously refuse if outside expertise. Inform
  editor of what you are prepared to consider.
• Skim paper immediately on receipt. Return
  immediately if not of professional standard.
  (Delay in doing this sends very bad signal.)
• If an editor comes to the well too often,
  politely decline on grounds of workload.

30
Role of report

• One of 1r inputs, where r of referees.
• Leave nothing unsaid or implicit. A point that
  comes up in every report will have more weight
  with the editor.
• You may have been chosen for a particular angle
  (e.g. empirical, or theoretical) so need to
  provide that focus.

31
Wastrel referees 1. No reply

• Really slows process great disservice to
  authors.
• Consequences? Yes
• Disrespect of editor
• Consequences for own submissions to journal?

32
Wastrel referees 2. Inadequate reports

• Actual report, in its entirety
• This paper provides an exemplary example sic!
  of applied economic research. The paper clearly
  and cogently introduces the reader to previous
  literature, explains the model used in this case,
  the data employed to estimate the model, the
  results of estimations and diagnostic tests
  completed. Thoughtful interpretations of the
  results are provided as well as some useful
  concluding comments. The information in the
  paper is clearly and accurately reported and is
  likely to be of interest to readers of the
  journal.

33
Wastrel referees 2. Inadequate reports

• Another one, also in its entirety
• Whilst I do not profess to be an expert in the
  application of the empirical technique employed,
  this paper is very topical, robust and accessible
  to a range of readers. In my opinion it adds
  substantially to the current debate about ____
  and raises concerns about perverse outcomes of
  ____. I thereby consider that the paper is
  eminently suitable for publication.

34
Elements of good report

• Summarise paper (briefly) shows author that you
  understand contribution (or not ? better
  presentation needed)
• Highlight strengths as well as weaknesses. This
  includes style issues as well as content.
• Suggest remedies for weaknesses (if possible).
• Ideas for shortening? Editors never ask for
  longer revisions!

35
Elements of good report

• Be civil and polite but dont pull punches. Any
  irritation with paper can be expressed in cover
  letter to editor no need to offend author.
  Golden rule imagine you are receiving this
  report.
• Cover letter should be extremely honest and
  include recommendation for editors decision.
• Be aware of journals level. Refer to other
  journals if unsure. If recommending rejection,
  suggest more suitable outlets in cover letter.

36
Dealing with resubmission

• Check that initial complaints have been dealt
  with, but check also that what you liked about
  original version has not been changed.
• If your suggestions have been followed and
  nothing else removed you should respect your
  original decision. Dont change your mind unless
  you have a really good reason its not fair to
  the author and indicates to the editor that you
  didnt take sufficient care first time around.

37
Other issues

• You may receive a paper from journal B that you
  have already rejected at journal A. If nothing
  has changed then your report need not either.
  Just tell editor and send previous report (I
  would not reveal identity of A, however.)
• Respect the double blind process dont Google
  the paper title (and its best not to reveal
  identity later either.)

38
UMPIRING IN THE MARKET OF IDEAS

• Ujwal Kayande

39
Umpiring in the Market of Ideas

• What is the market like?
• Players, umpires, match referees
• Why umpire?
• Umpire Behaviour
• How do they umpire
• Umpires are not anonymous
• Does academic age affect behaviour
• How do they respond
• The good, bad, and the ugly

40
What is the Market of Ideas like?

• Players, umpires, match referees
• Authors, reviewers, editors
• Too few umpires BIG problem
• 3 top journals in Marketing
• 762 papers (2005-2009)
• About 11,000 submissions
• 387 authors with more than two pubs
• Perhaps 200 active reviewers (known to Editors)
• 12 reviews per year per umpire HUGE
• Leads to increase in Desk Rejections

41
What is the Market of Ideas like?

• Why umpire?
• Be part of the market
• Shape the market
• Learn how to play successfully in the market
• Learn the rules of the game
• Get into the editors good books
• Chicken and egg problem
• Feel happy
• Did you review the award-winning paper?
• Is there a part of you in each paper that is
  published?

42
Umpire Behaviour

• How do they behave?
• Garud, Kayande, Huang (in progress)
• Examined 30,000 patent applications in a tech
  field
• Also 4000 submissions to a top business journal
• Interested in the costs of novelty
• Mapping out how the extent of departure from
  market knowledge affects
• Acceptance of idea
• Value of the idea, given acceptance

43
How do Umpires Behave?

• Novelty (i.e., departure from the centre of
  market knowledge)
• Good cause its different from what is known
• Definition of novelty
• Bad cause its different from what is known
• Non-conformance penalty
• On balance, bad outweighs good, resulting in
  rejection

44
How do Umpires Behave?
Risk Adjusted Preference
Novelty
45
How do Umpires Behave?
Time to Acceptance (Reverse Scale)
Novelty
46
How do Umpires Behave?
Number of Rejects (Reverse Scale)
Novelty
47
How do Umpires Behave?
Value, given acceptance (Citations)
Novelty
48
Why do Reviewers Reject?

• Reviewers are NOT anonymous
• Known to editors
• Need to satisfy editors/area editors
• Search for flaws
• What if I say accept, but there are big flaws
• Doesnt matter if you reject (editors like that)
• Need to show that they are smart
• They are just B_at__at_ds

49
Does Age Matter?

• Time since first review Age
• Probability of rejecting decreases significantly
  with age
• Supports many speculations on previous slide

50
How do Reviewers Respond

• Darrell Hair moment
• Reviewer doesnt know what he/she is talking
  about
• How would you feel?
• The reviewer is clearly not familiar with
  generalizability theory
• The paper they cited for support was written by
  me
• We thank Reviewer A for this brilliant idea. The
  paper has improved greatly as a consequence
• Were kicking ourselves that we didnt think of
  this ourselves thank you so very much for this
  suggestion to validate our model
• Letter to editor Your ignorance is only
  exceeded by your arrogance

51
The Good, the Bad, and the Ugly

• Submitted to JMR in 1995
• REVIEWER 1 (Constructive review MY HERO)
• The proposed approach is designed to optimize
  the sampling scheme to provide acceptable
  reliability for the managerial purpose at hand.
  This is a very good idea that should be pursued
  further. Right now, though, the paper doesnt
  live up to its considerable promise. To be an
  outstanding contribution that this paper is
  capable of being, further attention must be paid
  to the following issues
• Four pages of suggestions to improve
• REVIEWER 2 (Constructive review)
• I think the paper will eventually be a
  contribution, but right now, it tries to do all
  these three things, and ends up doing none of
  them particularly well (in depth)
• Editor
• I dont have good news on your paper. Reviewers
  2, 3, and 4 recommend rejection, while Reviewer 1
  has some suggestions. Thank you, etc etc.
• Reworked the paper over 8 months, resubmitted
  with an appeal to reconsider the paper in JMR
  Editor agreed to send to same reviewers.

52
The Good, the Bad, and the Ugly

• Reviewers 1 and 4 recommend revision
• Reviewers 2 and 3 hate it (identity crisis,
  dont know what they are trying to do, dont
  show how their method is better, etc etc.)
• Editor
• We have mixed reactions. Given those reactions,
  I am in a dilemma as to what to do with it.
  However, rather than reject it, I offer you the
  opportunity to revise and resubmit. It is a
  high-risk revision

53
The Good, the Bad, and the Ugly

• Resubmitted after 6 months
• Editor
• Reviewers 1 and 4 have no additional comments,
  recommend acceptance
• Reviewer 2 has no comments, and recommends
  rejection
• Reviewer 3 has some comments
• Decision Conditional Acceptance
• Accepted after 2 months, published in 1997
• Won the 1998 Lehmann Award for best PhD-based
  paper published in JMR or JM.

54
The BAD

• Reviewer 2
• Modeling is an important form of representation
  of our conceptual knowledge. However, the
  conceptual point needs to be an important one to
  warrant the technique and
• endeavor. I feel the current project is about
  something relative trivial and something already
  known in consumer psychology (e.g., concept
  incoherence results in doubt, or
• uncertainty, see Greg Murphy).
• The stated findings and contributions on p1
  (bottom half) and p4 (bottom half) are so known
  and obvious that it makes one wonder why bother.
• Published in Marketing Science in 2007.

55
The UGLY

• Reviewer 1
• This paper should definitely be rejected. It has
  fatal flaws that indicate the authors are
  research novices. The main flaw is not
  understanding the source of the DSS adoption
  problem. The problem source is not what the
  authors say it is, but that quantitative
  marketing models are terrible because they are
  too complex to understand.
• Submitted to Marketing Science, the top modelling
  journal in marketing. Paper co-authored with two
  Associate Editors of Marketing Science.

56
GOING FORWARD

• Read the paper immediately after you receive it
  a 30 min read at most.
• Identify red flags, note down on front page.
• Chew on the paper
• If this were your paper, how would you make it
  better?
• Review
• Summarise the papers main point
• Identify red flags
• Explain how paper could be improved by authors
• No decision in the review its the editors
  call
• Wish the authors the best as they continue work
  in the area
• Be polite, generous, fair, but tough

57
THINK ABOUT
58
THINK ABOUT
59
Q A