IMEC Presentation

1
Proposal Writing The Refereeing Process
Class 4 09.12.2004
2

3
Contents

• Introduction
• Peer review process
• Journals
• Conferences
• Research programmes
• The tasks of a referee
• Reviewing a research paper
• Preparing the referee report recommendations
• Evaluating a research proposal
• Acting as an editor or program chairperson
• How to become a referee?
• Final words

4
Disclaimer

• There is no fixed mechanism for refereeing
• There are simple rules that help transforming a
  review in a constructive document
• In time you will develop your own style of
  refereeing

5
Introduction

• A scientific paper is expected to provide a
  sufficient contribution to the knowledge base of
  its field
• Number of scientific papers and articles (2000)
  gt 600 000 (ISI)
• About 50 in the fields of science and
  technology
• The number of papers and articles submitted for
  publication is much larger
• refereeing process selects the ones to be
  published
• Examples of acceptance rates after refereeing
• Journals 10-20 (large variance)
• Conferences 10-50
• Workshops 30-90
• Refereeing is also used in selecting research
  projects to be funded

6
Introduction

• What is a sufficient contribution?
• new result, theoretical or experimental
• new insight
• novel synthesis of ideas
• useful survey
• useful tutorial
• What is not a sufficient contribution
• badly written
• erroneous data
• MPI Minimum Publishable Increment depends on
  the forum

7
Peer review process

• Peer reviews are carried out by anonymous
  referees who evaluate the sufficiency of
  contribution
• novelty, significance, correctness, readability
• Refereeing is public service to the scientific
  community
• professional obligation,
• carried out on volunteer basis
• requires high expertice
• helps in improving ones own expertice
• ensures the integrity of science

8
Peer review process of a journal
submission
editor
author
publish
accept reject revise
selection of associate editor
reviews recommendations
associate editors
referees
selection of referees checking of revised papers
9
Peer review process of a conference
submission
program chair
program committee
author
accept/ reject/ accept with revisions
selection of the referees checking of revisions
accept/reject/minor revision recommendations
referees
extra referees
10
Peer review process of a workshop
submit
program chair
program committee
author
accept/ reject
refereeing checking of revisions
extra referees
11
Peer review process of a research programme
submission
steering committee
proposer
accept with partial funding/ reject
referees
Notice not representative of all research
programmes
12
The tasks of a referee

• The reviewer grades a paper based on its novelty,
  significance,correctness, and readability
• In case of substantial conflicts of interest or
  if the paper is out of the field of the reviewer,
  the editor must be informed promptly
• Both positive and negative findings are
  summarized in a referee report
• Confidential part only for the editor/program
  committee Information that could reveal the
  identity of the reviewer or in minor conflicts of
  interest
• non-confidential part for the author/program
  committee
• Learn from the other reviews, if they are sent to
  you after the process

13
Why do it?

• Several reasons
• Enhance reputation (with editor/prog. committee)
• Expedites processing of your own papers
• Get on editorial board or program committee
• Good practice
• Increase your own critical appraisal ability
• Your papers become better
• Sometimes it gets preferential treatment for your
  papers
• but refereeing means more work!

14
Consideration

• Most reviews have strict deadlines
• By agreeing to review you take the responsibility
  of doing a thorough job
• If you cannot commit to this, notify the editor
  asap
• Editors understand you may not have the time, but
  are unforgiving if you commit and do a poor job
• Good editors keep a list

15
The right attitude I can learn something!

• Humbleness and an open mind needed 100
  self-confidence can be harmful
• Early assumptions on the correctness of the paper
  or the sufficiency
• of its references should be avoided
• an elegantly written paper may have zero actual
  contribution
• a paper with broken English may contain a major
  new idea
• The papers recommended for acceptance should have
  novelty and be correct
• If the reviewer cant check a fact or is unsure,
  this should be stated in the review report
• But dont waste your time on analysing in detail
  a paper that is never publishable
• a single crucial error is enough

16
Reviewing a research paper

• The paper to be reviewed is typically accompanied
  with a review form
• fill the five point scale questions last
• it is most important to write an itemized review
  report
• Relevance
• poor marginal fair x good
  excellent
• Originality
• poor marginal fair x good
  excellent
• Background knowledge of the subject and
  references
• poor marginal fair x good
  excellent
• Technical content
• poor marginal fair x good
  excellent
• Presentation
• poor marginal fair x good
  excellent

17
Reviewing a research paper analysis

• The analysis of a paper can be done by generating
  explanations to the following eight points (Smith
  1990)
• What is the purpose of the paper
• Is the problem clearly stated and have
• the key issues been pointed out?
• Is it clear what has been accomplished?
• Is the paper appropriate for the intended forum?
• If it is not, what could be a better choice?
• Is the goal significant has the work been worth
  doing?
• Are the results just trivial variations or
  extensions of previous
• results?
• Are there any new ideas, or novelties in research
  methodology?
• Citation analysis using electronic libraries are
  a big help!

18
Reviewing a research paper analysis (contd)

• Is the method of approach clear and valid?
• Is there something fundamentally flawed in the
  approach?
• Are the assumptions realistic and does that
  matter?
• Is the method new? Can it be generalized to other
  problems?
• Is the actual execution of the research correct?
• Are the mathematics and statistics correct?
  Check!
• Have the simulations been described in sufficient
  detail for replication?
• What about the boundary conditions?
• Do the results make sense?
• This part may require considerable effort from
  the reviewer...

19
Reviewing a research paper analysis (contd)

• Are the conclusions correct?
• What are the applications or implications of the
  results and are the results analysed to an
  adequate depth?
• Is the presentation satisfactory?
• Is the paper readable? Is it structured according
  to the convenstions of scientific publications?
• What did you as the reviewer learn?
• If you didnt learn anything, then the paper is
  not publishable(provided that you understood the
  paper)

20
Reviewing a research paper analyzing the
references

• It is researchers professional obligation to
  cite prior work
• the manuscript being reviewed includes claims of
  novelties regularly citing prior research
• the reviewer needs to check the validity of the
  claims
• most efficient to carry out the analysis using
  electronic libraries
• At minimum
• Check what is found using the key words of the
  article
• Study the references you dont know beforehand
• Check which recent papers cite the same
  references
• Check the references of those recent papers

21
Review structure
The actual refereeing form General comments on
the paper Specific comments on the
paper Confidential note to editor General idea
be professional and non-hostile write the review
in a style that you would like to receive for
your paper
22
The refereeing form
Forms might look quite different but basically
ask the same things Poorly designed ones just
have yes/no answers, good ones prompt the referee
to elaborate Make sure you read and understand it
well
23
Writing the referee report

• No fixed rules exist, the following ones are
  according to (Smith 1990)
• Most important make your opinions clear avoid
  perhaps and maybe evaluate the paper, not
  the author itemize the contributions
• State the recommendation and its justification
  the five point scale part of the evaluation form
  is not enough
• Show with a few summarizing sentences that you
  have understood the paper. The editor may use
  this part and compare your summary to those of
  the other reviewers
• Evaluate the significance and validity of the
  research goal
• Evaluate the quality of methodology, techniques,
  accuracy and presentation recommendations for
  revisions can be written here
• Make a clear recommendation for or against
  publication with justifications

24
Compiling the recommendations

• Classification of papers (Smith 1990)
• Very significant includes major results (lt1 of
  all papers)
• Interesting work, a good contribution (lt10)
• Minor positive contribution (10-30)
• Elegant and technically correct, but useless
• Neither elegant nor useful, but not wrong
• Wrong and misleading
• Unreadable, impossible to evaluate
• The acceptance level of the journals and
  conferences vary 1,2, and perhaps 3(-4)

25
Outcome

• Usually
• Accept the paper as it is
• Paper requires minor changes
• Paper requires major changes (with or without a
  new refereeing process)
• Reject publication of the paper
• You can only suggest, the choice is not yours
• Decision is based on at least 3 reviews

26
Research proposals

• A research proposal is a request for funding
  submitted to,
• MCyT, MECD, GENCAT
• European Commission
• NIH, NASA, NSF, ESF
• other funding organization such as a foundation
• The key difference to reviewing research papers
  is that
• the reviewers also evaluate the proposers
• Not all organization use peer review as a means
  for selecting proposals for funding

27
Evaluating research proposals

• The evaluation criteria vary between funding
  organizations
• Key criteria
• Is the research topic significant?
• Are the goals realistic?
• Has the proposer sufficient expertice and
  facilities to reach the goals?
• Is the requested funding reasonable?

28
Ethics of refereeing

• Objectivity
• Judge paper on its own merits
• Remove prejudice
• If you are not able to review it, return it
• Fairness
• Author may have different point of view /
  methodology / arguments
• Judge from their school of thought not yours
• Speed
• Be fast, but do not rush. Author deserves a fair
  hearing

29
Ethics of refereeing

• Professional treatment
• Act in the best interest of the author and
  conference/journal
• Specific rather than vague criticism
• Confidentiality
• Cannot circulate paper
• Cannot use without permission
• Conflict of interest
• Discuss with editor

30
Ethics of refereeing

• Honesty
• About your expertise and confidence in appraisal
• Courtesy
• Constructive criticism
• Non-inflammatory language
• Suggest improvements

31
Acting as an editor or program chairperson

• The editor
• maintains correspondence with authors and
  referees
• finds new referees if the ones assigned fail to
  act in given time
• decides on acceptance, rejection or a revision
  round based on 2-4 review statements.
• should distribute all review statements to the
  referees
• receives occasional negative feedback
• Review is not a vote! The editor is likely to
  line himself according to
• the best justified recommendations
• Conference program committees often rely on the
  numerical
• evaluations, occasionally resorting to vote

32
How to become a referee

• Writing a publication that is cited is the most
  certain way to become a referee
• Coordination or technical coordination of an EU
  RTD project is a direct road to proposal
  evaluations
• Refereeing is very rewarding, helps to keep
  up-to-date and aware of developments in fields
  adjacent to ones own specialty

33
Final words

• Good referee reports are valuable and free of
  charge
• help in improving the paper
• help in improving as a researcher
• help in improving as a referee
• Refereeing is a learning experience
• Scientific progress rests heavily on peer reviews

34

• PROJECT THESIS EDITOR OFFICE
• Using the articles prepared based on The Six
  Napoleons, we will set up an editorial office
• Each student will act as an associate editor of
  one article and will review three articles (see
  handout)
• Each student will peer review his/her three
  allocated articles and will return the referee
  report (see handout) to the assigned associate
  editor - deadline 17th December
• The associate editor will compile the final
  report and will return the final report with the
  individual referee reports to the Editorial
  Office - deadline 24th December
• Tutorial group discussion in January!

35

36
Proposal Writing

• In order to carry out research, in general
  financing is required.
• There are several national and international
  sources of funding and the process for obtaining
  funding is realised through proposal submission
  and review.
• The aim of this section is to INFORM you of the
  proposal process, proposal formats and existing
  funding bodies.
• The homework of this class will be to draft a
  proposal of your DEA/PhD to assist you in your
  resaerch planning, but NOT with a view to
  preparing a formal proposal in the style of
  those submitted for financing!

37
Content

• Why research ?
• Why should this be in a competitive context ?
• Why a research proposal ?
• Getting started
• What makes a good proposal ?
• Writing your proposal
• How to structure your proposal ?
• The review process
• Allocation of funding
• What next ?
• Getting help with your proposal ?
• Quick TIPS for writing a good proposal

38
Why research ?

• Why is the development of research within
  universities a must ?
• To maintain the quality of teaching programs.
• Provide the basis for undergraduate and graduate
  thesis research projects.
• Universities should be more than degree
  delivering institutions.
• Universities should be the basket for new
  knowledge and developments.

39
Why should this be in a competitive context ?

• Do universities have the financial capacity to
  develop and support research activities ?
• Where can the money be found to develop and
  support research ?
• How can the society gets the highest return on
  investment ?

40
Why a research proposal ?

• Convince others the project you have designed is
  important, worth the effort.
• Convince others that you have the ability to
  carry out the research design and report the
  findings.
• Generate funds to sustain the research units
  operation.

41
Getting started

• Know your subject. The reviewers will look for an
  up-to-date knowledge of the research area.
• Know your funder. Be aware of the priorities and
  interests of the funder you approach, and know
  that funders are unlikely to support the same
  idea twice.

42
Getting started

• Consult colleagues.
• Dont be afraid to discuss your proposal with
  colleagues, or even with the grants officer at
  the funding body.
• Early discussions can ensure that your proposal
  is targeted appropriately.

43
What makes a good proposal ?

• A well-prepared application should require
  minimal effort on the part of the reviewer.
• Proposals must demonstrate high scientific
  quality.
• The requested funds must be in proportion to the
  proposed project (cost-effectiveness).

44
Writing your proposal

• Allow plenty of time to prepare your proposal. A
  good starting point is to write a one-page
  summary of the whole project. This may take a
  while to get right, but once completed it will
  serve as an invaluable tool for writing your full
  proposal.
• Use your proposal to show the need and then fill
  the gap.

45
Writing your proposal

• Present your proposal in terms of the aims and
  objectives of the funder and not just your own
  make it clear how you will be helping them to
  fund their priorities.
• Consider the questions the funder will be asking
  Why fund you ? Why fund this ? Why now ? ...
  and make sure that the proposal answers them!

46
Writing your proposal

• Be aware that you will have limited to none
  opportunities to answer queries arising from a
  reading of your proposal.
• Consult the funders website and read clearly the
  call for research proposals as well as the
  criteria against which your proposal will be
  judged.

47
Writing your proposal

• Although it is the content that matters, good
  presentation is often crucial to making your
  proposal accessible to reviewers and keeping
  their interest.
• Use diagrams and tables to add clarity
• Bullet points and sections can break up text
• Keep to page, word and font size restrictions
  and
• Activate the spell checker while writing.

48
How to structure your proposal ?

• Check guidelines carefully failing to meet the
  funders format and specifications is one of the
  most common reasons for applications being
  returned.
• A common proposal structure normally consists of
  title, abstract, background, aims and objectives,
  methodology, work program, resources, outcomes
  (outputs dissemination), project management,
  reviewers.

49
How to structure your proposal ?

• Title This is the first impression the reader
  gets.
• The title should be short and clear, and the
  reviewer should be able to understand from the
  title the intentions of the research.
• A catchy title posing a question or including an
  apparant contradiction or acronym may be more
  easily remembered by a reviewer.

50
How to structure your proposal ?

• Abstract Should be a concise summary of the
  WHOLE project.
• Use the abstract to identify the need for this
  research, state what you intend to do, and how
  you intend to do it.
• Do not include unnecessary detail make each
  phrase count.
• And remember it is the first impression a
  reviewer gets of an applicants worth!

51
How to structure your proposal ?

• Background This section should be used to put
  the work into context what has been done before,
  and how will the proposed work add to it ?
• What is the innovative aspect in the research
  project ?
• Build your case by demonstrating your capability
  and familiarity in the area.

52
How to structure your proposal ?

• Aims and objectives The aims should describe
  what you intend to achieve by doing this piece of
  work.
• Your objectives are the small steps you need to
  reach in order to achieve your aim.
• Aims and objectives should be realistic,
  consistent, and link them to methods, timetable,
  and outcomes.

53
How to structure your proposal ?

• Methodology Methods should be detailed and well
  thought through.
• Explain why you have chosen a particular method.
• Base your explanation on literature references.
• If your own experience of a methodology is
  limited, consider working with collaborators.

54
How to structure your proposal ?

• Work program Make use of a Pert chart to
  illustrate the building blocks work packages
  of the research project. Be detailed in the
  description of the content of each work package
  (why, objectives, method(s), duration, when are
  you going to carry out each WP, partners involved
  in the realization, sequence of WP, etc.).

55
Example of a Pert chart
56
How to structure your proposal ?

• Work program This section contains also a
  diagrammatic work plan, called a Gannt chart.
• The Gannt chart or diagrammatic work plan should
  also be accompanied by a written description.

57
Example of a Gannt chart ( diagrammatic work
plan)
58
How to structure your proposal ?

• Resources The proposal should contain a detailed
  budget.
• The budget asked should be in proportion to the
  volume and complexity of the work activities.
• Be aware that funders vary as to what they are
  prepared to pay in terms of direct project costs,
  such as staff and equipment, and indirect costs,
  such as overheads.
• The funder might request to approve beforehand
  own inputs or inputs from other institutions
  participating in the project.

59
How to structure your proposal ?

• Outcomes, outputs ( deliverables) and
  dissemination In this section one should
  describe the contribution to knowledge and
  importance for future research, the benefits to
  users, and the broader relevance to
  beneficiaries. Highlight how results will be
  disseminated (publications, conferences,
  commercial exploitation, websites, ....).

60
How to structure your proposal ?

• Project management This might not be required
  for small projects.
• However for projects in which several partners
  are involved sufficient information has to be
  provided on how the project will be managed
  (timescales, milestones, communication, criteria
  to measure progress, how crisis situations and
  conflicts will be handled, etc.).

61
Example of Project Organization chart
62
How to structure your proposal ?

• Reviewers Often requested to suggest name of
  referees.
• Choose people who know you and your work
• Dont use reviewers within your own institution
• Use international reviewers and
• Be aware that applicants own referees write
  unfavourable reports.

63
The review process

• Expert assessment Traditionally applications
  will be assessed by 2 to 3 reviewers selected
  from the pool of experts.
• Reviewers will make an independent assessment of
  the scientific quality of the proposal.
• To be selected for funding at least 2 of the 3
  reviewers should provide a positive assessment.

64
The review process

• What are reviewers looking for ?
• High scientific quality
• Proposals that meet the funders priorities or
  fill a knowledge gap
• Novelty and timeliness
• Value for money
• A clear and well thought out approach and
• An interesting idea catch their attention!

65
The review process

• Awards committee Ranks the submitted proposals
  on the basis of the reviewers reports. Their
  operation and procedures can be very variable
  from funder to funder. They might for policy
  reasons of the funder deviate from the reviewers
  assessment.

66
Allocation of funding

• Position in the ranking is important it could
  mean the difference between success and failure.
  Proposals are often ranked into the following
  categories
• Fund
• Fundable
• Invite resubmission (used by some funders) or
• Reject.

67
What next ?

• If the project is retained for funding ? OK.
• If the project is found fundable ? ???
• If invited for resubmission ? revise proposal ?
  feedback from the reviewers panel.
• If rejected, can be very frustrating ? do not
  give up, try to get feedback ? remember it is a
  learning process !

68
Quick for writing a good proposal

• Allow plenty of time
• Start by writing a summary of your proposed
  project
• Demonstrate an up-to-date knowledge of your
  field
• Present your proposal in terms of the aims and
  objectives of the funder
• Avoid jargon say what you mean in clear, simple
  language
• Dont be afraid to state the obvious

69
Quick for writing a good proposal

• Allow a maximum of 4 charts (PERT, GANNT, PROJECT
  ORGANIZATION and BUDGET) - but include as many
  schematic representations of the concepts as
  possible
• Anticipate questions that may arise, before they
  arise
• Ask a colleague to review your proposal and
• Be enthusiastic about your idea if you dont
  sound interested, why should anyone else be ?

70
Funding Sources

• EUROPE
• European Comission (www.cordis.lu) - 4 year
  programmes with identified priorities and
  objectives. Currently Framework 6 - Framework 7
  soon begins
• European Science Foundation (www.esf.org)
• National Funding (www.medc.es, www.mcyt.es)
• US
• National Institute of Health (www.nih.gov)
• DARPA (www.darpa.mil)
• NASA
• Department Of Energy
• Department Of Agriculture

71
Homework

• Draft a proposal of your DEA/PhD project
• The proposal should include
• Title Page
• Table of Contents
• Overall and sub-objectives
• State-of-the-art and novelty of project
• Workplan - divide into workpackages, for each
  WP describe the tasks and sub-tasks, the
  resources required, risk analysis and
  contingency plan, as well as deliverables and
  milestones
• Pert Chart
• Gantt Chart
• Bibliography