CSR Review of Member Conflict Applications

1
CSR Review of Member Conflict Applications

• Working Group Members
• Samuel Edwards, SRA, CMIA Study Section, CHR
• Anshumali Chaudhari, SRA, VCMB Study Section
• Valerie Durrant, SRA, HOP-B Special Emphasis
  Panel
• Marc Rigas, SRA, BDMA Study Section

2
Acknowledgements

• Teresa Lindquist, Program Analyst Office of
  Planning, Evaluation and Analysis, CSR
• QVR Development Team (NIH/CIT), particularly
  Heather Allison and Don Tiedemann
• CSR IRG Chiefs
• Michael Martin, Director, Division of Physiology
  and Pathology, CSR

3
Overview

• What is the Law and what is Policy
• What is Practice
• Demographics of Member SEPs
• Comparison of member applications reviewed in
  Member SEPs and regular study sections with all
  CSR-reviewed applications
• Conclusions and Recommendations

4
Peer Review Regulations

• Revised February 4, 2004
• 42 CFR 52h.5d When a peer review group meets
  regularly, it is assumed that a relationship
  among individual reviewers in the group exists
  and that the group as a whole may not be
  objective about evaluating the work of one of its
  members. In such a case, a members application
  or proposal shall be reviewed by another
  qualified review group to ensure that a competent
  and objective review is obtained
• Does not define what constitutes a different
  review group (previous version stated no more
  than 50 members)

5
CSR Policy

• Member applications can be reviewed in an
  appropriate standing study section or special
  emphasis panel (SEP)
• SEP Composition
• Limited number (lt50) may be current or recent
  members of the conflicted members study section
• Chair of SEP cannot be a member of conflicted
  members study section
• SEP percentiling rules (30 50)

6
What is practice

• Distribution of review of member applications
  (combined 2005 council rounds)

7
The proportion of member applications reviewed in
regular study sections (SRG), member SEPS, and
others have remained relatively constant
8
Survey of IRG chiefs of review of member
applications

• Member applications are reviewed in a variety of
  ways within and across IRGs, mostly sister study
  sections and small phone SEPs.
• 95 of IRGs use multiple formats to review member
  applications.
• Appropriate expertise is the driving factor for
  how applications are reviewed

9
Percent of IRGs that reported using various
formats for the review of member applications
(Note Chiefs could select more than one option)
10
Most IRGs review some or all member applications
in groups (either sister study sections or SEPS)
that do not contain any other current or recent
members of the conflict study section
11
MEM SEPS are held throughout the round with the
peak being 1 month after the peak for regular
study section meetings
12
Most (approx. 90) Member SEPs review 1 5
applications
13
Seniority is similar for Member SEPs and standing
study section meetings
14
Overview

• What is the Law and what is Policy
• What is Practice
• Demographics of Member SEPs
• Comparison of member applications reviewed in
  Member SEPs and regular study sections with all
  CSR-reviewed applications
• Conclusions and Recommendations

15
How do study section member applications fare?

• Examined only R01 applications reviewed by CSR
• CSR Member SEP based on ZRG1 prefix and M suffix
  in the meeting code (plus Decade coding (02 07)
  beginning 2005)
• Compared scores (percentiles) for member
  applications reviewed in Member SEPs and standing
  study sections (SRGs), and to all R01
  applications reviewed in CSR (CSR All).
• Percentage of applications in the
• Top 10
• Top 20
• Bottom 70 (including streamlined applications)
• Differences in Type 1 and 2 applications?
• Differences for applications using human subjects?

16
Comparison of scoring of Types 1 and 2
applications reviewed in 2005 in Member SEPs and
SRG meetings and to CSR All
Type 1 applications
Type 2 applications
Cumulative Percent
Percentile
Percentile
17
Types 1 and 2 applications reviewed in Member
SEPs and Standing Study Section meetings (SRG)
for Years 2000 2005 compared to CSR All
Type 1 applications
Percentage of applications reviewed
0
18
Fewer member applications reviewed in Member SEPs
are streamlined compared to those reviewed in
standing study section (SRG) meetings
19
Types 1 and 2 applications the year prior to
beginning SS service compared to those reviewed
in Member SEPs or study section (SRG) meetings
and to CSR All
Percentage of applications reviewed
20
Comparison of applications reviewed in small SEPs
(Series 90) to member SEPs and to CSR all
21
Conclusions

• Member conflict applications comprise a
  significant segment of applications.
• Lots of members high productivity of members
  many member conflict applications
• Members are providing a service to NIH
• Goal is a fair review for member applications
  given that by regulation they cannot be reviewed
  in their parent study section, which may be the
  most appropriate study section

22
Conclusions Key findings

• No SYSTEMATIC bias in the review of members
  applications and, in particular, with the use of
  member SEPs. 
• Composition (seniority) is similar
• The percentage of Type 2 applications that
  receive scores in the top 10 percentile are very
  similar when reviewed in member SEPS or in
  regular study sections
• Members receive comparable scores before
  and during SS service. 
• Member applications do betterlikely because
  members are excellent scientists who write good
  applications.

23
The current system provides appropriate means for
reviewing member applications

• IRGs differ, study sections differ, and the
  review of member applications will differ
  accordingly.
• Flexibility is important to ensure most
  appropriate review venue for member applications.
  
• IRG chiefs are integral!

24
Concerns with member SEPs

• Confidentiality of reviewers in small SEPs
  adequate protection?
• Designer reviews?
• SRA workload

25
Recent changes

• Encouraged to schedule meetings earlier
• Encouraged to avoid 1 or 2 application meetings