Questionable Research Practices: The Initial Development of a Measure

1
Questionable Research Practices The Initial
Development of a Measure

• Kelly Wester, PhD
• University of North Carolina at Greensboro
• Mark S. Davis, PhD
• Justice Research Advocacy, Inc
• John Willse, PhD
• University of North Carolina at Greensboro

2
Research Misconduct

• Up to 44 of students and 50 of faculty report
  being exposed to at least 1 type of research
  misconduct (Shafir Kennedy, 1998)
• In 2005, one of the first researchers was barred
  from life from receiving federal funding (ORI,
  2005)
• A record of institutions reported a record of
  research misconduct cases in 2006 (ORI, 2006)

3
Questionable Research Practices

• Research misconduct plagiarism, falsification
  and fabrication
• Questionable Research Practices (QRP) departing
  from acceptable research practice of the relevant
  research community (Steneck, 2003)
• QRPs are thought to be more prevalent than
  research misconduct, estimates between 10 to 40
  in research communities (Davis, Wester, King,
  in press Steneck, 2003)
• Statistical errors
• Improper authorship
• Duplicate publications
• Sloppy or careless research
• QRPs can be more damaging to the research
  community (Alberts Shine, 1994 DeVries et al.,
  2006)

4
Measuring Questionable Research Practices

• A lack of instruments to collect data on
  Questionable Research Practices exist
• Measures exist to measure research misconduct
  (e.g.,)
• asking what an individual has seen others do
  (e.g., Anderson et al., 1994)
• Scientific Misconduct Questionnaire-Revised
  (Broome et al., 2005) 68 item questionnaire
  asks to report on others behaviors and those
  behaviors participant has observed
• Two studies did examine QRPs by researchers
• Davis et al. (in press) 8 research vignettes in
  paragraph format with likert-type scale of
  likelihood to engage in stated behavior
• Martinson et al. (2006) Asked IRBs the top 10
  misbehaviors/QRPs more likely to be sanctioned
  Then asked researchers if engaged in behavior
  yes/no response

5
Development of the Responsible Conduct of
Research
Measure (RCR-M)

• Original measure
• 96 behavioral statements written to correspond to
  8 of the 9 areas of ORIs RCR areas
• Data acquisition, management, sharing,
  ownership
• Mentor/trainee relationships
• Publication practices and responsible authorship
• Peer review
• Collaborative science
• Human subjects
• Research misconduct
• Conflict of interest and commitment
• 6 point Likert-type scale (1 extremely unlikely
  to 6 extremely likely)
• Reason for measuring likelihoodof engaging in
  QRP is due to
• Mixed results of self-reported, socially
  disapproved behavior (e.g., Del Boca Noll,
  2000 Bolub et al., 2002)
• Due to difficulty in measuring QRP and research
  misconduct (Anderson, 1996)
• That most researchers will report others
  behaviors and not their own (Wiles et al., 2006)

6
Pilot Studies of the RCR-M

• A total of 3 pilot studies were conducted
• Each in the behavioral and social sciences across
  multiple professions
• e.g., psychology, counselor education, political
  science, criminal justice
• Studies were conducted at private,
  Carnegie-designated research institutions

7
Pilot Studies 1 2 on the RCR-M

• Randomly selected participants from private
  research universities
• Social/behavioral science departments
• Surveys sent via email with RCR-M and
  demographics
• Pilot study 1
• 96 items were grouped into subscales based on the
  RCR areas
• Classical test theory (CTT) item analyses were
  conducted
• Small sample sizes (N30 with 14 to 25 valid
  responses per item)
• Items with low variance or small item-total
  correlations were reviewed and most were deleted
  from RCR-M
• Items were retained if they appeared central to
  the construct of interest
• Pilot Study 2
• 75 items were re-tested with a new sample
• A combination of CTT and principal components
  analysis (PCA) with promax rotation was used
• No mechanistic rules were applied for deciding on
  whether or not to drop items
• The most coherent interpretation of the PCA was
  based on a single-factor model (42 items,
  Cronbachs Alpha .89)

8
Third Final Pilot Study of RCR-M

• The final revision of the RCR-M was piloted in
  Summer/Fall 2006
• The survey was emailed to 1,867 participants
  (with 6 of emails being undeliverable)
• Participants were selected based on being a
  faculty member within a specified department
  (i.e., psychology, counselor education, special
  education, and criminal justice) and were in a
  research intensive private university
• 155 respondents answered every question
• The PCA analysis produced mixed results regarding
  dimensionality
• There was evidence that additional factors may
  exist but the clearest interpretation remained at
  the level of the entire scale
• The measure demonstrated acceptable reliability
  (a.84) with this sample
• This results provided verification of the scale
  properties and suggested that we have developed a
  reliable overall measure of responsible conduct
  of research

9
Limited Validity Evidence

• Analyses related to validity have not been
  examined at this point
• Convergent or divergent validity has not been
  examined is due to the lack of measures that
  assess research integrity, questionable research
  practices, or research misconduct
• One aspect of divergent validity has been
  examined
• We examined the relationship between RCR-M and
  social desirability
• Social desirability was measured by
  Marlowe-Crownes Personnel Reaction Blank short
  form (Reynolds, 1982)
• Social desirability was not significantly related
  to RCR-M (r.037, pgt.05)
• This result suggests that social desirability was
  not associated with responses on the RCR-M (or
  questionable research practices or research
  integrity)

10
Hypothesis Guessing? Sensitive Nature of QRP?

• When examining individual item responses, the
  results of 3rd pilot study indicated
• While the majority of respondents indicated they
  would NOT depart from RCR(a few items and
  responses include)
• 1.6 indicated they WOULD be willing to report
  non-significant results as significant in order
  to get their study published
• 1 indicated they would be willing to falsify
  data from one of their studies when writing a
  grant to increase funding chances
• 4 indicated they would be extremely unlikely
  3.5 were very/somewhat unlikely to remove
  human participants identifying information
  before sharing data with an outside researcher
• 5 indicated they would be extremely/very
  likely to insist on senior authorship on a
  students/associates paper that is jointly
  submitted for publication
• 5.8 indicated they would be very likely to
  submit multiple publications that cover the same
  research questions from the same dataset in order
  to lengthen their vita for tenure/promotion
• 3.7 reported extremely/very likely to leave
  important information out of methodology section
  of a manuscript to stay within page limitations
  of journal