Assessing Study Quality of Exercise Interventions

1
Assessing Study Quality of Exercise Interventions

• Marrissa Martyn-St James
• Carnegie Faculty of Sport and Education

2
Centenary PhD November 2008

• Systematic review and Meta-analysis of exercise
  effects on bone health in women
• Supervisory team Sean Carroll, Cathy Zanker, Ron
  Butterly

3
Systematic Reviews

• A systematic review is an overview of primary
  studies that used explicit and reproducible
  methods a Study of Studies
• Systematic approach to minimize biases and random
  errors
• Always includes materials and methods section
• May include a meta-analysis
• A statistical analysis which combines the results
  of several independent studies considered by the
  analyst to be combinable
• Chalmers and Altman 1994 Huque 1988

4
Stages of a Review

• Formulate review question
• What type of exercise best improves bone density
  in women?
• Locate studies
• Select studies that answer review question
• Extract data
• Analyse and present results
• Data synthesis meta-analysis?
• Interpret results

5
Key limitation in systematic review

• Meta-analyses are fundamentally limited by the
  quality of the underlying studies (the so called
  GIGO principle of garbage in, garbage out).
• Huw Davis, 2001, www.evidence-based-medicine.co.uk

6
What has been included?
The good, the bad and the ugly?
7
Reasons for study quality assessment in reviews

• As a threshold for inclusion of studies
• As a possible explanation for differences in
  results between studies
• In subgroup and analyses
• As weights in statistical analysis of the study
  results
• To guide the interpretation of findings and to
  aid in determining the strength of inferences
• To guide recommendations for future research

8
What is quality?

• Confidence that the studys design, conduct
  analysis and presentation have minimised or
  avoided bias in the intervention comparisons
• Did the trialists do the best they could?
• Should I believe the results?
• Moher 1995

9
Bias in controlled trials drug/placebo trials vs
exercise trials

• RCT
• Potentially eradicates bias by comparing two
  otherwise identical groups
• Selection bias
• biased allocation to comparison groups
• Performance bias
• unequal provision of care apart from treatment
• Detection bias
• biased outcome assessment
• Attrition bias
• Protocol deviations, withdrawals and losses to
  follow up

• Exercise studies very often not randomised
• Exercise studies allowing participants to
  self-select
• Control groups not adequately monitored
• Assessor may know group allocation and round
  up/down values e.g. weight
• High attrition in exercise studies. Differing
  nos. withdraw from groups. Accounted for? Last
  observation carried forward?

10
Tools to assess study quality

• Checklist the components are evaluated
  separately and do not have numerical scores
  attached to them
• Scale each item is scored numerically and an
  overall quality score is generated
• Checklists for appraising primary clinical
  research are now well-established
• Journal of the American Medical Association
  (JAMA) ,Jadad scale, Pedro scale, Oxford Pain
  Validity Scale, CONSORT
• These checklists are around the type of question
  (therapy, prevention, diagnosis, aetiology, or
  prognosis)

11
Instrument widely used in systematic reviews of
exercise effects Jadad. et al (1996) Clin Cont
Trials

• This is a three-item instrument that provides an
  assessment of bias, specifically randomization,
  blinding and withdrawals/dropout. All questions
  are designed to elicit yes (1 point) or no (0
  point) answers. The total number of points
  available ranges from 0 to 5.
• Was the study described as randomised?
• Was the study described as double blind?
• Was there a description of withdrawals and
  dropouts?

12

• It has been shown that studies that obtain 2 or
  less points are likely to produce treatment
  effects which are 35 larger than those produced
  by trials with 3 or more points.

13
Martyn-St James and Carroll (2006) Osteoporos Int
Jadad (1996) quality score ranged from 1 to 3
14
True reflection of exercise study quality?

• There is no true placebo for exercise
• Participants cannot be blind to study group
  allocation
• Treatment providers (exercise instructors) cannot
  be blind to study group allocation
• Points from instrument awarded for allocation and
  blinding therefore redundant

15
Part of PhD

• Development of quality assessment instrument for
  exercise interventions in chronic disease

16
Limitations of Quality Assessment

• inadequate reporting of trials
• limited empirical evidence of a relationship
  between parameters thought to measure validity
  and actual study outcomes

17
Quality of study conduct, or quality of reporting?

• A key difficulty in the assessment of risk of
  bias or quality is the obstacle provided by
  incomplete reporting. While the emphasis should
  be on the risk of bias in the actual design and
  conduct of a study, it can be tempting to resort
  to assessing the adequacy of reporting. Many of
  the tools reviewed by Moher et al. were liable to
  confuse these separate issues (Moher 1995).
  Moreover, scoring in scales was often based on
  whether something was reported (such as stating
  how participants were allocated) rather than
  whether it was done appropriately in the study.
• Cochrane Reviewers Handbook 2008

18
The CONSORT statement

• The CONSORT (Consolidated Standards of Reporting
  Trials) Statement, published in 1996 and revised
  in 2001, is a set of guidelines designed to
  improve the reporting of randomized, controlled
  trials (RCTs)
• The CONSORT statement comprises a 22-item
  checklist and flow diagram, along with some brief
  descriptive text. The checklist items focus on
  reporting how the trial was designed, analyzed,
  and interpreted the flow diagram displays the
  progress of all participants through the trial
• 2008 developed and extension of the CONSORT
  statement for trials of nonpharmalogical
  treatments

19
Development of instrument as part of PhD

• Based on MacDermid (2004) J Hand Therapy used
  in exercise and bone review Zehnacker (2007) J
  Geriat Physic Ther
• Eliminate items redundant in exercise trials
  blinding, care providers
• Include items to measure attempts to minimise
  bias in exercise trials, e.g. all participants
  measured at same time, equal attention to
  follow-up of treatment and controls, etc.
• Reporting in line with extended CONSORT statement
  for nonpharmalogical interventions

20
Outcomes

• Assist reviewers in assessing quality of studies
  selected for inclusion in reviews of exercise
  interventions
• Assist students in critically appraising
  literature
• Assist researchers in reporting of outcomes of
  exercise studies