Establishing cause and effect in evaluations of PBL (Presentation based on a paper referenced in these slides as Newman (2006))

1
Establishing cause and effect in evaluations of
PBL(Presentation based on a paper referenced in
these slides as Newman (2006))

• Does PBL work? Where is the evidence?
• Problem-based learning Special Interest Group
  Meeting
• 16 November 2006
• Mark Newman
• EPPI-Centre, Social Science Research Unit,
  Institute of Education, University of London

2
Should advocates of PBL be concerned about
causality ?

• PBL is one of many ways of organizing and
  delivering curricula
• Advocates of PBL are asking teachers to choose
  to use PBL in preference to other approaches
• If I am to be persuaded to choose PBL then I need
  convincing evidence that using it will help my
  students obtain better outcomes than other
  approaches

3
Yes !

• Convincing evidence needs to demonstrate
• That students following a PBL curriculum obtained
  better outcomes
• That the cause of this was the PBL curriculum

4
So how as a researcher do I demonstrate
descriptive causality ?

• Principles for demonstrating a causal
  relationship
• There is a temporal order in which cause must
  precede effect
• There is association that requires that the two
  events occur together
• There is elimination of alternatives in order to
  be able to claim that the effect was due to the
  specified intervention and not something else.
• Causal relationships are made sense of in terms
  of broader theoretical ideas or assumptions.
• (Blaikie 2000)

5
Threats to validity (bias)

• In practice we claim descriptive causality by
  eliminating various threats
• Threats to statistical conclusion validity
• Threats to construct validity
• Threats to external validity
• Threats to internal validity
• (Cook Campbell 1979)

6
Threats present different type of problem for
researcher

• Threats to statistical conclusion validity
• Threats to construct validity
• Threats to external validity
• Threats to internal validity

Specific from study to study not amenable to
design solutions have to be established
through replication/ argument / theory
Technical amenable to design solutions within a
single study replication will not solve
7
What are the threats to internal validity ?
(Newman 2006)
8
Designing out threats to internal validity ?
(Newman 2006)
9
Evidence for study design making a difference?

• How can we get a handle on this ?
• By comparing evaluations of PBL using different
  designs
• Difficulties here
• What is PBL
• No comprehensive SRS of PBL

• Any idea?
• PiIot SR by Newman et al (2003)
• More control of bias negative effect sizes
  (against PBL)
• Less control of bias positive effect sizes (for
  PBL)
• Smits study series
• As threats to validity eliminated positive
  effect sizes in favour of PBL disappeared

10
To summarize

• Advocates of PBL are making a causal claim for
  its benefits
• Establishing descriptive causality requires
  control of threats to internal validity
• The well designed and conducted randomized
  experiment is an efficient method for controlling
  many threats to internal validity
• Establishing generalised causality requires 1st
  that descriptive causality is established and can
  then only be demonstrated through further
  replication / SRS

11
Implications for the question does PBL work?

• What claims does PBL community / do individual
  researchers want to make
• Consider the claim we wish to make when designing
  studies
• Use randomized experiments where possible if we
  want to make causal claims
• Report studies in exhaustive detail using
  existing frameworks for describing PBL

12
References

• Blaikie, N. (2000). Designing Social Research.
  London Polity Press.
• Cook, T. D. Campbell, D. T. (1979).
  Quasi-experimentation design and analysis issues
  in field settings. Chicago Rand McNally.
• Newman, M. (2003). A pilot systematic review and
  Meta-analysis on the effectiveness of Problem
  Based Learning Newcastle Learning Teaching
  Subject Network for Medicine, Dentistry and
  Veterinary Medicine. Retrieved 06 March 2006 from
  the Higher Education Academy Subject Centre for
  Medicine, Dentistry and Veterinary Medicine,
  website, Resources, Articles and Reports.
  http//www.medev.ac.uk/resources/features/pbl
• Newman M (2006) Fitness for purpose evaluation in
  Problem Based Learning should consider the
  requirements for establishing descriptive
  causation. Advances in Health Sciences
  Education. 11391-402.
• Smits, P., De Buisonje, C., Verbeek, J., van
  Dijke, F., Metz, J., J ten Cate, T. (2002). Is
  problem-based learning more effective than
  lecture-based learning in postgraduate medical
  education? A randomized trial. In P.Smits (Ed.),
  Effectiveness of postgraduate education in
  occupational medicine (pp. 79-96). Amsterdam
  Netherlands School of Occupational Health,
  University of Amsterdam.
• Smits, P., van der weide, W., Verbeek, J.
  (2002). Evaluation of a postgraduate educational
  programme for occupational physicians on work
  rehabilitation guidelines for patients with low
  backpain A before and after comparison. In
  P.Smits (Ed.), Effectiveness of postgraduate
  education in occupational medicine (pp. 57-64).
  Amsterdam Netherlands School of Occupational
  Health, University of Amsterdam.
• Smits, P., Verbeek, J., van Dijke, F., Metz, J.,
  J ten Cate, T. (2002). Evaluation of a
  postgraduate educational programme for
  occupational physicians on work rehabilitation
  guidelines for patients with low back pain A
  controlled study. In P.Smits (Ed.), Effectiveness
  of postgraduate education in occupational
  medicine (pp. 67-78). Amsterdam Netherlands
  School of Occupational Health, University of
  Amsterdam.