Title: Establishing cause and effect in evaluations of PBL (Presentation based on a paper referenced in these slides as Newman (2006))
1Establishing cause and effect in evaluations of
PBL(Presentation based on a paper referenced in
these slides as Newman (2006))
- Does PBL work? Where is the evidence?
- Problem-based learning Special Interest Group
Meeting - 16 November 2006
- Mark Newman
- EPPI-Centre, Social Science Research Unit,
Institute of Education, University of London
2Should advocates of PBL be concerned about
causality ?
- PBL is one of many ways of organizing and
delivering curricula - Advocates of PBL are asking teachers to choose
to use PBL in preference to other approaches - If I am to be persuaded to choose PBL then I need
convincing evidence that using it will help my
students obtain better outcomes than other
approaches
3Yes !
- Convincing evidence needs to demonstrate
- That students following a PBL curriculum obtained
better outcomes - That the cause of this was the PBL curriculum
4So how as a researcher do I demonstrate
descriptive causality ?
- Principles for demonstrating a causal
relationship - There is a temporal order in which cause must
precede effect - There is association that requires that the two
events occur together - There is elimination of alternatives in order to
be able to claim that the effect was due to the
specified intervention and not something else. - Causal relationships are made sense of in terms
of broader theoretical ideas or assumptions. - (Blaikie 2000)
5Threats to validity (bias)
- In practice we claim descriptive causality by
eliminating various threats - Threats to statistical conclusion validity
- Threats to construct validity
- Threats to external validity
- Threats to internal validity
- (Cook Campbell 1979)
6Threats present different type of problem for
researcher
- Threats to statistical conclusion validity
- Threats to construct validity
- Threats to external validity
- Threats to internal validity
Specific from study to study not amenable to
design solutions have to be established
through replication/ argument / theory
Technical amenable to design solutions within a
single study replication will not solve
7What are the threats to internal validity ?
(Newman 2006)
8Designing out threats to internal validity ?
(Newman 2006)
9Evidence for study design making a difference?
- How can we get a handle on this ?
- By comparing evaluations of PBL using different
designs - Difficulties here
- What is PBL
- No comprehensive SRS of PBL
- Any idea?
- PiIot SR by Newman et al (2003)
- More control of bias negative effect sizes
(against PBL) - Less control of bias positive effect sizes (for
PBL) - Smits study series
- As threats to validity eliminated positive
effect sizes in favour of PBL disappeared
10To summarize
- Advocates of PBL are making a causal claim for
its benefits - Establishing descriptive causality requires
control of threats to internal validity - The well designed and conducted randomized
experiment is an efficient method for controlling
many threats to internal validity - Establishing generalised causality requires 1st
that descriptive causality is established and can
then only be demonstrated through further
replication / SRS -
11Implications for the question does PBL work?
- What claims does PBL community / do individual
researchers want to make - Consider the claim we wish to make when designing
studies - Use randomized experiments where possible if we
want to make causal claims - Report studies in exhaustive detail using
existing frameworks for describing PBL
12References
- Blaikie, N. (2000). Designing Social Research.
London Polity Press. - Cook, T. D. Campbell, D. T. (1979).
Quasi-experimentation design and analysis issues
in field settings. Chicago Rand McNally. - Newman, M. (2003). A pilot systematic review and
Meta-analysis on the effectiveness of Problem
Based Learning Newcastle Learning Teaching
Subject Network for Medicine, Dentistry and
Veterinary Medicine. Retrieved 06 March 2006 from
the Higher Education Academy Subject Centre for
Medicine, Dentistry and Veterinary Medicine,
website, Resources, Articles and Reports.
http//www.medev.ac.uk/resources/features/pbl - Newman M (2006) Fitness for purpose evaluation in
Problem Based Learning should consider the
requirements for establishing descriptive
causation. Advances in Health Sciences
Education. 11391-402. - Smits, P., De Buisonje, C., Verbeek, J., van
Dijke, F., Metz, J., J ten Cate, T. (2002). Is
problem-based learning more effective than
lecture-based learning in postgraduate medical
education? A randomized trial. In P.Smits (Ed.),
Effectiveness of postgraduate education in
occupational medicine (pp. 79-96). Amsterdam
Netherlands School of Occupational Health,
University of Amsterdam. - Smits, P., van der weide, W., Verbeek, J.
(2002). Evaluation of a postgraduate educational
programme for occupational physicians on work
rehabilitation guidelines for patients with low
backpain A before and after comparison. In
P.Smits (Ed.), Effectiveness of postgraduate
education in occupational medicine (pp. 57-64).
Amsterdam Netherlands School of Occupational
Health, University of Amsterdam. - Smits, P., Verbeek, J., van Dijke, F., Metz, J.,
J ten Cate, T. (2002). Evaluation of a
postgraduate educational programme for
occupational physicians on work rehabilitation
guidelines for patients with low back pain A
controlled study. In P.Smits (Ed.), Effectiveness
of postgraduate education in occupational
medicine (pp. 67-78). Amsterdam Netherlands
School of Occupational Health, University of
Amsterdam.