Rigour of evaluation

1
Rigour of evaluation

• Dr Carole Torgerson
• Senior Research Fellow
• Institute for Effective Education
• University of York

2
A careful look at randomized experiments will
make clear that they are not the gold standard.
But then, nothing is. And the alternatives are
usually worse.
Berk RA. (2005) Journal of Experimental
Criminology 1, 417-433.
3
Characteristics of a rigorous trial

• Once randomised, all participants are included
  within their allocated groups.
• Random allocation is undertaken by an independent
  third party.
• Outcome data are collected blindly.
• Sample size is sufficient to exclude an important
  difference.
• A single analysis is pre-specified before data
  analysis.

4
Education comparison with health education
Torgerson CJ, Torgerson DJ, Birks YF, Porthouse
J. (2005) A comparison of randomised controlled
trials in health and education. British
Educational Research Journal,31761-785. (based
on n 168 trials)
5
Problems with RCTs

• Failure to keep to random allocation can
  introduce selection bias
• Attrition - can introduce selection bias
• Unblinded ascertainment - can lead to
  ascertainment bias
• Small samples - can lead to Type II error
• Multiple statistical tests - can give Type I
  errors
• Poor reporting of uncertainty (e.g., lack of
  confidence intervals)

6
Which are RCTs?

• We took two groups of schools one group had
  high ICT use and the other low ICT use we then
  took a random sample of pupils from each school
  and tested them.
• We put the students into two groups, we then
  randomly allocated one group to the intervention
  whilst the other formed the control
• We formed the two groups so that they were
  approximately balanced on gender and pre-test
  scores
• We identified 200 children with a low reading
  age and then randomly selected 50 to whom we gave
  the intervention. They were then compared to the
  remaining 150.
• Of the eight schools two randomly chosen
  schools served as a control group

7
Mixed allocation

• Students were randomly assigned to either Teen
  Outreach participation or the control condition
  either at the student level (i.e., sites had more
  students sign up than could be accommodated and
  participants and controls were selected by
  picking names out of a hat or choosing every
  other name on an alphabetized list) or less
  frequently at the classroom level

Allen et al, Child Development 199764729-42.
8
Is it randomised?

• The groups were balanced for gender and, as far
  as possible, for school. Otherwise, allocation
  was randomised.

Thomson et al. Br J Educ Psychology
199868475-91.
9
Is it randomised?

• The students were assigned to one of three
  groups, depending on how revisions were made
  exclusively with computer word processing,
  exclusively with paper and pencil or a
  combination of the two techniques.

Greda and Hannafin, J Educ Res 199285144.
10
Non-random assignment confused with random
allocation

• Before mailing, recipients were randomized by
  rearranging them in alphabetical order according
  to the first name of each person. The first 250
  received one scratch ticket for a lottery
  conducted by the Norwegian Society for the Blind,
  the second 250 received two such scratch tickets,
  and the third 250 were promised two scratch
  tickets if they replied within one week.
•  

Finsen V, Storeheier, AH (2006) Scratch lottery
tickets are a poor incentive to respond to mailed
questionnaires. BMC Medical Research Methodology
6, 19.  doi10.1186/1471-2288-6-19.
11
What is the problem here?

• Pairs of students in each classroom were matched
  on a salient pretest variable, Rapid Letter
  Naming, and randomly assigned to treatment and
  comparison groups.
• The original sample those students were tested
  at the beginning of Grade 1 included 64
  assigned to the SMART program and 63 assigned to
  the comparison group.

Baker S, Gersten R, Keating T. (2000) When less
may be more A 2-year longitudinal evaluation of
a volunteer tutoring program requiring minimal
training. Reading Research Quarterly 35,
494-519.
12
What is wrong here?

• the remaining 4 classes of fifth-grade students
  (n 96) were randomly assigned, each as an
  intact class, to the 4 prewriting treatment
  groups

Brodney et al. J Exp Educ 199968,5-20.
13
Misallocation issues

• 23 offenders from the treatment group could not
  attend the CBT course and they were then placed
  in the control group.

14
Independent assignment

• Randomisation by centre was conducted by
  personnel who were not otherwise involved in the
  research project 1
• Distant assignment was used to protect
  overrides of group assignment by the staff, who
  might have a concern that some cases receive home
  visits regardless of the outcome of the
  assignment process2

1 Cohen et al. (2005) J of Speech Language and
Hearing Res. 48, 715-729. 2 Davis RG, Taylor
BG. (1997) Criminology 35, 307-333.
15
Attrition

• Attrition can lead to bias a high quality trial
  will have maximal follow-up after allocation.
• It can be difficult to ascertain the amount of
  attrition and whether or not attrition rates are
  comparable between groups.
• A good trial reports low attrition with no
  between group differences.
• Rule of thumb 0-5, not likely to be a problem.
  6 to 20, worrying, gt 20 selection bias.

16
Poorly reported attrition

• In a RCT of Foster-Carers extra training was
  given.
• Some carers withdrew from the study once the
  dates and/or location were confirmed others
  withdrew once they realized that they had been
  allocated to the control group 117
  participants comprised the final sample
• No split between groups is given except in one
  table which shows 67 in the intervention group
  and 50 in the control group. 25 more in the
  intervention group unequal attrition hallmark
  of potential selection bias. But we cannot be
  sure.

Macdonald Turner, Brit J Social Work (2005)
35,1265
17
What is the problem here?
18
What about matched pairs?

• We can only match on observable variables and we
  trust to randomisation to ensure that unobserved
  covariates or confounders are equally distributed
  between groups.

19
Matched Pairs on Gender
20
Drop-out of 1 girl
21
Removing matched pair does not balance the groups!
22
Blinding of Outcome Assessment

• Ascertainment bias can result when the assessor
  is not blind to group assignment, e.g.,
  homeopathy study of histamine showed an effect
  when researchers were not blind to the assignment
  but no effect when they were.
• Example of outcome assessment blinding Study
  was implemented with blind assessment of outcome
  by qualified speech language pathologists who
  were not otherwise involved in the project1

1 Cohen et al. (2005) J of Speech Language and
Hearing Res. 48, 715-729.
23
ITT analysis examples

• Seven participants allocated to the control
  condition (1.6) received the intervention,
  whilst 65 allocated to the intervention failed to
  receive treatment (15). The authors, however,
  analysed by randomised group - CORRECT approach.
• It was found in each sample that approximately
  86 of the students with access to reading
  supports used them. Therefore, one-way ANOVAs
  were computed for each school sample, comparing
  this subsample with subjects who did not have
  access to reading supports. -INCORRECT

Davis RG, Taylor BG. (1997) Criminology 35,
307-333. Feldman SC, Fish MC. (1991) Journal of
Educational Computing Research 7, 25-36. .
24
The CONSORT guidelines, adapted for trials in
educational research

• Was the target sample size adequately determined?
• Was intention to teach analysis used? (i.e. were
  all children who were randomised included in the
  follow-up and analysis?)
• Were the participants allocated using random
  number tables, coin flip, computer generation?
• Was the randomisation process concealed from the
  investigators? (i.e. were the researchers who
  were recruiting children to the trial blind to
  the childs allocation until after that child had
  been included in the trial?)
• Were follow-up measures administered blind? (i.e.
  were the researchers who administered the outcome
  measures blind to treatment allocation?)
• Was precision of effect size estimated
  (confidence intervals)?
• Were summary data presented in sufficient detail
  to permit alternative analyses or replication?
• Was the discussion of the study findings
  consistent with the data?

25
Flow Diagram

• In health care trials reported in the main
  medical journals authors are required to produce
  a CONSORT flow diagram.
• The trial by Hatcher et al, clearly shows the
  fate of the participants after randomisation
  until analysis.

26
Flow Diagrams
Hatcher et al. 2005 J Child Psych Psychiatry
online
27
(No Transcript)
28
(No Transcript)