1
                                                
                                                
I think you should be more explicit here in step
two...''
2
INTRODUCTION TO RESEARCH DESIGN Stephen
Gorard s.gorard_at_bham.ac.uk
http//www.tlrp.org/capacity/rcbn.html http//tri
als-pp.co.uk http//streamingportal.multistream.c
o.uk/ngflcymru/eng_webstream_pres02.htm
3
The argument for research design A key ethical
concern for those funding, conducting or using
publicly-funded social science research must be
the quality of the research, and the security of
the conclusions drawn This approach to research
ethics calls for appropriate scepticism to be
used more widely the warranting
principle Traditional research methods training
and experts are not enhancing quality in this
warranting way. Working towards an ideal design
can be an important part of any research
enterprise, even where that design is not
possible It is helpful to consider the research
enterprise as a cycle of complementary phases and
activities, because this illustrates how all
methods can have an appropriate place in the full
cycle of research
4
Reay, D. and Lucey, H. (2000) Children, school
choice and social differences, Educational
Studies, 26, 1, 83-100
Interviews involved 15 parents of primary school
children (p.85). a majority of the
working-class parents concurred with their child
a significant deviation from this class trend was
mothers of mainly black working-class boys
(p.90).
5
Reay, D. and Lucey, H. (2000) Children, school
choice and social differences, Educational
Studies, 26, 1, 83-100
Interviews involved 15 parents of primary school
children (p.85). a majority of the
working-class parents concurred with their child
a significant deviation from this class trend was
mothers of mainly black working-class boys
(p.90). Class? working-class and at least
one other Sex of parent? male/female Ethnicity?
black and at least one other Sex of child?
male/female If we assume equal distribution of
categories. majority of the working-class
parents could be 4 cases, mothers of mainly
black working-class boys could not even be one
case.
6
International comparisons indicate that
intergenerational mobility in Britain is of the
same order of magnitude as in the US, but that
these countries are substantially less mobile
than Canada and the Nordic countries. Intergenera
tional mobility fell markedly over time in
Britain, with there being less mobility for a
cohort of people born in 1970 compared to a
cohort born in 1958.
Source Blanden, Machin and Gregg (2005)
Intergenerational mobility in Europe and North
America, London Centre for Economic
Performance, p.2
'The comprehensive system was brought in to try
to improve social mobility, but the opposite has
happened' Sir Peter Lampl (http//www.suttontrust
.com/press068.asp).
According to the LSE, the extra places made
available by the expansion of universities in the
Eighties and Nineties have largely been filled by
students from better-off backgrounds
(http//www.suttontrust.com/press068.asp).
7
Internationally comparable estimates of
intergenerational mobility
Country Sons born Sons earning measure Measure of parental status Partial correlation
Britain 1970 Age 30 Average parental 19801986 .271
Norway 1958 Average 19921999 Father 1974 .139
Denmark 1958-1960 Average 19982000 Father 1980 .143
Sweden 1962 Average 19961999 Father 1975 .143
Finland 1958-1960 Average 19952000 Father 1975 .147
Source Blanden, Machin and Gregg (2005)
Intergenerational mobility in Europe and North
America, London Centre for Economic
Performance, p.6
8
Table 3 Transition Matrix for Britain, Sons Born
in 1958 Sons earnings quartile aged 33 in
1991 Bottom 2nd 3rd Top Bottom parent
quartile (when son aged 16) .31 .28 .23 .17
Table 4 Transition Matrix for Britain, Sons Born
in 1970 Sons earnings quartile aged 30 in
2000 Bottom 2nd 3rd Top Bottom parent
quartile (when son aged 16) .38 .25 .21 .16

Source Blanden, Machin and Gregg (2005)
Intergenerational mobility in Europe and North
America, London Centre for Economic
Performance, p.8
9
The format of a warrant as a syllogism Observ
ation/Data This is an A Conclusion Therefore,
this is B
10
The format of a warrant as a syllogism Explicit
Warrant All As are B Observation/Data This
is an A Conclusion Therefore, this is B
11

• An effect of education?
• p.238 The data are based on the entire
  population of schools Consequently, the results
  can be considered as representative of the whole
  population of pupils who take both KS3 and GCSE
  exams.
• Multi-level model of the effect of single-sex
  teaching.
• p.246 The effect of school type is highly
  significant
• Malacova, E. (2007) Effects of single-sex
  education on progress in GCSE, Oxford Review of
  Education, 33, 2, 233-259

12

• Contradictory trends
• Between 1994 and 2004, reports of studies using a
  researcher manipulated intervention declined from
  45 to 33 in the US.
• But causal statements in non-intervention studies
  grew from 34 to 43.
• In particular, complex statistical approaches
  such as HLM (MLM) and structural equation
  modelling were routinely misunderstood by
  researchers as testing causation.
• Robinson, D., Levin, J., Thomas, G., Pituch, K.
  and Vaughn, S. (2007) The incidence of causal
  statements in teaching-and-learning research
  journals, American Educational Research Journal,
  44, 2, 400-413

13

• Impact of feedback?
• Using national data and multi-level modelling to
  compare the progress of schools given performance
  feedback by NFER.
• p.102 Although no actual samples have been
  drawn Statistical checks were carried out and no
  significant difference between the groups was
  found i.e. in existing background measures.
• p.103 Schools who participated in PASS showed
  a significant difference (plt0.05) in attainment
  compared to those who received feedback as part
  of another project.
• Hammond, P. and Yeshanew, T. (2007) The impact of
  feedback on school performance, Educational
  Studies, 33, 2, 99-113

14

• A school effect?
• Looks at successive years of positive value-added
  scores in one England district, 1993-2002.
• An improving school is one where VA increases in
  successive years
• p.271- the pupil intake and time trend
  explanatory variables included in the fixed part
  of the value-added model (Model A) were
  statistically significant (at 0.05 level).
• p.261 it appears that only one in 16 schools
  managed to improve continuously for more than
  four years at some point over the decade in terms
  of value-added.
• Thomas, S., Peng, WJ. And Gray, J. (2007)
  Modelling patterns of improvement over time
  value-added trends in English secondary school
  performance across ten cohorts, Oxford Review of
  Education, 33, 3, 261-295

15
Result of academisation? DfES - We expect that
all Academies will make steady upward progress
Good teaching, excellent facilities and motivated
pupils will deliver real improvements in
educational standards. BBC - The government has
released GCSE figures from three of its new
flagship Academies in England. All the schools,
which were set up in deprived areas, showed
remarkable improvements in results.
Bexley Business Academy 1997 1998 1999 2000 2001 2002 2003
Percentage eligible for FSM 53 49 52 50 49 46 42
GCSE points per candidate - 23 20 20 22   23
16
Achievement and attainment tables
Back to the Department for Children, Schools and
Families LA Nottingham, City of   Greenwood
Dale School Sneinton Boulevard Nottingham
Nottinghamshire NG2 4GL 0115 9103200 MIXED
Key Stage 2 to 4 Contextual Value Added
Measure Key Stage 2 to 4 contextual value added
measure 1068.6 Upper limit of CVA confidence
interval 1077.2 Lower limit of CVA confidence
interval 1059.9 Coverage indicator - of
students included in Key Stage 2 to 4 CVA 93
17
Sources of errors in CVA
Missing data - examples over 15 of KS4 PLASC
missing one of FSM, sex, ethnicity over 5 of
KS4 NPD have no equivalent for KS2 over 10 of
PLASC records unmatched in NPD Incorrect data
examples coding (neither male nor
female) entry/transcription storage
(dropout) storage (binary conversion) Measuremen
t error examples IDACI and the ecological
fallacy Validity of assessment, marking and
moderation Definitions such as SEN, ethnicity,
or first language
18
Propagation of errors in CVA
Imagine a pupil with a predicted KS4 of 100 and
an actual score of 110 Imagine that all scores
are only 90 correct a conservative view The
predicted score really lies between 90 and
110 The actual score really lies between 99 and
121 CVA calculates a residual of 10 The real
residual is between -11 and 31 We do not even
know if the pupil has done better or worse than
expected
19
The link between KS2 points and GCSE benchmark,
secondary schools in England, 2004 Correlation of
0.87
Source Gorard, S. (2006) Value-added is of
little value, Journal of Educational Policy, 21,
2, 233-241
20
The link between GCSE benchmark and DfES KS2 to
GCSE value-added, secondary schools in England,
2004 Correlation of 0.84
Source Gorard, S. (2006) Value-added is of
little value, Journal of Educational Policy, 21,
2, 233-241
21
Guess what?

• Secondary data analysis
• Feminist theory and approaches
• The role of theory
• Ethnogaphy
• Working overseas
• Adult learning
• Discourse analysis
• .

22
Outline of a full cycle of research