Title: Example: Evaluating School Effectiveness
1- Example Evaluating School Effectiveness
- in Lancashire LEA using
- Value Added Measures
- Dr Sally Thomas
- Plan
- UK context
- What is value added
- Key issues in school effectiveness research
- Lancashire LEA Value Added Project
- Conclusions
- Further research
2 - UK CONTEXT
-
- In early 1990s political attention and market
driven reforms sought to increase the
accountability of schools in the UK. - Since 1992 schools' raw examination performance
has been published annually by the Department for
Education and Skills. These league tables have
been widely criticised as unfair to schools with
disadvantaged intakes. - Other reforms in the last 10 years include a
national inspection system (OFSTED), a national
curriculum and national testing via standard
assessment tasks and examinations. - In this context school effectiveness and
improvement research has been given a far greater
emphasis than it enjoyed previously. - For example, the UK government has now accepted
the need for value added measures of student
progress and the importance of school
self-evaluation has been emphasised.
3Sir Ron Dearings Report to Secretary of State
for Education (1993) Without a value added
dimension, the obvious basis for judgement is
that higher scores represent better practice
and lower scores worse. This could lead to
unwarranted complacency on the part of some
schools ... and, conversely, to despair on the
part of others ...
4 Education White Paper 1996 (Page 53) The
Government's priority is to foster the internal
will and capacity of schools to generate their
own improvement ... staff and governors of every
school should feel that it is directly for them
to monitor the quality of the education they
provide and improve schools ...
5The Autumn Package 1999 DfEE QCA
OFSTED Practice in the classroom is being
sharpened as teachers make considered
professional judgements based on enhanced, more
informative performance data available to
them. Jacqui Smith Parliamentary Under Secretary
of State for Education and Employment
6(No Transcript)
7- What Is Meant by Value Added?
- The rationale for the value added technique is
that - Raw results describe the grades that students
have obtained -
- Raw results do not describe how well a school or
college has performed - Value added results describe how effective a
school or college is in promoting students'
achievement
8- What Is Meant by Value Added?
- Value added is a measure of the relative progress
made by pupils in a school over a particular
period of time (usually from entry to the school
until public examinations in the case of
secondary schools, or over particular years or
curriculum stages in primary schools) in
comparison to pupils in others schools in the
same sample. -
- It compares outcomes after adjusting for varying
intake achievement and reflects the relative
boost a school gives to a pupils previous level
of attainment in comparison to similar pupils in
other schools. - The concept of value added is, therefore, both an
indicator of a schools effectiveness and a tool
for head teachers and their staff to use to
analyse the extent to which they have effectively
raised pupil achievement. - However, it is not a magic wand. It has real
limitations, which need to be well understood.
9- Value Added Technique can be applied
- across any phase of education
- Primary
- Secondary
- Post 16
- The common aim of these studies is to look at
factors outside the control of the school or
college (such as students' prior attainment,
gender, ethnicity and social class) that may have
an impact on assessment or examination results,
and where appropriate, to control for these
factors in the analysis.
10(No Transcript)
11- Methodology Multi-Level Modelling
- Methodological advances have facilitated the
development of value added measures of school
effectiveness. These include Multi-level
modelling techniques which are considered the
most accurate and flexible tools for examining
the hierarchical nature of pupil attainment data
(Goldstein, 1995). -
- Essentially these methods are a generalised form
of multiple regression. - This approach allows the statistical analysis to
compare different models and separate out the
effect of the school experience on individual
pupil outcomes (what pupils achieve) and the
extent to which pupil intake characteristics
(prior attainment, socio-economic background)
affect pupil outcomes. - Therefore, accurate baseline information about
pupils' prior attainment is crucial to calculate
the value added component. - Value added measures can also be fine-tuned
using additional background information about
pupils, such as their gender, ethnicity and
social class. - Using this approach, the residual or value added
score for each school in the sample can be
calculated.
12- School Effectiveness Research
- Some Key Issues have been addressed using
multilevel modeling - Is there evidence of internal variations in
effectiveness (eg subject/departmental
effectiveness and differential effectiveness for
different groups of pupils, different curriculum
stages) - How stable are school effects over time? What are
the trends in schools raw and value added
performance? - What are the long term effects of schools? Can
the continuity of previous school effects on
students later performance be examined? What is
the impact of pupil mobility? - What is the impact of classroom grouping on pupil
performance? - Do national or regional differences exist in
school effectiveness? - See also Scheerens 1992 Teddlie Reynolds
2000
13- The Lancashire Value Added Project
- The Lancashire value added project was set up in
1992 and has involved over 130 secondary schools
and over 140,000 pupils. - The project aims to provide an innovative system
of school evaluation and self-evaluation via the
feedback of student performance, attitude and
other data such as teacher and parent attitudes. - Crucially, the evaluation process is not intended
for external accountability purposes, rather a
confidential tool for internal accountability and
school improvement. - Since the early beginnings, the project has
expanded to incorporate a number of different
types of value added evaluation feedback and the
methodology employs state of the art
statistical techniques such as multilevel
modelling.
14- The Data Employed for Lancashire Analysis
(1993-2005) - Outcome measures
- Total GCSE Score (excluding GNVQs)
- Total GCSE Score (including GNVQs)
- 5 Best GCSE score
- English
- Mathematics
- Science
- (Calculated using GCSE grades treated
numerically, A 8, A 7, B 6, C 5, - D 4, E 3, F 2, G 1, Other 0)
- Are used to create a total of 44 separate value
added measures including for each outcome
measure - separate measures for each year and a three year
rolling average - separate measures for all pupils and pupils
grouped according to their previous attainment in
three bands - From 2004 28 new value added scores for seven
optional GCSE subjects
15- The Data Employed for Lancashire Analysis
(1993-2005) -
- Variables controlled for in value added analysis
- Verbal CAT sub-test
- Quantitative CAT sub-test
- Non-verbal CAT sub-test
- Age in months
- Male or Female
- Entitlement to free school meals
- Mobility - attending more than one secondary
school - Ethnicity using DfES categories
- Special Education Needs
16(No Transcript)
17(No Transcript)
18(No Transcript)
19(No Transcript)
20(No Transcript)
21(No Transcript)
22 Innovation in School Evaluation The Lancashire
VAP The Case Study School I think it was
really fairly quickly that we began to realise
what a powerful tool that we had got. For first
of all, I suppose selfishly, I was their head and
for me as a management tool it was the one thing
I would never give up, because it has enabled me
to understand so many things better than I have
understood before, and to deal with them, in a
more dynamic and positive way (Head teacher)
23- Limitations of Value Added Methodology
- Importance of statistical significance and
confidence intervals when making comparisons
between schools and departments. - Data accuracy and measurement error.
- Validity of student outcome measures.
- Validity of background factors. (e.g. How well
can we control for factors outside the control of
the school, such as additional private tuition?) - Retrospective nature of the data.
- Finally, are we measuring what we value or
valuing what we measure? . it is important to
emphasise that value added measures of student
progress are not the only criteria that should be
employed to judge school effectiveness, other
more qualitative measures of school processes (eg
quality of teaching) are also vital for this
purpose.
24A Critique of SESI A mechanistic methodology,
an instrumentalist view of educational processes
and the belief that educational outcomes can and
should be described independently of such
processes. Elliot (1996, pg. 200)
25- Conclusions
- The importance of taking account of background
factors and prior attainment using appropriate
value added models in order to estimate the
influence of the school is clear. Current
evidence points to the importance of schools
utilising value added information as part of a
framework for self-evaluation, alongside other
data or evidence (such as that provided in the
DFES autumn package or via pupil, teacher and
parent questionnaires). -
- However, the evidence also shows that
effectiveness is best seen as a feature that is
outcome and time specific. Therefore judgements
about schools need to address at least five key
questions - Effective in promoting which outcomes?
- Effective over what period of time?
- Effective for whom?
- Effective for which curriculum stage?
- Effective in what educational policy or regional
context?
26- Further Research Is Needed to
- Examine the Nature and Dimensions of School
Effectiveness and Improvement -
- Develop a broader range of outcomes (e.g.,
measures of pupil affective and vocational
outcomes in addition to academic outcomes). - Examine what school characteristics or processes
are likely to enhance a broader range of
outcomes. - Examine limitations of the data. For example, how
well can we control for factors outside the
control of the school such as private tutoring? -
- Analyse the national value added databases to
further examine the patterns, conditions and
constraints of school effectiveness and
improvement across a range of regional, policy or
socio-economic contexts. - Explore the relationship between effectiveness of
different levels of education system (e.g.,
national, regional, local, school, department,
classroom, individual).
27- Further Research Is Needed to
- Examine the Implications for Policy and Practice
- Examine the long-term impact of accountability
and school self-evaluation processes on the
quality of teaching and learning? - Examine what conditions appear to enhance, or
alternatively form barriers, to school
effectiveness? - Examine the strategies, or levers, appear to
improve school effectiveness? For example, what
is the affect of providing feedback data to
schools on their effectiveness? - Examine the impact of value added approaches on
organisational culture and learning, assessment
policy and practices or equal opportunities. - Explore the application of value added
methodology to broader issues in educational
research and evaluation.