Example: Evaluating School Effectiveness - PowerPoint PPT Presentation

1 / 27
About This Presentation
Title:

Example: Evaluating School Effectiveness

Description:

(C) Sally Thomas. Example: Evaluating School ... The Government's priority is to foster the internal will and capacity of schools ... (C) Sally Thomas ... – PowerPoint PPT presentation

Number of Views:24
Avg rating:3.0/5.0
Slides: 28
Provided by: edw7
Category:

less

Transcript and Presenter's Notes

Title: Example: Evaluating School Effectiveness


1
  • Example Evaluating School Effectiveness
  • in Lancashire LEA using
  • Value Added Measures
  • Dr Sally Thomas
  • Plan
  • UK context
  • What is value added
  • Key issues in school effectiveness research
  • Lancashire LEA Value Added Project
  • Conclusions
  • Further research

2
  • UK CONTEXT
  •  
  • In early 1990s political attention and market
    driven reforms sought to increase the
    accountability of schools in the UK.
  • Since 1992 schools' raw examination performance
    has been published annually by the Department for
    Education and Skills. These league tables have
    been widely criticised as unfair to schools with
    disadvantaged intakes.
  • Other reforms in the last 10 years include a
    national inspection system (OFSTED), a national
    curriculum and national testing via standard
    assessment tasks and examinations.
  • In this context school effectiveness and
    improvement research has been given a far greater
    emphasis than it enjoyed previously.
  • For example, the UK government has now accepted
    the need for value added measures of student
    progress and the importance of school
    self-evaluation has been emphasised.

3
Sir Ron Dearings Report to Secretary of State
for Education (1993) Without a value added
dimension, the obvious basis for judgement is
that higher scores represent better practice
and lower scores worse. This could lead to
unwarranted complacency on the part of some
schools ... and, conversely, to despair on the
part of others ...  
4
Education White Paper 1996 (Page 53) The
Government's priority is to foster the internal
will and capacity of schools to generate their
own improvement ... staff and governors of every
school should feel that it is directly for them
to monitor the quality of the education they
provide and improve schools ...
5
The Autumn Package 1999 DfEE QCA
OFSTED Practice in the classroom is being
sharpened as teachers make considered
professional judgements based on enhanced, more
informative performance data available to
them. Jacqui Smith Parliamentary Under Secretary
of State for Education and Employment
6
(No Transcript)
7
  • What Is Meant by Value Added?
  • The rationale for the value added technique is
    that
  • Raw results describe the grades that students
    have obtained
  •  
  • Raw results do not describe how well a school or
    college has performed
  • Value added results describe how effective a
    school or college is in promoting students'
    achievement

8
  • What Is Meant by Value Added?
  • Value added is a measure of the relative progress
    made by pupils in a school over a particular
    period of time (usually from entry to the school
    until public examinations in the case of
    secondary schools, or over particular years or
    curriculum stages in primary schools) in
    comparison to pupils in others schools in the
    same sample.
  •  
  • It compares outcomes after adjusting for varying
    intake achievement and reflects the relative
    boost a school gives to a pupils previous level
    of attainment in comparison to similar pupils in
    other schools.
  • The concept of value added is, therefore, both an
    indicator of a schools effectiveness and a tool
    for head teachers and their staff to use to
    analyse the extent to which they have effectively
    raised pupil achievement.
  • However, it is not a magic wand. It has real
    limitations, which need to be well understood.

9
  • Value Added Technique can be applied
  • across any phase of education
  • Primary
  • Secondary
  • Post 16
  • The common aim of these studies is to look at
    factors outside the control of the school or
    college (such as students' prior attainment,
    gender, ethnicity and social class) that may have
    an impact on assessment or examination results,
    and where appropriate, to control for these
    factors in the analysis.

10
(No Transcript)
11
  • Methodology Multi-Level Modelling
  • Methodological advances have facilitated the
    development of value added measures of school
    effectiveness. These include Multi-level
    modelling techniques which are considered the
    most accurate and flexible tools for examining
    the hierarchical nature of pupil attainment data
    (Goldstein, 1995).
  •  
  • Essentially these methods are a generalised form
    of multiple regression.
  • This approach allows the statistical analysis to
    compare different models and separate out the
    effect of the school experience on individual
    pupil outcomes (what pupils achieve) and the
    extent to which pupil intake characteristics
    (prior attainment, socio-economic background)
    affect pupil outcomes.
  • Therefore, accurate baseline information about
    pupils' prior attainment is crucial to calculate
    the value added component.
  • Value added measures can also be fine-tuned
    using additional background information about
    pupils, such as their gender, ethnicity and
    social class.
  • Using this approach, the residual or value added
    score for each school in the sample can be
    calculated.

12
  • School Effectiveness Research
  • Some Key Issues have been addressed using
    multilevel modeling
  • Is there evidence of internal variations in
    effectiveness (eg subject/departmental
    effectiveness and differential effectiveness for
    different groups of pupils, different curriculum
    stages)
  • How stable are school effects over time? What are
    the trends in schools raw and value added
    performance?
  • What are the long term effects of schools? Can
    the continuity of previous school effects on
    students later performance be examined? What is
    the impact of pupil mobility?
  • What is the impact of classroom grouping on pupil
    performance?
  • Do national or regional differences exist in
    school effectiveness?
  • See also Scheerens 1992 Teddlie Reynolds
    2000

13
  • The Lancashire Value Added Project
  • The Lancashire value added project was set up in
    1992 and has involved over 130 secondary schools
    and over 140,000 pupils.
  • The project aims to provide an innovative system
    of school evaluation and self-evaluation via the
    feedback of student performance, attitude and
    other data such as teacher and parent attitudes.
  • Crucially, the evaluation process is not intended
    for external accountability purposes, rather a
    confidential tool for internal accountability and
    school improvement.
  • Since the early beginnings, the project has
    expanded to incorporate a number of different
    types of value added evaluation feedback and the
    methodology employs state of the art
    statistical techniques such as multilevel
    modelling.

14
  • The Data Employed for Lancashire Analysis
    (1993-2005)
  • Outcome measures
  • Total GCSE Score (excluding GNVQs)
  • Total GCSE Score (including GNVQs)
  • 5 Best GCSE score
  • English
  • Mathematics
  • Science
  • (Calculated using GCSE grades treated
    numerically, A 8, A 7, B 6, C 5,
  • D 4, E 3, F 2, G 1, Other 0)
  • Are used to create a total of 44 separate value
    added measures including for each outcome
    measure
  • separate measures for each year and a three year
    rolling average
  • separate measures for all pupils and pupils
    grouped according to their previous attainment in
    three bands
  • From 2004 28 new value added scores for seven
    optional GCSE subjects

15
  • The Data Employed for Lancashire Analysis
    (1993-2005)
  •  
  • Variables controlled for in value added analysis
  • Verbal CAT sub-test
  • Quantitative CAT sub-test
  • Non-verbal CAT sub-test
  • Age in months
  • Male or Female
  • Entitlement to free school meals
  • Mobility - attending more than one secondary
    school
  • Ethnicity using DfES categories
  • Special Education Needs

16
(No Transcript)
17
(No Transcript)
18
(No Transcript)
19
(No Transcript)
20
(No Transcript)
21
(No Transcript)
22
  Innovation in School Evaluation The Lancashire
VAP The Case Study School I think it was
really fairly quickly that we began to realise
what a powerful tool that we had got. For first
of all, I suppose selfishly, I was their head and
for me as a management tool it was the one thing
I would never give up, because it has enabled me
to understand so many things better than I have
understood before, and to deal with them, in a
more dynamic and positive way (Head teacher)
23
  • Limitations of Value Added Methodology
  • Importance of statistical significance and
    confidence intervals when making comparisons
    between schools and departments.
  • Data accuracy and measurement error.
  • Validity of student outcome measures.
  • Validity of background factors. (e.g. How well
    can we control for factors outside the control of
    the school, such as additional private tuition?)
  • Retrospective nature of the data.
  • Finally, are we measuring what we value or
    valuing what we measure? . it is important to
    emphasise that value added measures of student
    progress are not the only criteria that should be
    employed to judge school effectiveness, other
    more qualitative measures of school processes (eg
    quality of teaching) are also vital for this
    purpose.

24
A Critique of SESI A mechanistic methodology,
an instrumentalist view of educational processes
and the belief that educational outcomes can and
should be described independently of such
processes. Elliot (1996, pg. 200)
25
  • Conclusions
  • The importance of taking account of background
    factors and prior attainment using appropriate
    value added models in order to estimate the
    influence of the school is clear. Current
    evidence points to the importance of schools
    utilising value added information as part of a
    framework for self-evaluation, alongside other
    data or evidence (such as that provided in the
    DFES autumn package or via pupil, teacher and
    parent questionnaires).
  •  
  • However, the evidence also shows that
    effectiveness is best seen as a feature that is
    outcome and time specific. Therefore judgements
    about schools need to address at least five key
    questions
  • Effective in promoting which outcomes?
  • Effective over what period of time?
  • Effective for whom?
  • Effective for which curriculum stage?
  • Effective in what educational policy or regional
    context?

26
  • Further Research Is Needed to
  • Examine the Nature and Dimensions of School
    Effectiveness and Improvement
  •  
  • Develop a broader range of outcomes (e.g.,
    measures of pupil affective and vocational
    outcomes in addition to academic outcomes).
  • Examine what school characteristics or processes
    are likely to enhance a broader range of
    outcomes.
  • Examine limitations of the data. For example, how
    well can we control for factors outside the
    control of the school such as private tutoring?
  •  
  • Analyse the national value added databases to
    further examine the patterns, conditions and
    constraints of school effectiveness and
    improvement across a range of regional, policy or
    socio-economic contexts.
  • Explore the relationship between effectiveness of
    different levels of education system (e.g.,
    national, regional, local, school, department,
    classroom, individual).

27
  • Further Research Is Needed to
  • Examine the Implications for Policy and Practice
  • Examine the long-term impact of accountability
    and school self-evaluation processes on the
    quality of teaching and learning?
  • Examine what conditions appear to enhance, or
    alternatively form barriers, to school
    effectiveness?
  • Examine the strategies, or levers, appear to
    improve school effectiveness? For example, what
    is the affect of providing feedback data to
    schools on their effectiveness?
  • Examine the impact of value added approaches on
    organisational culture and learning, assessment
    policy and practices or equal opportunities.
  • Explore the application of value added
    methodology to broader issues in educational
    research and evaluation.
Write a Comment
User Comments (0)
About PowerShow.com