Datadriven school improvement through the VCE Data Service

1
Data-driven school improvement through the VCE
Data Service

• A workshop provided for the conference
• Schools as Learning Organisations Measuring
  performance to maximise educational
  effectiveness.
• March 9-10, 2004
• Glenn Rowley
• Victorian Curriculum and Assessment Authority

2
What is the VCE Data Service?

• The VCE Data Service (VCEDS)
• is an online service provided by the VCAA to all
  schools through the VASS system
• is a reporting system that aims to assist schools
  with their school improvement initiatives
• provides a variety of reports that allow schools
  to review assessment performance data in
  different ways.

3
The Origins of the VCE Data Service

• VCEDS was developed in 2002 following an
  extensive R D trial with 50 schools since the
  late 1990s (the VCE Data Project).
• The VCE Data Project explored new ways of feeding
  school data back into schools for their use in
  guiding school improvement efforts.
• VCEDS was fully implemented in January 2003, when
  the 2002 VCE data were loaded into the system.

4
How does VCEDS work?

• VCEDS
• is an online service to schools.
• is secure.
• is accessed only through the VASS system.
• provides access to your school data and only
  your schools data.

5
Who has access to VCEDS?

• Each VASS coordinator has access to VCEDS through
  the opening menu of the VASS system (look under
  School Administration).
• The VASS coordinator can grant access to other
  users with the approval of the Principal.
• These users have access to VCEDS only not the
  rest of the VASS system.
• Each user has his/her own security disk, which
  cannot be shared.

6
What does VCEDS provide?

• VCEDS provides the facility for school staff to
  conduct online analyses of their own school VCE
  data, for the current year, or for the last six
  years.
• VCEDS was designed to produce all of the reports
  previously available through DET, plus many
  more.
• All VCEDS reports are easily sent to the printer
  or pasted into other documents.

7
What reports does VCEDS provide?

• Combined Study reports
• Study Scores
• Assessment grades
• Completion rates
• Individual Study reports
• Study Scores
• Estimates adjusted for the ability and gender
  distribution of your school cohort

8
Comparisons external to your school

• In each report, you can specify what comparisons
  you would like to receive
• school average results with average results for
  All schools
• school average results with average results for
  Like schools.
• school average results with average results for
  Schools in your sector.
• school average results with average results for
  Like schools in your sector.

9
Comparisons internal to your school

• In every study, you can compare results
• By gender,
• By class grouping,
• By year (1998-2003 inclusive),
• By any subgroups in the school for which you can
  provide data (e.g. campus, language background,
  year level).

10
Unique to VCEDS Adjusted Estimates

• The VCE Data Project established clearly that,
  other things being equal, you can expect higher
  levels of performance from a student who
• has greater ability
• is in a class with more able students
• is female.

11
Adjusted Estimates

• VCEDS calculates what each of your students could
  have been expected to get on the basis of
• their ability, as indicated by their GAT scores
• the average GAT performance of the class group
  they are in and
• their gender.
• It then determines how well your students have
  done, compared to how well they could have been
  expected to do.
• The difference is called an adjusted estimate.

12
How does VCEDS report Adjusted Estimates?

• If students performed precisely as predicted in a
  given study taking into account their abilities
  and their gender composition, then the school
  will have an estimate of zero.
• If the students perform less well than predicted,
  the school will have a negative estimate.
• If students perform better than predicted, the
  school will have a positive estimate.

13
What sort on numbers do the Adjusted Estimates
produce?

• The scale on which adjusted estimates are
  reported is arbitrary.
• To make them interpretable, they have been placed
  on a scale that extends from 10 to 10.
• In each study, 10 is the adjusted estimate for
  the highest-scoring school, and 10 is the
  adjusted estimate of the highest-scoring school.

14
Why the GAT?

• The GAT provides measures of the general
  knowledge and skills that students have built up
  during their school years.
• It provides an indication of how well students
  are equipped to undertake VCE studies.
• Data show it to be a good predictor of VCE
  performance and can be used as an indicator of
  where students would be in their VCE studies,
  other things being equal.

15
Are Adjusted Estimates the same as value-added
scores?

• Statistically Yes. Value-added scores assess
  increase or growth over time, and are calculated
  in the same way.
• Substantively No. The baseline measure (the
  GAT) is not a measure of achievement as are Study
  Scores.
• There is no suggestion that these adjusted
  estimates measure growth over time. They measure
  achievement in excess of expectation.

16
Reporting Constraints

• VCEDS does not report
• Study score statistics and assessment grades
  where studies have less than 10 students
• Ability and Gender Adjusted estimates where there
  is
• less than 400 total enrolments with complete data
  
• less than 20 schools with enrolments
• less than 5 students in the school
• insufficient correlation between study and GAT
  scores

17
Reporting percentile distributions

• E.g.. The 25th percentile is the score below
  which 25 of cases lie.

Max score
75th percentile
50th percentile
25th percentile
Min score
18
Reporting adjusted estimates

• All ability and gender adjusted estimates are
  associated with a degree of uncertainty. This
  uncertainty can be quantified and a confidence
  interval can be constructed around each estimate.

19
Confidence Intervals

• Confidence intervals are displayed by means of
  error bars around the estimates. They have
  been set so that schools are judged to have
  significantly different ability and gender
  adjusted estimates (at the 5 significance level)
  only if their error bars do not overlap.

20
Confidence Intervals- Example

Estimates
Error Bars
21
Entrée to VCEDS Combined studies
22
Entrée to VCEDS Individual studies
23

• All VCE Studies Combined ReportTest College

24
All VCE Studies Combined ReportTest College
25
Each report is provided in table form as well as
graphical form
26
Frequency of Assessment Grades across Studies
27
Frequency of Assessment Grades across Studies
28
Completed of Studies Undertaken
29
Completed of Studies Undertaken Report
30
Like School Group
31
(No Transcript)
32
Single VCE Study (Study Scores)
33
English Study Scores2003 Test College
34
Analysis of English Study Scores by within-school
subgroup
35
English Study Scores over time
36
Selected VCE Studies 2003
37
Selected VCE Studies Adjusted Estimates
38
Physics Adjusted Estimates over time
39
Biology Adjusted Estimates over time
40
Business Management Study Scores
41
Business Management - Adjusted Estimates
42
Trend data Legal Studies over time
43
Adjusted scores Legal Studies over time
44
A strategy for approaching VCEDS

• Look for high level trends
• Look at study scores for a Year
• Look at individual study scores over time
• Look at ability and gender adjusted estimates for
  a Year
• Look at particular ability and gender adjusted
  estimates over time
• Drill down within study scores and ability and
  gender adjusted estimates

45
What to do next

• Gaining access to VCEDS reports
• See your VASS coordinator to set up a user
  account.
• Online support
• See the online Help pages and user manual
• VCAA support
• For issues related to VASS access, ring the VASS
  Helpline 1800 827 721
• For assistance with interpretation of data, email
  Glenn Rowley at
• rowley.glenn.l_at_edumail.vic.gov.au

46
Where does VCEDS fit in relation to the ACER Data
Interpretation Service?

• VCEDS is a free service to all schools. The VCAA
  has tried to anticipate your VCE data analysis
  needs, and to make the most needed analyses and
  reports available to you at the click of a mouse.
• ACER provides an analysis and interpretation
  service for data provided by your school, whether
  it be VCE data, AIM data, or other.
• The two services are complementary, and are not
  in competition with each other.

47
What next for VCEDS?

• The VCAA is presently conducting How to
  workshops in all regions. Over 1000 teachers
  have enrolled.
• During 2004, the VCAA is embarking on a major
  review of VCEDS. Feedback from users will
  determine what will be changed, and what new
  features will be added.

48
Issues to consider as you use VCEDS

• VCEDS has been designed to be as comprehensive as
  possible, without subjecting you to information
  overload.
• Is the software sufficiently user-friendly?
• Are there analyses that you would want to see
  added to the VCEDS suite?
• Is security so tight that access is difficult?
• How much support do you need? Is a program of
  annual PD sessions in February sufficient?
• Is this the way in which you would like to see
  AIM reporting move?
• Please email your ideas to Glenn Rowley at
• rowley.glenn.l_at_edumail.vic.gov.au