Retrospective and predictive evidence enhancing PISA assessments

1
Retrospective and predictive evidence enhancing
PISA assessments

• Satya Brink, Ph.D.
• Learning Policy Directorate
• Human Resources and Skills Development Canada
• Government of Canada

OECD Japan seminar Raising the Quality of
Educational Performance at School Tokyo, June
23-24, 2005
Learning Policy Directorate, HRSDC, Canada
2
Retrospective and predictive evidence enhancing
PISA assessments

• PISA assessments measure performance due to
  cumulative years of schooling
• PISA performance can also predict future
  performance because it measures competence
  rather than curriculum knowledge.

Canadian students performed well in PISA
assessments, but there were some research results
on early and later performance that were
unexpected.
Learning Policy Directorate, HRSDC, Canada
3
NLSCY - Grade 6
Inter-Provincial Differences in Mathematics Scores
NLSCY - Grade 2
NLSCY - Grade 4
NLSCY - Grade 6
TIMSS - Grade 7
TIMSS - Grade 8
SAIP93 - Age 13
SAIP97 - age 13
SAIP93 - Age 16
SAIP97 - Age 16
IALS - Youth Aged
16 to 25
-1.5
-1.0
-0.5
0.0
0.5
1.0
1.5
2.0
Years of Schooling
Learning Policy Directorate, HRSDC, Canada
Willms, Vulnerable Children, 2002
4
Early differences in reading outcomes persist
through schooling
Mean Reading Score by Readiness Group (Assessed
in Kindergarten)
Canadian Test of Basic Skills
80
70
60
50
40
Low reading
Average reading
Variable reading
30
High reading
20
10
0
Grade 10--Reading Comprehension
Grade 6--Reading
Grade 8--Reading
Learning Policy Directorate, HRSDC, Canada
Elkin, Class of 79, 2002
5
Policy questions

• What problems affect learning outcomes?
• Universal or targeted? Mix?
• How do we target?
• How durable are the results?

Investment in a national system of surveys with
data on learning

• Measured outcomes within surveys
• Longitudinal surveys
• Life course coverage
• Trace trajectories (outcomes) and chains of risk
  (negative determinants) and protective factors
  (positive determinants

Learning Policy Directorate, HRSDC, Canada
6
Vulnerability was defined as the occurrence of
low measured current outcomes that indicate a
higher chance of negative outcomes later in life.
Children were vulnerable if they had at least
one learning or behaviour problem that they could
not overcome without an intervention.

• In 1998-1999, 24.4 of children had at least one
  learning or behaviour problem.
• There were approximately 1.1 million vulnerable
  children between the ages 0 to 11 nationwide in
  1998-1999.
• In every cycle, close to 75 of Canadian
  children aged 0-11 had no identifiable behaviour
  or learning problems.

Source NLSCY
Learning Policy Directorate, HRSDC, Canada
7
Behavioural problems were more prevalent than
learning problems

• 18 of Canadian children have a behavioural
  problem
• 11.8 of Canadian children have a learning
  problem
• Only 2.9 of children have both a learning and a
  behavioural problem.
• Because of the higher prevalence of behavioural
  problems among boys, 30.8 were vulnerable
  compared to 24.4 of girls.

Percentage of Children in Canada
Cycle 2 NLSCY
Learning Policy Directorate, HRSDC, Canada
8
Vocabulary performance of 4 to 5 year olds
Source National Longitudinal Survey of Children
and Youth
Learning Policy Directorate, HRSDC, Canada
9
Why some policies have not worked. Targeting by
income can result in mis-targeting
Children with high outcomes who dont need help
but benefit when policies target by low income
Children with low outcomes who need help but do
not benefit when policies target by low income
Adapted. Willms, vulnerable children, 2002
Learning Policy Directorate, HRSDC, Canada
10
Outcomes of younger children were impacted more
by home language than older children
Source National Longitudinal Survey of Children
and Youth
Learning Policy Directorate, HRSDC, Canada
11
Differences in math scores of children with
disabilities
Math scores of children by disability or chronic
illness
Source NLSCY 1998-99
Learning Policy Directorate, HRSDC, Canada
12
Grade repetition among aboriginal and
non-aboriginal students during school years
Learning Policy Directorate, HRSDC, Canada
13
Vulnerability is not a permanent state for most
children
1994
1996
56.2
14.9
71.1
71.9
15.7
Vulnerable
28.1
28.9
13.2
Resilient
Positive Development
Long term Vulnerable
Newly Vulnerable
Source NLSCY
Learning Policy Directorate, HRSDC, Canada
14
Trajectories suggest Universal preventive
approaches and early intervention
1994-95
1996-97
1998-99
13,5
10,6
14,2
(0-2 years old)
(3-4 years old)
(5-6 years old)
26
103
Low Score
49
50
1.7
Low Score
Not Low Score
51
53
1.8
14
400
Low Score
15
44
1.5
Low Score
Not Low Score
Not Low Score
85
8.6
253
74
297
8
86
2557
209
Low Score
40
83
2.8
Not Low Score
Low Score
Not Low Score
60
126
4.3
8.2
243
Low Score
10
Percent ()
Not Low Score
Number of obs.
Not Low Score
90
2105
71.2
92
2348
Total Percent ()
Source NLSCY, all cycles, MSD and PPVT Scores
Learning Policy Directorate, HRSDC, Canada
15
Learning and behaviour problems of graduates and
dropouts
Graduates Graduates Graduates Dropouts Dropouts Dropouts
Learning Grade average final year Total Men Women Total Men Women
Learning 80-100 - A 42.3 35.5 48.5 12.9 10.7 16.4
Learning 70-79 - B 43.2 45.8 40.8 34.5 31.3 39.6
Learning 60-69 - C 13.6 17.4 10.1 34.8 38.5 29.0
Learning 50-59 - D 0.9 1.2 0.6 14.2 15.6 12.0
Learning Under 50 - F 0 0 0 3.6 4.0 3.0
Behaviour skipped class once a week or more 21.2 25.9 16.9 57.6 60.3 53.4
Behaviour drank alcohol once a week or more last year of high school 29.3 37.6 21.7 39.3 43.0 30.8
Behaviour used marijuana once a week or more last year of high school 9.1 12.1 6.1 28.1 34.4 17.9
Learning Policy Directorate, HRSDC, Canada
16
Start of the pathways of 18-20-year-olds in
December 1999
Learning Policy Directorate, HRSDC, Canada
17
Pathways of youth December 1999 and December
2001
Age 20 December 1999
Age 22 December 2001
Source Education and labour market pathways of
young Canadians between age 20 and 22 An
overview, Statistics Canada and HRSDC, 2004
Learning Policy Directorate, HRSDC, Canada
18
There is an inverse relationship between age and
literacy performance, even after controlling for
educational attainment
Relationship between age and literacy scores on
the document literacy scale, with adjustment for
level education, 2003
Norway
Canada
Source ALL, 2003
Learning Policy Directorate, HRSDC, Canada
19
Some policy directions based on longitudinal
evidence

• Importance of early childhood learning
• Mix of preventive, universal investments and
  targeted interventions
• Behaviour problems may require interventions
  outside school
• Quality of initial education ensures that full
  benefit is drawn from further learning
• Policies for second chance and life long learning

Learning Policy Directorate, HRSDC, Canada