Thoughts on Relevance

1
Thoughts on Relevance Robert C. Granger,
Ed.D. Remarks prepared for Society for Research
on Educational Effectiveness Alexandria, VA /
March 1-3, 2009
2

• Why relevance?
• Relevant for what?
• Relevant for whom?

2
3
Main Messages

• The RD cycle of developing, testing, or
  identifying innovations at a small scale and then
  scaling up these pockets of excellence does not
  work.
• The role of research is to better sift through
  practices implemented at scale, not to discover
  innovations.
• Instead of focusing on individual student or
  teacher performance, focus on classrooms.

3
4
There are many small pockets of excellence in
the education market. But because of its
fragmentation, few of these have been able to
scale. With the proper incentives and
infrastructure, many innovations could be
surfaced or created and more importantly, adopted
broadly, resulting in dramatic increases in
education production getting greater student
achievement gains per dollar invested. Bill
and Melinda Gates Roll Call Op Ed February 10,
2009
4
5
A robust new federal Office of Educational
Entrepreneurship and Innovation within the
Department of Education would expand the
boundaries of public education by scaling up
successful entrepreneurs, seeding
transformational educational innovations, and
building a stronger culture to support these
activities throughout the public sector. Sara
Mead and Andrew Rotherham Paper for
Brookings Metropolitan Policy Program October,
2008
5
6
Comparison of Clinical Trial Phases
Phase I Objective Determine the metabolic and
pharmacological actions and the maximally
tolerated dose. Phase II Objective Evaluate
effectiveness, determine the short-term side
effects, and identify common risks for a specific
population and disease. Phase III Objective
Obtain additional information about the
effectiveness on clinical outcomes and evaluate
the overall risk-benefit ratio in a
demographically diverse sample. Phase IV
Objective Monitor ongoing safety in large
populations and identify additional uses of the
agent that might be approved by the FDA.
6
7
Stages of Research in Research Prevention Cycle
7
8
IES Goal Structure for Grantmaking

• Goal Description
• Identification Projects
• Development Projects
• Efficacy and Replication Trials
• Scale-Up Evaluations
• 5. Measurement Projects

8
9
9
10
A Snapshot Intervention Trials in 2006

• National Institute of Mental Health (NIMH) 51
  trials 3 implementation/dissemination trials
• National Institute of Drug Abuse (NIDA) 79
  trials 14 implementation/dissemination
  trials
• National Institute on Alcohol Abuse and
  Alcoholism (NIAAA) 53 trials 2
  implementation/dissemination trials.
• National Academies Table F-1, page 522.

10
11
11
12
Contradiction and initially stronger effects are
not unusual in highly cited research of clinical
interventions and their outcomes. The extent to
which high citations may provoke contradictions
and vice versa needs more study. Controversies
are most common with highly cited nonrandomized
studies, but even the most highly cited
randomized trials may be challenged and refuted
over time, especially small ones. Ioannidis,
John P.A., JAMA, 2005294218.
12
13
Part II (of the report) describes the
substantial progress in prevention science since
1994. It describes numerous (quotation marks
added) efficacious or effective prevention
programsthus far, however, prevention programs
have generally not been widely implemented in
schools and communities and have done little to
reduce behavioral health problems in America.
National Academies, p. 295.
13
14
Barriers to Scaling Innovations

• Program may not fit community needs, strengths,
  or capacities.
• Real-world implementation may differ dramatically
  from the way originally tested.
• Lack of ownership of the program.
• Few evidence-based programs have the capacity to
  provide technical assistance and training.
• An evidence-based program may not target outcomes
  relevant to community.

14
15
Barriers to Scaling Innovations (continued)

• Key program components may be modified, thereby
  reducing outcomes.
• Essential program components not always evident.
• Lengthy period to develop community awareness,
  common vision, and program.
• Potential for ineffectiveness or iatrogenic
  effects.
• Challenges in obtaining funding for sustaining a
  unique program.

15
16
16
17
The Decade of Experiment (1951-1961)

• Recruitment and training programs for liberal
  arts undergraduates
• Differentiated staffing with classroom aides
• Increased use of technology (TV, tape, teaching
  machines)
• Projects to introduce modern business methods
  into school management including a focus on
  teacher salaries. (50 million in 1955 393
  million in 2009).

17
18
Comprehensive School Improvement Program 12
Practices

• Team teaching
• Use of non-professional personnel in schools
• Flexible scheduling
• Variable-size pupil groups for instruction and
  new space arrangements
• Use of audio-visual resources, including
  educational television
• Programmed instruction

18
19
Comprehensive School Improvement Program 12
Practices (continued)

• Language laboratories
• Educational data processing by machine
• Independent study
• Advanced placement and early admissions
• Non-graded school programs
• School and university partnerships for
  curriculum improvement, and pre- and in-service
  preparation

19
20
Comprehensive School Improvement Program Key
Assumptions

• The purpose of a school is to promote learning.
• Learning is a continuous process and must be
  related to an individual students abilities and
  needs.
• Curriculum in all content areas should be built
  on a continuum from the beginning of the
  completion of formal education, rather than be
  frozen by grade levels or age of pupil.
• There needs to be a constant and continuous
  examination of the ways by which schools
  facilitate learning in order to take advantage of
  discoveries and developments.

20
21
The Summative Judgment on Lighthouse Projects
There was not a willingness on the part of the
projects neighbors to acknowledge its light
giving nature A Foundation Goes to School,
p.42.
21
22
22
23
A Brief Aside
We must instill in our students the expectation
of tedium and disappointment and the duty of
thorough persistence, by now well-achieved in the
biological and physical sciences. We must expand
our students vow of poverty to include not only
the willingness to accept poverty of finances,
but also poverty of experimental results.
Campbell and Stanley, p. 2 3.
23
24
Another Approach (Practice at Scale)
To avoid disillusionment several aspects may be
noted. First, the claims made (earlier) for the
rate and degree of progress which would result
from experiments were grandiosely over optimistic
and were accompanied by an unjustified
depreciation of non-experimental wisdom. The
initial advocates assumed that progress in the
technology of teaching had been slow just because
(italics in original) (the) scientific method had
not been applied they assume traditional
practice was incompetent just because it had not
been produced by experimentation.
24
25
Evolutionary Perspective on Cumulative Wisdom and
Science
..applied practice and scientific knowledge are
seen as the resultant of an accumulation of
selectively retained tentatives remaining from
the hosts that have been weeded out by
experience. ..but the selective, cutting-edge
of this process of evolution is very imprecise in
the natural setting. Experimentation enters at
this point as the means of sharpening the
relevance of the testing, probing, selection
process. Experimentation thus is not in itself
viewed as a source of ideas necessarily
contradictory to traditional wisdom. It is
rather a refining process superimposed upon the
probably valuable cumulations of wise practice.
Campbell and Stanley, p. 4.
25
26
We know that about 15 of the variance in
school-induced change in achievement is between
schools and the rest is in the schools and that
at least in relative terms, teachers who add
value are widely dispersed. We also know that
teachers matter but we are not sure why or
how. In elementary school classrooms, 30 of the
time is spent on administrative matters.
26
27
Why Classrooms?

• Such settings are where development occurs.
• Attempts to change individual behavior without
  affecting its setting are rarely effective.

27
28
Conceptual Articles on Classrooms as Systems
Tseng, V., Seidman, E. (2007). A systems
framework for understanding social settings.
American Journal of Community Psychology,
39(3-4), 217-228. Cohen, D. K., Raudenbush, S.
W., Ball, D. L. (2003). Resources,
instruction, and research. Educational
Evaluation and Policy Analysis, 25, 119-142.
28
29
Classrooms as Systems
Macro Forces
Resources
Social Processes
Macro Forces
Youth Outcomes
Social Settings
Organization of Resources
29
30
Bob Pianta and colleagues at the University of
Virginia http//www.virginia.edu/vpr/CASTL
30
31
Reed Larson and colleagues at the University of
Illinois at Urbana-Champaign http//www.youthdev
.uiuc.edu/
31
32
Charles Smith and colleagues at the David P.
Weikart Center for Youth Program Quality For
staff http//www.forumforyouthinvestment.org/abou
t/staff For content http//www.highscope.org/Con
tent.asp?ContentId117
32
33
Going Forward (Including a Modest Proposal for
Secretary Duncan or Some Entrepreneurial
Researchers)

• Classrooms are the unit of analysis.
• Improvements in measurement.
• Improvements in non-experimental causal analyses.
• Mixed-method work.
• An engineering approach to the improvement of
  change strategies.

33
34

• Dear Mr. Secretary
• 1 billion (100 _at_ 10 million).
• Make the welfare waiver approach work for
  education.
• Study shifts in policy but put classrooms in
  the middle.

34
35
THANKS!
35