How To Be A Savvy Consumer Of Education Research

1

• How To Be A Savvy Consumer Of Education Research
• An educated consumer is our best customer
• Lauren Morando Rhim, Ph.D.
• University of Maryland College Park
• Presentation at the annual meeting of the
• National Association of Charter School
  Authorizers
• Philadelphia, PA
• October 25-26, 2004

2
Realities of Education Research

• Expensive and time consuming
• Data are hard to find/obtain
• Rarely randomized
• Highly political
• Conclusions frequently deceptive and
  over-simplified

3
Strategies to Improve Education Research

• Compensate for non-randomization through other
  methods such as longitudinal and
  quasi-experimental designs
• Compensate for non-randomization by controlling
  for confounding variables to the extent possible
• Rigor, rigor, rigor

4
Key Questions to Consider When Evaluating
Education Research

• Who conducted the study?
• Who paid for the study?
• What were the research questions?
• What method did the researchers utilize to answer
  the questions?
• Did the researchers employ the methods in a
  rigorous and transparent manner?
• Do the data support the findings?

5
Red Flags

• Grand statements without citations in literature
  review
• Limited or missing methodology sections
• Vague language
• Simplistic findings for complicated concepts
• Conclusions that are not connected to data
  collected or extend beyond the data (e.g.,
  findings in a single state being generalized to
  charter schools in every state)

6

• Example 1 Survey
• In a Foundation supported survey of 35 charter
  schools, directors answered questions about their
  special education practices. The participating
  charter schools served between 25 and 2,000
  students and included both classroom-based and
  non-classroom-based charter schools. The survey
  questions were open-ended and explored the
  charter schools relationships with school
  districts and their satisfaction with their role
  in the special education system
• Although the questions were open-ended, the
  general level of satisfaction with special
  education arrangements could be determined from
  individual responses.

7

• Example 2 Survey
• The SASS was designed to support estimates at
  the national, regional, and state levels for
  public school districts, schools, principals,
  teachers, and school library media centers. The
  public school sampling frame was based on the
  1997-1998 Common Core of Data, a file of
  information collected annual by NCES from all the
  state education agencies and believed to be the
  most complete public school listing available at
  the time of sample selection. Public schools not
  in existence in the school year 1997-1998 and not
  operating as a result of a split with an existing
  school were not included. The frame contains
  regular public schools and special purpose school
  such as special education, vocational, and
  alternative schools. The frame was enhanced with
  a list of schools operated by the Department of
  Defense. After the deletion of duplicate
  schools, schools outside the United States, and
  schools that only teach pre,indergardent,
  kindergarten, or postsecondary students, 88,266
  schools remained on the public school frame
• An independent verification of charter school
  information was provided by the national Charter
  School Directory 2000, Sixth Edition (Dale 2000).
  Census personnel used this resource to verify
  the eligibility status of specific charter
  schools.

8

• Example 3 Case Study
• The descriptive focus of the study and a limit
  on the number of site visits that we were able to
  conduct required selection of a purposive sample
  of schools. We based sample selection on
  variables that define important characteristics
  of charter schools and drew the sample from the
  population of 335 charter schools that were
  surveyed in 1997 by the National Study of Charter
  schools. The National Study data set was the most
  comprehensive source of information about charter
  schools. We chose five selection variables that
  our review of previous research suggested
  influence how charter schools are able to serve
  students with disabilities. For each variable, we
  established categories with ranges of expression
  of the variable, based on data available from the
  National Study. The variables were, in order of
  priority, (1) proportion of students with
  disabilities enrolled, (2) federal public charter
  school grant recipient status, (3) level of
  operational autonomy based on the extent of the
  schools control over admissions and budgets, (4)
  grade levels served, and (5) geographic region.

9

• Example 4 Case Study
• At most schools, a two-person team spent two
  days on-site, conducting observations, reviewing
  student and school recrods, and interviewing
  staff, parents, and students. A set of five
  protocols guided data collection, although all
  interviews were open-ended. Site visitors
  prepared for visits by reading charters and other
  reports send by school operators, modifying the
  protocols to reflect extant data and the expected
  on-site respondents, and reviewing information on
  the host states charter school law. Interviews
  with the school spokesperson (usually the
  principal or director) took approximately 11/2
  hours. We conducted group interviews (focus
  groups) with teachers, parents of students with
  disabilities, and students with disabilities or
  other at-risk learners. The teacher and parent
  interviews last approximately two hours.

10
Example 5 Newspaper ArticleCharters Score Below
Public Schools Exclusive At 235 Texas campuses,
passing rate was 42 percent1016 PM CDT on
Wednesday, October 20, 2004By TERRENCE STUTZ /
The Dallas Morning News

• AUSTIN More than eight years after they were
  launched as a bold experiment in education,
  Texas' charter schools as a whole are performing
  well below other public schools on state tests,
  according to a Dallas Morning News review of
  data.
• At the state's 235 independent charter campuses
  that administer the Texas Assessment of Knowledge
  and Skills, 42 percent of the students passed,
  according to figures compiled for The News by the
  Texas Education Agency. Statewide, the overall
  passing rate for all public school students was
  nearly 67 percent. The state passing rate for
  students classified as low-income was 56 percent
  charter school proponents often point out that
  charter students are more likely to come from
  poor families.
• It's pretty clear the overall record of charter
  schools is not as good as regular public schools
  with comparable student populations," said Ted
  Melina Raab of the Texas Federation of Teachers.
• Mr. Raab's group supported charter schools when
  they were approved by the Legislature and signed
  into law by Gov. George W. Bush in 1995.
• A few have done well, but many more have done
  poorly," he said, referring to TAKS results
  showing at least three out of four pupils failed
  at nearly 30 percent of the charter schools in
  Texas earlier this year

11
Example 6 Newspaper ArticleProgress Outpaces
Public High SchoolsCharters gain in MEAP
scoresBy Brad Heath / The Detroit News

• Michigans charter high schools are making
  faster progress toward meeting state standards
  than other public high schools, though they still
  trail far behind.
• Charters performance in reading, writing, math
  and science increased faster than the state
  average. Their scores on Michigan Educational
  Assessment Program tests already are higher than
  in the states inner-city schools.
• Proponents say the scores, released last week,
  are fresh proof that the schools are living up to
  their promise of improving education for kids
  struggling in traditional schools. But they
  acknowledge that like most schools theyre a
  long way from meeting a federal requirement that
  every student master basic skills. And critics
  say their gains are mostly a sign of how far
  charters have left to go.

12
Closing Thoughts

• Charter schools and issues related to school
  choice are highly political and therefore subject
  to less than objective reporting
• Advocates and opponents are passionately
  committed to promoting their positionssometimes
  in spite of data
• Information is ubiquitous and many outlets are
  not edited for quality or rigor
• Authorizers must be educated in order to be savvy
  consumers of the volumes of research related to
  charter schools.