What is our evidence from research?

1
What is our evidence from research?

• Teacher Leader Institute
• San Antonio, June 5 2003
• Francisco X. Gómez B.
• Director, NDEC

2
Volunteers Needed

• Scientific Study randomized trial
• You will be assigned randomly to one of four
  groups
• You dont know what group you will be assigned to
  before you volunteer
• You will have to consume (or not consume) daily
  for 30 years as follows.

3
(No Transcript)
4
Randomized Trials

• Randomized trials are not the only legitimate
  form of scientific inquiry
• They are not necessarily the best or highest
  quality type of research
• Even in medicine, epidemiological,
  quasi-experimental, clinical other methods are
  more common advance knowledge
• The medical model is not always relevant

5
What is science? Says who?

• Who defines or should define what science is?
• What criteria are to be used?
• Traditionally, scholars from a given field define
  criteria of what constitutes good science good
  research
• These standards are not rigid methods need to
  adapt to different situations opportunities for
  learning

6
Guidance to Reading First

• Each State educational agency that receives
  Reading First funds must assess and evaluate, on
  a regular basis, the progress of local
  educational agencies that receive subgrants in
  meeting the goals of the Reading First program.
• U.S. Department of Education (2002).

7
Reporting Requirements for SEAs/1

• Implementation Evidence The SEA must
  demonstrate that it has met all program
  requirements and obligations related to the
  implementation and administration of the Reading
  First program.
• Achievement Gains The SEA must specifically
  identify the schools and local educational
  agencies within the State that report the largest
  gains in reading achievement.

8
Reporting Requirements for SEAs/2

• Program Effectiveness The SEA must report on
  the progress the SEA and local educational
  agencies are making in reducing the number of
  students in grades 1 through 3 served by Reading
  First who are reading below grade level. SEAs
  should select methods of collecting and reporting
  this information that will result in the
  submission of data that are valid and reliable.

9
Reporting Requirements for SEAs/3

• Reducing Students Reading Below Grade Level
  Statewide The SEA must report on whether it and
  local educational agencies have significantly
  increased the number of students reading at grade
  level or above, including whether the percentages
  of students in certain categories reading at
  grade level or above have increased.

10
Disaggregated Data Tables

• Disaggregated data tables help answer questions
  on achievement gains for different groups
• Discontinued students from various groups tend to
  exhibit fairly uniform outcomes
• School districts can request District Reports
  with disaggregated data for 75
• School Data Summaries contain disaggregated data
  - free

11
Scientifically Based Research

• The U.S. Department of Education has outlined
  four criteria for local and state educational
  agencies to evaluate research findings.
• Agencies that meet these criteria are considered
  to be using scientifically based research
  principles for evaluation.

12
(No Transcript)
13
Criterion 1/a

• Use of rigorous, systematic and empirical methods
• This is the definition of a scientific research
  methodology. The evaluation methodology for
  Reading Recovery evaluation can be found in the
  Methodology section of the National Report. Site,
  District and School Reports contain abbreviated
  versions.

14
Criterion 1/b

• The Data Collection Procedures Manual, which must
  be followed equally by all participants in
  Reading Recovery, is an example of a rigorous,
  systematic and empirical methodology.

15
Criterion 2/a

• Adequacy of the data analyses to test the stated
  hypotheses and justify the general conclusions
  drawn
• The RR methodology has 8 research questions
• The data in the reports provide the answers to
  these questions

16
Criterion 2/b

• RR evaluations are usually descriptive because
  they report on the implementation of an
  intervention in a population.
• The same basic design can be used to test
  hypotheses.
• Conclusions in national, state other reports
  are based on data

17
Criterion 3/a

• Reliance on measurements or observational methods
  that provided valid data across evaluators and
  observers and across multiple measurements and
  observations

18
Criterion 3/b

• The six tasks of the Observation Survey have been
  normed and shown to be reliable.
• See Clay, M. M. (1993a/2002). An observation
  survey of early literacy achievement. Portsmouth,
  NH Heinemann.
• 2002-2003 random sample data will allow for a
  re-norming of the OS.

19
Criterion 4/a

• Acceptance by a peer-reviewed journal or approved
  by a panel of independent experts through a
  comparably rigorous, objective and scientific
  review (U.S. Department of Education, 2002).

20
Criterion 4/b

• A study of the effectiveness of the Reading
  Recovery intervention was both reviewed by a
  panel of independent experts and accepted for
  publication in a peer-reviewed journal.
• See Pinnell, G. S., Lyons C. A., Deford, D. E.,
  Bryk, A., Seltzer, M. (1994). Comparing
  instructional models for the literacy education
  of high-risk first graders. Reading Research
  Quarterly, 29, 839.
• Other peer-reviewed research articles can be
  found in the What Evidence Says document
  published by RRCNA.

21
Quality Levels of Evidence

• Adapted from Reyna, 2002
• Official U.S. position on quality of research
• Case Studies most subjective least convincing
  level of evidence testimonials. A very common
  program evaluation methodology in public
  education today. For theory building.

22
Quality Levels of Evidence/2

• Correlational Study without statistical controls-
  
• For example, X number of students scored Y on
  Test A, therefore a good instructional method was
  used no basis for comparison no measure of
  statistical significance

23
Quality Levels of Evidence/3

• Correlational Study with statistical controls
• This would be a study that compares two groups,
  without any before-after data without
  controlling who is in those groups. Allows
  statistical testing but inferences that can be
  drawn are limited.

24
Quality Levels of Evidence/4

• Quasi-experiment, including before and after
• Two groups are selected according to pre-defined
  criteria (e.g., RR RS).
• The groups are measured before after treatment
  (e.g., Fall, Mid-Year Spring testing of RR
  RS)
• As close as you can come to a lab experiment

25
Reading Recovery Evaluation

• Uses a quasi-experimental study design
• Study groups are Reading Recovery Random Sample
  (or DLL RS-DLL)
• All groups measured before after
• Rigorous, uniform criteria for belonging to a
  group
• Uniform measurement (Observation Survey)
• Uniform implementation (training model ensures
  fidelity)

26
Quality Levels of Evidence/5

• Randomized trial-
• Study groups assigned randomly
• Pre Post testing
• Statistical significance
• Extraneous factors eliminated as much as possible
• Very difficult costly

27
Randomized Trials in Education

• Held up as ideal by U.S. Department of Education
  but very difficult in practice
• Ethical issues which study group will your child
  or school be part of? Any volunteers?
• Methodology issues schools are not labs
• Raudenbush cites only 2 examples
• Thomas Cooks studies of Comer schools
• The Pinnell et al. Reading Recovery study
• (Raudenbush, 2002)

28
Treatment Effect Studies

• Once a treatment effect has been established in a
  randomized trial, it does not need to be
  re-established in further studies if the
  treatment is unchanged.
• Example, no studies within last few decades of
  effectiveness of aspirin the effectiveness of
  the treatment established long ago
• Pinnell et al. study results still stand

29
RR Scientific Research

• RR meets the toughest requirements set by U.S.
  Department of Education we have a randomized
  trial on record
• Routine monitoring research uses second-highest
  quality level, the Quasi-experimental approach
• Randomized trials are not the end-all, be-all of
  scientific inquiry
• Monitoring scientific research are two
  different things
• Monitoring research provides evidence of
  effectiveness in scaling up

30
References 1/2

• Clay, M. M. (1993a/2002). An observation survey
  of early literacy achievement. Portsmouth, NH
  Heinemann.
• Pinnell, G. S., Lyons C. A., Deford, D. E., Bryk,
  A., Seltzer, M. (1994). Comparing instructional
  models for the literacy education of high-risk
  first graders. Reading Research Quarterly, 29,
  839
• Raudenbush, S. (2002). Scientifically based
  research. Paper submitted at February 6, 2002
  U.S. Department of Education seminar. Found on
  5/29/03 at http//www.ed.gov/offices/OESE/esea/res
  earch/raudenbush-paper.html

31
References 2/2

• Reyna, V.F. (2002). The logic of scientific
  research. Paper submitted at February 6, 2002
  U.S. Department of Education seminar. Found on
  5/29/03 at http//www.ed.gov/offices/OESE/esea/res
  earch/reyna-paper.html
• U.S. Department of Education (2002). Guidance for
  the Reading First Program. Washington, D.C.
  Author
• http//www.ed.gov/offices/OESE/readingfirst/Readin
  gFirstGuidanceFINAL.pdf

32
Download presentation

• Log in to NDEC
• Click on Download Reports and Data Files
• This other relevant documents will be listed
  under Related Documents