Assessment of Teacher Preparation: Issues and Illustrations 20052008 Louisiana Value Added Assessmen

1
Assessment of Teacher Preparation Issues and
Illustrations 2005-2008 Louisiana Value Added
Assessment

• George H. Noell, Ph.D.

2
Overview of todays talk

• Early work that lead to VAM initiative
• How connections opportunities can emerge
• Evolution of the early work
• Data, funding, flaws, and relationships
• Ramping up to a sustained effort
• Coordination, funding, flaws, and relationships
• Some interesting findings

3
Some consideration of the complexity of the
journey

• The scope of the effort in Louisiana
• The complexity of the work
• IT, models, IRB, software
• Ramping up to a sustained effort
• Coordination, funding, flaws, and relationships

4
Acknowledgements

• Commissioner of Higher Education E. Joseph
  Savoie, Past State Superintendent Cecil Picard,
  Current State Superintendent Paul Pastorek, and
  Assoc. Commissioner Jeanne Burns.
• Louisiana Department of Education Division of
  Planning, Analysis, and Information Resources
  (David Elder Division Director, Allen
  Schulenberg, Robert Kaufman, Kelvin LaCroix,
  Steve Gunning, Sam Pernici, Bobby Franklin, and
  Roth Aymond)
• LSU Research Team (Veronica Gulley, Bethany
  Porter, Anna Beth Ball, Maria Patt, Amanda Dahir)

5
Redesign in Louisiana the Blue Ribbon Commission
6
How redesign lead to Louisianas Assessment

• The principle of meaningful assessment
• Putting all of the key players in the same room
• Creating a climate of striving and mission
• Creativity and shared resources ()

7
Challenges in Assessing Teacher Preparation

• The challenge of measures
• Achievement versus opinions
• Geography
• Heterogeneous schools classes
• Data management
• Technical issues
• The plausible counter factual (Rubin)

8
General Assessment Approach

• Establish empirical expectations
• Measure Performance
• Compare Expectations Performance
• Act on the Results
• Detailed follow-up policy issues

9
Moving to sustained work

• Funding then and now
• Establishing a small work team
• Establishing the big statewide team as well
• Working in public view and listening

10
Selected methodological issues with VAM in Ed.

• Meaning of the data you have
• Teacher experience variable
• Scaling of your tests
• Who taught the student versus tested
• Reliability of identifiers students vs.
  teachers
• Modeling issues
• Just fit fixed models because that is the right
  thing to do
• Just drop the Special Ed. kids, what a mess
• Of course you need cross year school fixed
  effects

11
Selected issues 2

• How big is a BIG effect?
• How important is 0.1 SD or 5 points
• Poverty SLD and 3 to 12 points
• What is a reasonable apportioning of the variance
• What would that mean
• What do demographic effects mean is a massively
  specified VAM?
• Learning disability and free lunch effects
  change in time

12
Selected Issues 3

• Value added models, growth models, and their
  union and disunion
• When effects dont make sense
• Is it too complicated yet or already too
  complicated

13
2004-2008 Study Breakdown of Data

• Years 2004-2005, 2005-2006, and 2006-07
• Student Grade Levels Grades 4-9
• Content Areas Mathematics Science Social
  Studies Language Arts and Reading
• Tests Iowa Test of Basic Skills, i-LEAP, and
  LEAP-21
• Pathways for New Teachers Undergraduate and
  Alternate Certification Programs (Master of Arts
  in Teaching, Practitioner Teacher Program, and
  Non-Masters/Certification-Only)

14
New Experienced Teachers

• New Teachers
• 1st and 2nd year teachers
• Regular certificates
• Completed TPP within 5 years
• Experienced Teachers
• 3rd or subsequent year teacher with a regular
  certificate
• Teaching within area of certification.

15
Important Decisions Defining New Teachers (2007
study data)
16
Criteria for Inclusion of New Teachers in Study

• Inclusion for each content area
• 25 or more new teachers in grades 4-9
• At least 10 new teachers per year
• Teaching within certification
• Remained with student full academic year
• Not all universities had sufficient new teachers
  to be included in the analysis. Another year of
  data will be added and will increase the number
  that exceed 25 graduates.

17
Size of the Data Basefor the Study Per Year

• Districts All school districts in Louisiana
• Students approximately 250,000
• Teachers 7,000
• Schools 1,300

18
Hierarchical Linear ModelsNesting and
Interacting by Year
19
Teacher Preparation Effect Estimates
355
Prior (2005) Achievement
Current (2006) Achievement 350
Gifted
African American
Free Lunch
Male
10 days absent
20
Post-Redesign Programs Pre-Redesign Programs

• Post-Redesign Programs Programs that were
  redesigned for grades PK-3, 1-5, 4-8, and 6-12
  and began admitting pre-service teachers on July
  1, 2003.
• Pre-Redesign Programs Universities stopped
  admitting candidates to the programs on July 1,
  2003. A phase out period is occurring for
  pre-redesign programs.

21
Effect Estimates for Post-RedesignAlternate
Certification ProgramMathematics
22
Effect Estimates for Post-RedesignAlternate
Certification ProgramLanguage Arts
23
Effect Estimates for Post-RedesignAlternate
Certification ProgramReading
24
ACT Scores of New Teachers

• ACT scores of new teachers within programs did
  not account for variance in teacher preparation
  program effectiveness.
• ACT mathematical scores of individual new
  teachers across programs was a modest predictor
  of teacher effectiveness in mathematics.

25
Impact of Teachers Who Are Not Content Certified
Teachers who are certified in the content area
they are teaching are more effective than those
who are not certified to teach that content.
26
Placing Results in Context Mathematics from
2007 study
27
Summary Observations from the Data

• Some TPP prepare new teachers contribution to
  achievement is similar to experienced teachers.
• Effectiveness varies across and within TPP.
• ACT scores of new Louisiana teachers do not
  account for variance in teacher preparation
  program effectiveness.
• ACT mathematics scores of individual teachers is
  a modest predictor of teacher effectiveness in
  mathematics.
• Certified teachers are more effective than
  teachers who are not certified to teach the
  content.

28
Summary Observations from the Process

• Sustaining system change and/or the long term
  research agenda
• Data Policy
• Relationships funding
• Working in the public view
• Visibly responding to criticism
• Sustaining institutional support
• The many agendas in play
• Louisianas statewide research team