AUTHENTIC%20ASSESSMENT%20AND%20PROGRAM%20EVALUATION%20IN%20EARLY%20CHILDHOOD

1
(No Transcript)
2
(No Transcript)
3
(No Transcript)
4
Scaling Progress in Early Childhood Settings
(SPECS)STEPHEN J. BAGNATO, Ed.D.,
NCSPProfessor of Pediatrics PsychologyDirector
, Early Childhood PartnershipsDirector, SPECS
(PAPREKA/PEIOS)Childrens Hospital of
Pittsburgh/UCLID CenterUniversity of Pittsburgh
School of Medicinesteve.bagnato_at_chp.eduwww.uclid
.org
5
SPECS Program Evaluation Research Team Leaders
PAPREKA and PEIOSStephen J. Bagnato, Ed.D.,
NCSP, Director Candace Hawthorne, Ph.D., OTR/L,
CoordinatorIlene Greenstone, MA, CoordinatorPip
Campbell, Ph.D., OTR/L, CoordinatorAssisted by
Western and Eastern PA Research Teams
6
What is the Authentic Assessment Alternative to
Conventional Testing in Early Childhood?
7
(No Transcript)
8
Inauthentic Measurement in Early Childhood

• Much of developmental psychology early
  childhood assessment as it now exists is the
  science of the strange behavior of children with
  strange adults in strange settings for the
  briefest possible periods of time.
  (Bronfenbrenner, 1979, p. 19)

9
(No Transcript)
10
(No Transcript)
11
Authentic Assessment in Early Childhood

• Natural observations of ongoing child behavior in
  everyday settings and routines vs. contrived
  arrangements
• Reliance on informed caregivers (teachers,
  parents, team) to collect convergent,
  multi-source data across settings
• Curriculum-based measures linked to program
  goals, content, standards, expected outcomes
• Universal design equitable assessment content
  and methods
• Intra-individual child progress supplemented by
  inter-individual normative comparisons
• NAEYC/DEC/HS PA DAP Assessment Standards
  Practices

12
(No Transcript)
13
(No Transcript)
14
(No Transcript)
15
Are There Professional and Pennsylvania Standards
for Authentic Assessment in Early Childhood/Early
Intervention?
16
(No Transcript)
17
(No Transcript)
18
Selected Professional Standards for Early
Childhood Assessment (DEC, 2004 NAEYC, 1997,
HS, 2000)

• Reliance on developmental observations-ongoing
  observational assessments overtime
• Performance on authentic, not contrived,
  activities
• Integration of assessment and curriculum
• Child progress on past performances as the
  reference, not group norms
• Choose materials that accommodate the childs
  special functional needs
• Use only measures that have high treatment
  validity
• Rely on curriculum-based measures as the
  foundation or mutual language for team
  assessments
• Defer a diagnosis until evaluation of a childs
  response to a tailored set of interventions
• Use scales with sufficient item density to detect
  even small increments of progress

19
6 Best Practice Criteria for Authentic
Assessment in Pennsylvanias ECE Programs

• Purpose Assess for program planning not
  diagnosis or exclusion eliminate readiness
  testing practices
• Method No tabletop testing Deemphasize
  scores observe and assess functional skills that
  link to the curriculum standards
• Context Observe and record evidence of natural,
  ongoing child development and behavior in
  typical, everyday routines not contrived settings
• Process Rely on teachers/caregivers to observe
  and record child progress 2-3 times each year
• Standards-based Align all assessments and their
  item content with ELS and curricula link
  assessment with expected outcomes
• Parent Partnerships Enable parents to have
  central role in providing observational data on
  progress

20
Measures for PEIOS/PAPREKA Research Balance of
Attributes

• Tension to balance research rigor with utility
  Choice of measures based on following elements
• Simplicity
• Authenticity
• Utility
• Evidence-base
• Standards-referenced
• Functional content
• Sensitivity

21
(No Transcript)
22
The Pennsylvania Early Intervention Outcomes
Study(PEIOS)Documenting the Benefits of Early
Intervention Supports in PA to Fulfill State and
Federal Mandates
23
What are the missions, research questions and
authentic measurement design for PEIOS?
24
PEIOS Fast Facts

• Aim Document early intervention outcomes for
  state federal mandates
• County agencies and MAWAs in 6 PA Regions mapped
  to PQP
• Random selection
• Collect data on entry-level functioning compared
  to performance at followup
• Use both/either an existing measure and a common
  functional measure across programs ABAS II
• File reviews to code program and service
  intensity
• Multiple research strategies to analyze
  interrelationships among program intensity and
  child/family outcomes
• Classify outcomes by OSEP/ECO categories

25
PEIOS Outcome and Research Measures

• Child Measure Caregiver
• Adaptive Behavior Assessment System II (ABAS
  Harrison Oakland, 2004)
• Child Measure Caregiver
• Program-Identified Measure (BDI DC COR)
• Program Measure PEIOS Team
• Program Specs (Bagnato, 2005)
• Program Measure PEIOS Team
• Developmental Specs (Bagnato, 2005)

26
(No Transcript)
27
Adaptive Behavior Assessment System II (ABAS,
2003)

• Multi-dimensional observational judgment-based
  rating scale (3-point) of functional competencies
• Ages 0-89years
• Early childhood forms Parent Teacher/Provider
• Nationally standardized Ages 0-52100 all forms
  and all ages5270
• Norm-referenced scores General Adaptive
  Composite (100,15) Subskills (10,3)
• Excellent technical research base disability
  studies Aligns with DSM IV AAMR categories
• Psychological Corporation

28
(No Transcript)
29
(No Transcript)
30
(No Transcript)
31
(No Transcript)
32
ABAS Rating Format

• 0..Is not able cant too young physical
  limits
• 1..Never/almost never when needed prompts
• 2..Sometimes when needed with/out help
• 3..Always/almost always when needed before
• G..Check if you guessed

33
ABAS Domains Parent Form (241) Teacher (216)

• Communication
• Community Use
• School Living/Home Living
• Functional Pre-Academics
• Health Safety
• Leisure
• Self-Care
• Self-Direction
• Social
• Motor

34
ABAS Disability Research Clinical and Matched
Control Samples

• Mental Retardation
• Developmental Delay
• Biological Risk Factors (e.g., prematurity
  drugs)
• Motor and Physical Impairments
• Language Disorders
• Autism and PDD
• Learning Disability
• ADHD
• Alzheimers Disease
• Neuropsychological Disorders
• Behavior and Emotional Disorders
• Deaf and Hard of Hearing

35
PEIOS Program Evaluation Research Measurement
Model

• January 2006 Identify and train PEIOS program
  evaluation liaisons
• January-February 2006 Random selection of
  children
• January-February 2006 Train on use of the ABAS
  II (if chosen) with liaisons, teachers, others
• January-April 2006 Collect EI-entry child data
• January-April 2006 Conduct file reviews to
  document program intensity
• May 2006 Collect child progress data using ABAS
  II or program-chosen measure (each September and
  May)

36
TIME IN INTERVENTION
CHILD FAMILY PROGRESS OUTCOMES
PROGRAM INTENSITY
37
PEIOS LONGITUDINAL REPEATED MEASURES REGRESSION
RESEARCH DESIGN AND TIMELINE
REGION 1
September
May
REGION 2
REGION 3
ABAS II Program Scale D-SPECS P-SPECS
ABAS II Program Scale D-SPECS P-SPECS
38
What are the federal OSEP/ECO outcome indicators
and reporting timelines for PEIOS?
39
OSEP/ECO Child Outcome Indicator Domains

• Positive social-emotional skills (including
  social relationships)
• Acquisition and use of knowledge and skills
  (including early language/communication early
  literacy)
• Use of appropriate behaviors to meet ones needs
• Domains, sub-domains, and item content of
  measures are mapped to these integrated
  functional areas

40
Sample LaRosa Database
41
OSEP/ECO Child Outcome Indicator Metrics

• of children who reach or maintain functioning
  level comparable to same-age peers
• of children who improve functioning toward
  same-age levels
• of children who did not improve functioning
• of children maintaining own rate and preventing
  regression
• of children showing specific curricular skill
  improvements compared to own previous skill
  levels
• of children whose developmental progress
  profiles exceed own pre-intervention
  (maturational) expectations and those of their
  local EI peer group (IEI CEI PCI, HLM-EAPS)

42
Statistical Impact of ECI on Child Progress
Exceeding Maturation after 31 Months of
Programming Pooled HR/DD groups n 104
plt.000 48thilegt68thile 95 confidence
interval
43
OSEP/ECO Family Outcome IndicatorsDraft
Reconciliation

• Based on ratings on the Family Outcomes Survey
  (in development)
• Understand their childs strengths, abilities,
  and special needs
• Know their rights and advocate effectively for
  their children
• Help their children develop and learn
• Have support systems
• Access desired services, programs, activities in
  their community

44
Timelines for State Reporting of Child Outcome
Data to OSEP(Recent Report from OSEP/ECO
National Meeting , Washington, DC, 1/12-13/06

• December 2005 SPP Measurement plan submitted
• February 2007 APR Report on EI-entry child data
  onlyno progress data wanted
• February 2008 APR 1st progress report
• February 2009 APR 2nd progress report
• February 2010 APR 3rd progress report

45
What is the collaborative model for training and
implementation in PEIOS and PAPREKA?
46
DIRECTOR
PAPREKA COORDINATORS Western
PA
Eastern PA
RESEARCH SYSTEMS DATA MANAGER
EVALUATION ASSISTANTS
STATISTICIAN PROGRAMMER
REGIONAL PA CONSULTANT
47
COUNTY MAWA SCHOOL DISTRICT PARTNERSHIP- Administr
ators
COORDINATORS
LIAISON
SPECS TEAM
REGIONAL CONSULTANT
TEACHERS CAREGIVERS
EVALUATION ASSISTANT
48
(No Transcript)
49
The Pennsylvania Pre-Kindergarten Analysis
(PAPREKA)A 4-Year Independent Program
Evaluation Research Collaborative to Document the
Impact and Outcomes of Partnership for Quality
Pre-Kindergarten (PQP)
50
SPECS Pennsylvania Early Childhood Intervention
Outcome Studieswww.uclid.org Early Childhood
Partnerships SPECS

• Heinz Pennsylvania Early Childhood Initiatives
  (ECI) 1997-present) (Bagnato, etal., 2002 2005)
• Pennsylvania Pre-Kindergarten Analysis (PAPREKA
  2005-2009 (Bagnato etal. 2005)
• Pennsylvania Early Intervention Outcomes Study
  (PEIOS 2005-2008) (Bagnato etal., 2005)
• TRACE Center of Excellence for Early Childhood
  Assessment (2002-2007 Dunst, Trivette, Bagnato)
• The Efficacy of a Direct Instruction Add-On to a
  DAP Curriculum in 4KIDS at Braddock (2005-2007)
• Pennsylvania Preschool Integration Initiative
  (PAPII 1989-1993) (Bagnato Neisworth, 1993)

51
The SPECS Team Evaluation of the
Heinz Early Childhood Initiatives
52
SPECS Evaluation of Heinz Pennsylvania ECI
Outcomes (1997-2004)

• Nearly 4000 at-risk children
• Urban and rural communities Pittsburgh, Erie,
  Central PA, York, Lancaster, Beaver
• School district collaborations
• Steady developmental gains
• Delayed group gains
• Social-behavioral gains, even for disorders
• Early school success for gt400 children K-5
• Reduced grade retention and special education
• Program quality improvements
• Improvements in parenting skills
• Evolving community networking
• Analysis of time-in-intervention effects
  (dosage)

53
What are the missions, research questions and
measurement design for PEIOS?
54
PAPREKA Missions for PQP

• Document child and program outcomes attributable
  to PQP
• Analyze the comparative impact of various school
  district/community ECE partnership models on
  child and program outcomes
• Analyze the early school success of PQP children
  during their K year
• Field-validate the PA Early Learning Standards
  and develop an ELS assessment rubric
• Set the stage for future followup of PQP children
  into early grades and linkage with PSSA results

55
PAPREKA Research Questions for PQP

• PQP children will show a pattern of actual
  progress which will outpace their maturational
  expectancies.
• Program quality and interactive teaching styles
  will predict child progress and outcomes.
• Certain types of partnership models will better
  predict child outcomes and program quality.
• For both children at-risk and with delays, those
  who participate and remain engaged in the PQP
  programs for the longest periods of time
  (dosage) will show the most significant
  progress.
• PQP children will demonstrate early school
  success in preschool and kindergarten based upon
  three criteria blind teacher assessments on the
  BSSI greater of ELS attainments and reduced
  grade retention and special education placement
  rates compared to historical district benchmarks.

56
TIME IN INTERVENTION
PARTNERSHIP MODEL
CHILDRENS EARLY SCHOOL SUCCESS OUTCOMES
PROGRAM ASPECTS
57
What is the authentic measurement approach and
timeline for PAPREKA?
58
Natural learning environment

• ...the ongoing/routine/typical circumstances and
  contexts of a childthe childs natural
  developmental ecology that forms the
  environmental basis for real-life early learning
  and adaptation (OSEP, 1998).

59
SPECS Approach for PAPREKA

• Design research model through collaboration with
  community partners (participatory action
  research)
• Train teachers for ongoing child assessments
• Conduct authentic assessments in everyday
  preschool settings using natural observations
• Provide feedback to teachers parents for
  individualized early learning plans
• Evaluate progress on Key Performance Indicators
  (KPI) child, program quality, teacher-child
  interactions, early school success outcomes
• Classify type of partnership model in each site
• Assess K achievements of children
• Research impact of PQP

60
(No Transcript)
61
(No Transcript)
62
(No Transcript)
63
(No Transcript)
64
PAPREKA Measures

• Basic School Skills Inventory 3 (BSSI-3)
  (Hammill, etal, 1998)
• Preschool and Kindergarten Behavior Scale (PKBS)
  (Merrell, 2003)
• Early Childhood Environment Rating Scale
  (ECERS-R) Screener (Cassidy etal, 2005)
• Teaching Styles Rating Scale (TSRS) (McWilliam,
  etal, 1996)
• PA Early Learning Standards (ELS) Rubric (in
  development)

65
Sample MIS Database
66
A Sample of Marys Skills 22 months old

• Some Things Im Really Good At
• put 2 words together (examples 'Want more,' 'Get
  down')
• refer to myself by name
• participate in and enjoy nursery rhymes and
  finger games like 'Humpty Dumpty' or 'Itsy Bitsy
  Spider'
• Some Things Im Working On
• copy sounds made by others
• drink from a regular cup without help
• answer simple questions
• Some Things I Want to Learn Next
• dress myself, but I may need some help or
  supervision
• share toys with other children when asked by an
  adult
• count two or three objects (saying the number as
  I touch the object)

Database Merged Document
67
Class Snapshot 2Student Progress on Selected
Assessment Items
Highlighted Bolded Item s Areas of More
Program Support
68
(No Transcript)
69
Basic School Skills Inventory (BSSI)

• Learning readiness skills for children
• Authentic teacher observational ratings
• Ages 48-107 months (Pre-3rd grade)
• 6 Domains Spoken language Reading Writing
  Math Behavior Daily living
• Standard and T-Scores (100/15 50/10)
• Functional skills/benchmarks for learning
• Graduated scoring 0, 1, 2, 3 (mastery)
• Norms 757 children 5 states
• PRO-ED

70
BSSI Subscale Samples

• Spoken Language
• Uses complete sentences when talking
• Listens to and retells a story in sequence
• Initiates and maintains conversations with others
• Reading
• Recognizes upper/lower case letters
• Names letters when sounds are spoken
• Has basic site vocabulary of 5 words

71
BSSI Subscale Samples

• Writing
• Writes from left to right
• Writes first name without a model
• Writes single letters when asked (b, h, m, t, a,
  e)
• Mathematics
• Counts objects in set of fewer than 10
• Counts aloud from 1-20
• Understands concepts of 1st, 2nd, 3rd

72
BSSI Subscale Samples

• Classroom Behavior
• Makes friends easily
• Takes turns
• Uses teacher feedback to improve learning
• Can attend to activity for 5 minutes
• Daily Living Skills
• Enters and exits school by self
• Assumes responsibility for own belongings

73
BSSI Rating Scale

• When completing the BSSI, a four-point rating
  scale is used to rate each behavior or skill
• 0 (Does not perform)
• 1 (Beginning to perform)
• 2 (Performs most of the time)
• 3 (Performance indicates mastery)

74
National Normative Comparisons for ECI Childrens
Early Learning Abilities Second Semester Grades
K-1 Basic School Skills Inventory-Revised
Mean Standard Score
Reading
Math
Daily living skills
Social skills
Academic competence
Domain
75
Acquisition of Kindergarten Precursor Skills
Matching PA Early Learning StandardsAchievement
in Final 6 Months of ECI
Learning Skill s Achieved
76
Statistical Impact of ECI on Child Progress
Exceeding Maturation after 31 Months of
Programming Pooled HR/DD groups n 104
plt.000 48thilegt68thile 95 confidence
interval
77
(No Transcript)
78
Preschool and Kindergarten Behavior Scales (2003)

• 36-72 months
• Dual focus on social skills and problem behaviors
• Competencies/concerns with treatment validity
• Cooperation, interaction, independence, and
  behavior domains (attention, withdrawal)
• Parent and professional ratings
• Norms 2,855
• Graduated scoring and SS
• PRO-ED

79
Preschool and Kindergarten Behavior Scales (2003)

• SOCIAL SKILLS Never Rarely
  Sometimes Often
• Works or plays independently 0
  1 2 3
• Follows instructions from
• adults
  0 1 2 3
• Shows self-control 0
  1 2
  3
• Participates in family or
• classroom discussions 0
  1 2 3
• Follows rules
  0 1 2
  3
• Takes turns with toys and
  
• other objects
  0 1 2
  3

80
Preschool and Kindergarten Behavior Scales (2003)

• PROBLEM BEHAVIOR Never Rarely
  Sometimes Often
• Acts impulsively without
• thinking
  0 1 2
  3
• Must have his/her own way 0
  1 2 3
• Is restless and fidgety 0
  1 2 3
• Withdraws from the company
• of others
  0 1 2
  3
• Is overly sensitive to criticism
• or scolding
  0 1 2
  3
• Disrupts ongoing activities 0
  1 2 3

81
Social-Behavioral Progress Pattern for ECI
Children Behavior Disorder Group (18) T1-T5
Mean Standard Score
Domain
82
Roles for Authentic Assessment in PAPREKA

• Teachers
• September and May
• (Note January for 2006)
• BSSI-3
• PKBS

• PAPREKA Team
• September and May
• (Note January for 2006)
• ECERS-R Screen
• TSRS

83
Linking Assessment to State Early Childhood
OutcomesPA Early Learning Standards (ELS)

• Approaches to Learning
• Creative Arts
• Language Literacy
• Logical-Mathematical
• Personal-Social
• Program Partnerships
• Science
• Social Studies

• Demonstrate initiative curiosity
• Expresses self through movement music
• Understands language sounds
• Develops space-shape concepts
• Develops self-control skills
• Form school partnerships with ECE
• Acquire knowledge about matter and living things
• Understand role of self in community

84
Linking BSSI Items to the PA Early Learning
Standards
BSSI Area or Domain BSSI Item PA EL Standard (for 4- 5-year-olds)
Spoken Language Answers questions correctly after listening to a story SL 11 Develop and expand listening and understanding skills RL 1
Reading Follows printed instructions (reads and follows directions) R 20
Classroom Behavior Demonstrates a readiness to respond when his or her turn comes during group activities CB 11
Daily Living Skills Follows directions relating to paper-and-pencil tasks DL 10
85
Linking BSSI Items to the PA Early Learning
Standards
BSSI Area or Domain BSSI Item PA EL Standard (for 4- 5-year-olds)
Mathematics Assigns the correct numeral to a set of objects M 7 Learn about numbers, numerical representation, and simple numerical operations LM 1
Understands the concepts of first, second, and third M8
Daily Living Skills Tells time within 5 minutes from analog watch or clock face DL 19

86
Overview of Programmatic Measures ECERS TSRS

• ECERS-R Screener
• 16 core items
• Two factors
• Activities-Materials (9)
• Language-Interaction (7)

• TSRS
• 20 items
• Focus on teacher-child interaction during
  instruction
• 8 teaching strategy variables Redirects,
  Introduces, Elaborates, Follows, Informs,
  Acknowledges, Praises, Affect

87
PAPREKA LONGITUDINAL REPEATED MEASURES REGRESSION
RESEARCH DESIGN AND TIMELINE
PARTNERSHIP MODEL 1
September
May
PARTNERSHIP MODEL 2
PARTNERSHIP MODEL 3
BSSI-3 PKBS ECERS-R TSRS
BSSI-3 PKBS ECERS-R TSRS
88
Roles of Implementation Sites in the Partnership
Implement Assessment Gain Incentive Funds

• Designate PQP Evaluation Liaison
• Train with PAPREKA team to learn assessments
• Train teachers in assessments
• Gain signed parent consent
• Ensure completion of assessments by teachers
  2x/year
• Collect completed assessment forms and transfer
  hard copies to PAPREKA Coordinator
• Show teachers how to use feedback letters

89
Roles of PAPREKA Team for Implementation Sites

• Meet with partnership to form trusting
  collaboration
• Maintain confidentiality
• Train liaison and teachers in authentic
  assessment best practices
• Complete programmatic observations 2x/year
• Collaborate to classify type of partnership in
  each program
• Produce child feedback letters
• Issue 1 year-end report card
• Analyze aggregate data on child outcomes

90
PAPREKA Team Roles with The PQP Planning Sites

• Schedule collaborative meetings to develop our
  partnership early as a basis for future work
• Offer early training sessions on assessment
  purposes and eventual evaluation model and
  process
• Demonstrate computer child feedback options for
  programs to guide teaching and quality
  improvements
• Consult with partnerships to facilitate their
  eventual implementation proposal
• Collect extant information from sites on various
  programmatic features
• Begin process of classifying sites by type of
  partnership model

91
Not everything that can be measured counts, and
not everything that counts can be measured
(Einstein, 1951)
92
The MisMeasure of Man (Stephen J. Gould, 1981)

• We pass through this world but once. Few
  tragedies can be more extensive than the stunting
  of life, few injustices deeper than the denial of
  an opportunity to strive or ever hope, by a limit
  imposed from without, but falsely identified as
  lying within (p.28).