Results Demonstrated: Federal Evaluating State Educational Technology Programs ESETP Grants

1
Results Demonstrated Federal Evaluating State
Educational Technology Programs (ESETP) Grants

• Melinda George
• SETDA

2
2006 National Trends Report

• Promising Interim Results States have been
  targeting NCLB II D funds on the three program
  goals increasing student achievement, closing
  the digital divide, and integrating
  research-based technology practices into
  learning.
• Highly Qualified Teachers 43 percent of the
  states went beyond the Title II Ds 25 minimum
  funding requirement to focus additional resources
  toward professional development. 66 of states
  reported judging the quality of their competitive
  grant applications against evidence-based
  principles on professional development.
• Student Achievement Over 40 of states required
  LEAs that received NCLB II D competitive grant
  funds in their states to focus on reading or
  mathematics. States are not only building the
  conditions essential to effective technology use,
  but they are also beginning to see results as
  measured in increased student learning.
• Technology Integration 25 of states currently
  mandate that technology planning be grounded in
  school improvement and/or that the school
  improvement plans include technology integration.

3
National Trends ReportResearch

• Sixty-one percent (61) of states require
  competitive grant recipients to report findings
  based on improvements as compared to baseline
  data.
• 45 of the states are providing training on
  program evaluation, and 43 of the states are
  providing guidance for EETT local evaluations.
• 28 of states have required some experimental or
  quasi-experimental impact studies to determine
  the impact of NCLB II D programs.
• Over 88 of states are collecting data annually
  from either districts, schools, or both. States
  are increasingly triangulating data sources
  (e.g., district surveys, school surveys, teacher
  surveys, student surveys, and site visitations).

4
National Trends ReportState Examples

• Alaska - Alaskas Write On! students posted
  significantly higher gains in language arts than
  those who did not participate in this technology
  writing initiative. 5th graders showed more
  positive results in Writing, and 6th graders
  showed more positive results in Reading.
• Michigan - Freedom to Learn (FTL) one-to-one
  initiative 7th-grade reading scores jumped from
  29 to 41 percent, and 8th-grade math scores
  increased from 31 to 63 percent.
• Louisiana Online Algebra I 98 of grade 8
  students in the online Algebra course scored
  Basic or above on Louisiana's high stakes, LEAP
  21 test, with 43 scoring Mastery or above
  online grade 9 students had a higher mean score
  on the IOWA math exam than the control students
  84 of students enrolled in the Algebra I project
  successfully completed the course, up from 61
  the previous year.

5
E2T2 ProfessionalDevelopment Model

• Gary Phye
• Iowa

6
EducationalInterventionE2T2ProfessionalDevelop
mentModel4th Grade MathCoSN 2006
7
MATH - 4th Grade
Keystone
Council Bluffs
Council BluffsKeystone
8
Plots of Experiment vs. Control based on NCE
scores (Reading Total or Math Total)
MATH - 4th Grade
Notice both the difference between starting
points and a large difference between slopes
(targeted group started with lower avg. scores
and improved at greater rate than control
group) This difference between slopes is
significant (at .05 level) Experimental group
Effect Size 0.487
9
M1 Effect size 0.312M2 Effect size 0.168M3
Effect size 0.255
MATH - 4th Grade
M1 Math Concepts EstimationM2 Math
Problem SolvingM3 Math Computation
10
Strategies
Keystone
11
This test compares the mean score difference for
year 1 and year 2 between the Experiment group
(participants) and Control group
(non-participants). The result ( p lt 0.05 ) shows
that the mean difference for 2 years is
significantly different between control group and
experiment group at 0.05 level.
Keystone (4th Grade Math Total)
Effect Size (Exp Group) 0.43
12
KEYSTONE
Math Grade 4 (Low-Med-Hi Analysis)
Note, only analyses for Low and High were found
to be statistically significant. The medium
group, while not significant, is depicted in the
Low-Med-Hi Analysis just for informational
purposes.
Part A Low Group (ITBS)
Effect Size (Math) 1.13
13
Not significant
KEYSTONE
Math Grade 4 (Low-Med-Hi Analysis)
Part B Middle Group (ITBS)
Effect Size (Math) 0.23
14
KEYSTONE
Math Grade 4 (Low-Med-Hi Analysis)
Part C High Group (ITBS)
Effect Size (Math) - 1.04
15
EVALUATING WEST VIRGINIAS EETTMODEL SCHOOLS
PROJECT

• Dale Mann, Ph.D., Managing Director
• Interactive, Inc.

16
Summary IN THE TIS SCHOOLS

• There are statistically significant gains in
  WESTEST Reading scores from students moving from
  the 4th to 5th grades.
• Students use computers more.
• Teachers use computers more.
• Teachers use computers more for productivity
  applications.

17
West Virginia StatewideAchievement Gains
Background

• Q Can computer use improve student learning?
• A YES

18
West Virginia StatewideAchievement Gains Current

• Q Can computer use improve student learning?
• A YES 5th Grade students in the TIS schools
  (compared to non-TIS schools) made statistically
  significant score gains on the states WESTEST.
  

19
From BS/CE to Technology Model Schools

• Question How does technology improve
  achievement?
• Answer When it is used by teachers and
  students.
• Question How can more teachers be encouraged to
  use more technology?
• Answer In-school technology helpers (TIS) in
  Technology Model Schools

20
Policy Questions

• Did the in-school, TIS strategy work and how do
  we know?
• Now that the TIS specialists are done (June
  2005), what is happening? Are teachers
  continuing to use technology?

21
RESEARCH DESIGN
22
Experimental/Control Pre/Post
23
Using Technology To Measure Technology

• Pagers on teachers web surveys
• Free pager rings once per day, every other week
• When pager rings, teacher and a randomly selected
  student take a web-survey
• Pages are randomly scheduled during alternate
  weeks (not more than 5 per week)

• Meters in desktops
• Software runs in the background 24/7
• Records file activity only
• Installed on classroom computers only
• Data e-mailed directly to Interactive, Inc.
• ZERO burden on teachers

24
HOW DID WE DO COLLECTING DATA?
25
Pagers Web-Surveys

• 2,100 pager-triggered responses from teachers
• 2,000 responses from students.
• Responses from 110 different teachers.
• Individual teacher responses range from 1 to 36.

26
Computer Monitors

• Data from 184 computers in 25 schools.
• Installed on 287 computers.
• 2 reports a day X 184 computers X 5 months
• 35,000 e-mail reports of computer use.

27
Triangulated Data

• Paper pencil qres from 90 of all 4th 5th
  grade teachers
• Paper pencil qres from 100 of the TISs
• Computer meters
• Random interval web surveys
• TIS logs
• Interactive, Inc. SDE f2f observations and
  interviews
• ? Self-report, smile check data

28
FINDINGS (Based on Web-Based Survey Data)
29
More Student Computer Use In TIS Schools
At the time the pager was activated, were any of
your students using computers?
30
More Teacher Computer Use In TIS Schools
At the time the pager was activated, were YOU
using computers?
31
More Teacher Productivity Use In TIS Schools
What were you doing?
32
Summary IN THE TIS SCHOOLS

• Students use computers more
• Teachers use computers more
• Teachers use computers more for productivity
  applications
• There are statistically significant gains in
  WESTEST Reading scores from students moving from
  the 4th to 5th grades.

33
Questions?

• Interactive, Inc.
• 61 Green Street
• Huntington, New York 11743phone 631 351
  1190fax [phone]
• e interinc_at_aol.com

34

• ED PACEEducational Development for Planning and
  Conducting EvaluationsSaul Rockman
• ROCKMAN ET AL

35
Goals

• Provide empirical data on student achievement in
  virtual foreign language courses using a
  quasi-experimental design
• Develop a framework for scientifically based
  research that can be used at the local, state,
  and national level to measure the impact of
  technology enhanced programs on student
  achievement

36
Summative Model

• Research Questions
• Does participation in Virtual Spanish program
  affect
• Student achievement and Spanish learning?
• Longer term student planning / interest?
• Method
• Quasi-experimental

• Outcome Measures
• State standardized test
• Writing assessment
• UNIACT Interest Inventory
• - ACT Explore (8th)
• ACT Plan (10th)
• Spanish assessment

37
Illustrative Implementation Findings thus far

• Overall satisfaction (survey data)
• High and consistent levels of satisfaction among
  students, staff, and administrators.
• High levels of satisfaction among parents in the
  targeted sites 90 believe their child had made
  a lot of progress 82 report that progress
  exceeded expectations.
• Positive attitudes among almost all VS students
  (nearly 90) toward learning a foreign language
  and doing so in a VS environment.

38
Spanish Achievement Summary

• Overall, VS and F2F students performed very
  similarly on the Spanish assessments.
• VS slightly outperformed F2F on the language
  section of the multiple-choice assessment
• F2F slightly outperformed VS
• On the listening and reading comprehension
  sections of the multiple-choice assessment
• On some aspects of the written Spanish assessment
• On the oral Spanish assessment

39
Spanish Achievement Summary

• Comparing these two groups is informative because
• Many students were tested - approximately 250
  Virtual Spanish and 170 face-to-face students.
• The two groups were very similar in English
  language achievement prior to participating in
  Spanish. (Thus, it is unlikely that any
  differences in their Spanish achievement are due
  to the students general language skills).

40
Spanish Multiple Choice
Virtual Spanish students scored slightly higher
on the Language section and slightly lower on the
Listening and Reading sections.
41
Spanish Written Assessment
The two groups scored similarly on the Written
Assessment. VS scored slightly higher in 2
areas, slightly lower in 4 areas, and the same in
one area.
Ratings could range from 1-4
42
Spanish SOPA (Oral)
F2F students scored higher on the oral
assessment.
Ratings for Spanish 1 students typically range
from 1-5
43
Instructional Teams Use of Spanish

• Student performance was higher in sites where
  facilitators
• modeled Spanish for students.
• asked students to respond in Spanish.
• checked for understanding, provided feedback.
• compared aspects of English and Spanish.
• In sites where facilitators used more Spanish,
  students performed significantly better in
• Listening, reading, writing

44
Students Spanish Use

• Higher student performance in reading, writing,
  and listening was associated with
• oral practice
• oral communication
• reading Spanish silently and out loud
• Lower performance in reading, writing, and
  listening was associated with
• close-ended activitiesfilling in the blank,
  matching, completing sentences
• Students tend to score higher in reading
  comprehension when they spend more time on
  computer-based activities.

45
Communication and Collaboration

• Students performed better in Spanish listening,
  language, reading, and writing in classrooms
  where
• facilitators gave students more contingent,
  specific feedback.
• students were more likely to initiate
  communication with the instructional team.
• Students performed better in listening and
  writing in classrooms where
• there was more communication among instructional
  team members.

46
Research on the Effectiveness of Two Models of
Implementing Educational Technology
Barry Golden Wisconsin Department of Education
SETDA
47
Inquiry Based Research Models

• Big6 Information Literacy (Problem Based
  Learning)
• 61 Trait Writing Model (Inquiry Approach)
• Control Group (No Interventions)
• 34 Districts 96 Teachers 8,000 7/8th Science
  and Social Studies

48
Data Types

• Pre-Post Teacher Training Survey
• Annual Pre-Post Teacher Survey
• Teacher Model Fidelity Surveys (4X/yr.)
• Student Survey (4x/yr.)
• Annual Achievement testing
• Classroom Observations
• Work Sample Analysis (5x/yr.)

49
WiLATA Taxonomy
Remember
Understand
Apply
Analyze
Evaluate
Create
50
(No Transcript)
51
WiLATA

• Wisconsin
• Learning and Teaching Assessment
• Cognitive Dimension
• Meta-cognitive Dimension

52
Cognitive Dimension (Teacher-Student)
53
Cognitive Dimension RatingAutomatic
Recall Synthesize Analyze Apply Judge
Create

• 0Lacking
• 1Minimal
• 2Good
• 3Strong
• 4Thorough

54
Meta-Cognitive Dimension(Teacher-Student)
55
Meta-Cognitive RatingNon- meta-cognitive
Declarative Procedural Conditional

• 0Lacking
• 1Minimal
• 2Good
• 3Strong
• 4Thorough

56
WWW.SETDATAPP.ORG