Measuring Outcomes of Assistive Technologies: Case Studies from a School System Session S2103

1
Measuring Outcomes of Assistive Technologies
Case Studies from a School SystemSession S2103

• Dr. Cheryl Temple and Denise Belmont
• ATIA January 2009
• Assistive Technology Services
• Fairfax County Public Schools
• Fairfax, Virginia

2
Session Outcomes

• Participants will learn the data collection
  designs, procedures, and measurements used to
  evaluate effectiveness of assistive technologies.
• 2. Participants will learn about the outcomes of
  each of the case studies and the influence of
  assistive technologies for classroom instruction.
• 3. Participants will learn about using data to
  support effective educational decision making
  regarding the use of assistive technologies for
  student achievement.

3
Our Purpose for Collecting Data

• Why we decided to collect data and information
  about our services
• To demonstrate efficacy of our AT services for
  students and to inform decisions related to AT,
  including program planning and budgeting
• To measure the effect of AT on the academic
  progress of students with disabilities
• To highlight the importance of using formative
  assessment when making AT decisions for
  individual students

4
How We Started Year 1

• Began with single study
• Started with mechanics of writing with 4th grade
  students
• Wanted to follow students and their writing
  abilities over 2 years using AT and find out if
  these students were able to pass state writing
  tests.

5
How We Started Year 2

• Expanded the topics we decided to explore
• included student use of AT
• services provided to students, staff, and
  families
• special programs offered by our department
• Used staff volunteers to participate in
  collecting data for topics of their interest

6
Where We Are Now Year 3

• Continue to examine student use of AT
• Collaborate with George Mason University in small
  scale studies on the use of accessible
  instructional materials

7
Data Collection Plan

• Created plan for what to study, how to collect
  information, and how to analyze the information
• Speech Recognition
• Writing Camps
• Longitudinal Writing Study
• Writing Mechanics and Composition for 5th and
  11th Grades using AT
• Phonics Software
• AAC Trainings
• High Stakes Writing Assessments

8
Speech Recognition
Did a students writing improve over time with
the use of speech recognition software?
9
Data Collection

• 3 students, all trained on speech recognition
  program (Dragon Naturally Speaking v.9.0)
• Design
• Baseline data was collected in fall 2007 using
  speech recognition software.
• Final data was collected in spring 2008 using
  speech recognition software.
• Data factors, such as needed prompts, were
  collected from ATS resource teachers for each
  writing sample.

10
Discussion and Implications
Student A
More sentences/paragraphs in spring Uses
software independently No mechanical errors in
spring data only 2 errors in baseline data For
this student, the use of speech recognition
eliminated most mechanical errors. What is
unknown is how many student would have had
without the use of speech recognition. Took less
time in spring but produced less work Added more
supporting information in the spring sample
qualitatively which is also supported by an
increase in the number of words/paragraph
11
Discussion and Implications
Student B
Increased independence of software use over
time. Speech recognition was able to provide end
of sentence and capitalization support embedded
within the software program. However, internal
sentence punctuation, such as commas, were still
problematic. Student understanding of grammatical
rules will still be important to teach directly.
12
Discussion and Implications
Student C
Uses software independently and did not require
much prompting for the writing process More
words/sentences and much more sentences/paragraphs
in spring Was able to work in the same amount
of time but was producing more work as evidenced
by of words Over time, students can become
more productive with writing using speech
recognition
13
What have we learned?

• Speech recognition can support a student with
  some writing mechanics but not all.
  Capitalization and ending punctuation can be
  supported through speech recognition software.
• As students become more proficient in the use of
  speech recognition software, they are able to
  write more in the same amount of time.

14
WRITE ON

• How are teachers who participated in a two week
  multimedia student writing camp using AT in their
  classrooms?

15
Data Collection

• 8 teachers attended Write On Camp Training and
  participated in 1 week AT training and 1 week
  direct student support during a writing camp for
  5th grade students
• Design
• Survey sent to 6 teachers (2/3 special educators)
  via Survey Monkey with 100 response rate
• Survey sent in Spring 2008, 9 months after
  training
• Survey developed in conjunction with camp
  director
• Survey Monkey provides an analysis by percent for
  each question. Data analyzed based on number of
  respondents

16
Discussion and Implications

• After camp, 2/3 of the teachers ask students to
  use AT when completing paragraph writing
  assignments. Some use it for daily and weekly
  journals.
• 2/3 of the teachers think AT may improve a
  students writing 2/3 think AT has a positive
  effect on student academic progress 2/3 feel
  confident in supporting student use of AT
• Teachers recognize the benefit of using AT to
  improve students writing
• Several teachers did not work in a classroom
  setting requiring practice in writing skills. 2/3
  of the teachers believe that learning the
  software during camp had no effect on their
  teaching practices
• Perhaps the reason that 2/3 believed there was no
  effect is because they are in a situation that
  requires no AT at this time. Future camps may
  want to consider the appropriateness of the
  teacher participants.
• Some teachers wanted more time practicing with
  the software
• Even with a 2 week experience, some teachers need
  further follow up support to feel comfortable
  with the software.
• Some teachers believe learning the software
  helped them to provide support to all students
  with writing difficulties

17
Frequency of Software Use

• Daily, Microsoft Word is used most frequently.
• Monthly, Inspiration is used most frequently.
• DraftBuilder is very rarely used.
• Perhaps Microsoft Word is used most frequently
  because it is the most accessible, least
  restrictive, and less stigmatizing

18
Examples

• Did learning the assistive technology software
  during WRITE ON Camp change your educational
  practices?
• Gave me confidence to expose students to new
  technology/software and to work with the students
  in expanding both their skills and my own.
• In what ways, if any, do you think your use of
  assistive technology to access the curriculum OR
  for instruction might be different as a result of
  the training?
• It really helps me to differentiate instruction
  for struggling readers and writers in ways that
  are much easier than they would have been years
  ago.

19
ROCK ON

• How did the students feel the ROCK ON experience
  benefited them during the school year?

20
Data Collection

• 12 eighth grade students
• Design
• Structured personal interview of students during
  school hours, 6 8 months after camp
• To ensure accuracy of student comments, both
  researchers took notes during the interviews
• Notes were compared and common themes were
  extracted

21
Discussion and Implications

• Things about Camp students remember
• Mobile team challenge, using computers, using
  SOLO, using graphic organizers, friends, using MS
  Word, writing and typing, responding to writing
  prompts, being filmed by Apple 21
• Students favorite things about Camp
• Goo Software, Mobile Team Challenge, Snack
• Important to have interactive physical activities
  to provide a variety in routine
• Social aspects of Camp is very important at this
  age
• Aspects of software students like
• Students like software that helps them plan
  ideas, that is easy to use, reads to them, which
  alleviates handwriting issues, supports spelling,
  and helps them with school work
• When students realize how software helps them,
  they are more willing to use the software

22
Discussion and Implications

• How do students use software?
• Students are using software for homework,
  essays, writing prompts. They are using software
  in English class with mobile carts. 5 students
  report not using any special software.
• There is not a lot of carry over of specialized
  software learned at Camp
• Reasons some students are not using software
• accessibility, belief that software slows them
  down
• Students are not proficient for the software to
  be a useful tool
• Self advocacy for software use
• 7 students told teacher about software, 4
  students shared the portfolio
• Students need to learn to self-advocate for use
  of software because some teachers are not
  providing access
• ATS resource teachers need to help students share
  previous AT experiences with new teachers

23
ROCK ON

• What are the AT classroom practices of the
  teachers who were invited to attend the follow on
  training to ROCK ON?
• How did attending the follow on training
  influence AT classroom practices?

24
Data Collection

• 13 teachers invited to follow on training
• 11 teachers sent survey via Survey Monkey, 5
  respondents
• Design
• Survey sent in Spring 2008, 5 months after
  training
• Survey developed in conjunction with camp
  director
• Survey Monkey provides an analysis by percent for
  each question of the respondents

25
Discussion and Implications

• Based on survey results (5 respondents)
• 80 have access to Mobile Laptop Cart available
  for the department
• All teachers asks students to use AT when
  completing 3-5 paragraph writing assignments.
  Some use it for weekly journals.
• Student use AT for longer writing assignments
  may not be efficient to use some AT for shorter
  assignments
• 80 of the teachers think AT may improve a
  students writing 60 think AT has a positive
  effect on student academic progress 80 feel
  confident in supporting student use of AT
• Teachers recognize the benefit of using AT to
  improve students writing not all students
  choose to take advantage of the AT student buy
  in is important if AT is going to improve
  academic progress
• As a result of the follow on training, 50 feel
  well prepared to use AT they learned 75 needed
  follow up support after the training. Support
  was provided on site by ATS resource teachers.
• One day training is helpful to begin the process
  of implementing AT in the classroom, further
  individual support is needed for most teachers

26
Software Use

• No teachers ask students to use software daily.
  This might be because of block scheduling.
• Weekly, Microsoft Word and CoWriter are used most
  frequently.
• Monthly, Inspiration is used most frequently.
• Write OutLoud is the software used the least in
  this sample.
• Perhaps Microsoft Word is used most frequently
  because it is the most accessible, least
  restrictive, and less stigmatizing

27
BLOG ONBuilding Language Organization and Grammar

• Effect on student writing during a school year
  after attending a two week long writing program?

28
Data Collection and Analysis

• 9 high school students attended BLOG On with 2
  week direct student support for writing
• Design
• Student writing was collected as draft and final
  products during BLOG On and at midyear and end of
  year following BLOG On Camp
• SPSS used to analyze writing mechanics and
  writing composition
• Writing composition data was evaluated using 11th
  Grade Composing SOL Test Rubric

29
Discussion and Implications

• All students Passed Proficient on End of Year
  Writing Tests.
• No discernable pattern was found in data for
  writing mechanics.
• Difference for most students in elaboration,
  organization, and unity.
• 2 week writing program made the most difference
  in writing composition skills and minimal impact
  on writing mechanics skills

30
Advocacy Questionnaire

• Self-advocacy questionnaire was administered to
  students at end of BLOG On program and at end of
  school year

Self Advocacy Questionnaire
31
Student Writing Samples

• There are many reasons why parents dont like
  FacebookThe adult strangers can send their lower
  body parts to children on Facebook.
• Alcohol affects the brain in dangerous ways.
  For instance it can hurt the brain so much your
  kid might not be normal.
• My brother helped and inspired me to become a
  football player. Every thing my brother did on
  the field I would do the same as him.Every thing
  Ive Learn I give all the credit to my brother.
• Im starting an international business plan. An
  international business company of a good smelling
  body soap. A body soap called SGS Smell Good
  Body Soap. Its a candy smell for a woman and a
  nice body smell for the man.

32
4th 5th Grade Writing A Longitudinal Study of
Students Writing

• How does using AT affect the mechanics of
  writing for elementary students on the ATS
  caseload over a two year period of time?

33
Data Collection

• 30 students in 06-07 data collection
• 14 students in 07-08 data collection
• data was analyzed if it was collected for all
  12 data sessions. This accounts for the
  attrition of 16 students.
• Design
• In November, February, and May, ATS resource
  teachers asked students to write in response to a
  prompt for 10 minutes not using AT and then a
  week later asked the same students to write in
  response to a similar prompt for 10 minutes using
  AT.
• No AT - handwritten or word processing with no
  adaptations
• With AT - Kidspiration, Inspiration,
  DraftBuilder, CoWriter, WriteOutLoud, NEO with
  CoWriter, MS Word with adaptations

34
For most students, using AT was beneficial for
decreasing spelling errors over the 2 year period
of time.
35
Discussion and Implications

• When students began study, they all had more
  errors in spelling when not using AT, except for
  2 for whom it made no difference.
• At the end of the study, they all had more errors
  in spelling when not using AT, except for 2 who
  made more errors using AT.
• Over the 2 year period of the study, 9 students
  of 14 (65) had fewer and 2 students of 14 (14)
  had the same number of spelling errors when using
  AT.
• For most students, using AT was beneficial for
  decreasing spelling errors.

36
AT software that was used in this study provides
minimal support in the area of punctuation.
37
Discussion and Implications

• When students began study, 5 had more errors, 5
  had less errors in punctuation when not using AT,
  and 4 for whom it made no difference.
• At the end of the study, 5 had more errors in
  punctuation when not using AT, 2 had more errors
  with AT, and 6 for whom it made no difference.
• Over the 2 year period of the study, 7 students
  of 14 (50) had fewer and 3 students of 14 (21)
  had the same number of punctuation errors when
  using AT.
• Using AT did not have as much an impact on
  decreasing punctuation errors as it did for
  spelling errors.
• AT software that was used in this study provides
  minimal support in the area of punctuation.

38
Using AT was beneficial for decreasing
capitalization errors.
39
Discussion and Implications

• When students began study, they all had more
  errors in capitalization when not using AT,
  except for 2 for whom it made no difference and 2
  who had more errors using AT.
• At the end of the study, they all had more errors
  in capitalization when not using AT, except for 4
  whom it made no difference. All students made
  fewer or the same number of errors when using AT.
• Over the 2 year period of the study, 10 students
  of 14 (71) had fewer and 3 students of 14 (21)
  had the same number of capitalization errors when
  using AT.
• Using AT was beneficial for decreasing
  capitalization errors.
• Some AT writing software provides support for
  capitalization automatically. Over half of the
  students were using CoWriter.

40
(No Transcript)
41
SOL 5th Grade Writing Test Results

• Of 11 students with data available 7 passed and 4
  failed
• Only 2 of the students who failed were using AT
  as a test accommodation

42
10 Minute Writing Samples Student 1
February 07 using No AT tigrsIliketigrsbecosthecan
ranfastand Ijustliketigrsbecastheyaremyfavaranmaa
l
February 07 using AT (CoWriter and
WriteOutLoud) My favorite food is pizza. I like
pizza because it has cheese on top. Pizza is very
very good.
43
Continued Student 1
May 08 using No AT A place I wolund like to vist
is egpet Epcot I think it would be cool to vist
my Dad said that he would take me there someday.
My Dad had been to egpet .
May 08 using AT (WriteOutLoud) My mom makes me
laugh. Mom always makes me laugh. She always do
funny things.
44
10 Minute Writing Samples Student 2
November 06 using No AT On the best day I ever
had I went to disiny world and went on a parets
at the carben ride and went swimig and also an
unter day was cristmas day and lots o toys and
peens and candys
November 06 using AT (CoWriter w/MS Word) My
best present was a pirate ship with toy pirates.
45
Continued Student 2
May 08 using No AT I wold like to visit italy and
see the Leaning towar of pezza and se the
calaseum. But ther is one thing I relly like to
do is try the Italain pizza. I herd they serv it
in rectanals mostly not crcles. I have a friend
name antany and hes bin to italy a lot of Times.
He brot back some coll stuff like this cool mask.
Id also like to visit astraulya. My ant went
ther She saw a kinds of coll stuff she ever got
to do bungy Jumping. Id like to do that
May 08 using No AT (WriteOutLoud)what I think
is funny is comedy shows like spungebob and
sumtimes siginfed. other then tv is playing with
my frends like when sumone brot glases for a nerd
coustum and eveyboty put the glases on and they
looked funny. And yo moma jokes are funny to.
46
Effect of AT on the Writing Process from
Brainstorming to Editing Using SOLO
How does the use of SOLO effect writing mechanics
and composition throughout the writing process
for 5th grade students with disabilities?
47
Data Collection/Analysis

• 8 students in 5th Grade at 1 elementary school.
• 4 students are on ATS caseload 4 students are
  not on ATS caseload
• Design
• ATS, Special Educator, and ESOL teacher planned
  and presented weekly writing lessons followed by
  writing prompt and practice
• Writing mechanics analyzed using SPSS
• Composition analyzed using 5th grade SOL writing
  rubric. Two coders were used to increase
  reliability.

48
AT helped the most with spelling over time,
especially students on caseload.
49
(No Transcript)
50
(No Transcript)
51
Discussion and Implications - Mechanics

• From baseline to final sample, there was minimal
  positive change in error reduction for
  capitalization and punctuation.
• AT helped the most with spelling over time,
  especially students on caseload.
• When using DraftBuilder, help is provided in
  spelling but not as much in capitalization and
  punctuation.

52
Composition Analysis

• Central Idea
• Elaboration
• Organization
• Unity

53
On ATS Caseload
A lot of variability but by the final sample, all
students had a 3 or 4 in Central Idea
Evaluation Criteria
54
(No Transcript)
55
On ATS Caseload
All but one student had a 3 or 4 by the final
sample in Organization
56
(No Transcript)
57
Discussion and Implications - Composition

• Students not on caseload were less variable than
  students on caseload.
• A lot of variability but by the final sample, all
  students had a 3 or 4 in Central Idea.
• All but one student had a 3 or 4 by the final
  sample in Organization.
• DraftBuilder does help a student with writing
  composition.

58
What Have We Learned

• When matching technology to student needs, if the
  needs are mechanics other then spelling,
  DraftBuilder does not help.
• When matching technology to student needs, if the
  needs are composition, DraftBuilder could
  support the students needs.

59
Growth of Writing Using Technology
How does the use of AT effect writing mechanics
and composition throughout the writing process
for 11th grade students with disabilities?
60
Data Collection

• 11 Students in 11th grade at 1 high school
• All students are on the ATS caseload
• Teacher with support from ATS resource teacher
  guided monthly writing assignments.
• Design
• Baseline - paper-pencil or portable word
  processor
• November - DraftBuilder
• December and January Student choice

61
Writing Assignments

• Each month, teacher gave a different type of
  writing assignment
• Baseline - book report
• November - compare/contrast essay
• December - persuasive essay
• January - SOL writing prompt

62
Data Analysis

• Writing mechanics analyzed using SPSS
• Composition analyzed using teacher created rubric
  based on 11th grade SOL writing rubric.
• The teacher scored the writing samples collected
  providing authentic evaluation.

63
Technology Use - Mechanics

• Spelling
• Capitalization
• Punctuation

64
3 of 9 students had initial decrease in spelling
errors and maintained the lower level over time.
6 of 9 students had a decrease in spelling errors
over time.
65
(No Transcript)
66
(No Transcript)
67
Discussion and Implications - Mechanics

• 6 of 9 students had a decrease in spelling errors
  over time.
• 3 of 9 students had initial decrease in spelling
  errors and maintained the lower level over time.
• There were no data patterns to report for
  percentage of punctuation and capitalization
  errors related to software use or trends over
  time. This is similar to the Effect of AT on the
  Writing Process from Brainstorming to Editing
  Using SOLO data project.
• When using DraftBuilder, help is provided in
  spelling but not as much in capitalization and
  punctuation.

68
Technology Use - Composition

• Central Idea
• Elaboration
• Organization
• Unity

69
7 of 9 students improved in composition skills
from baseline to final writing sample
70
Discussion and Implications - Composition

• 7 of 9 students improved in composition skills
  from baseline to final writing sample. 1 student
  stayed the same and 1 student scored lower than
  baseline (he used no technology for final 2
  writing assignments).
• Technology does help a student with writing
  composition.

71
What Have We Learned

• Technology can support a students spelling and
  composition abilities.
• When matching technology to student needs, if the
  needs are mechanics other than spelling,
  DraftBuilder does not help.

72
SOL Results
73
Phonics Software

• Planned to collect monthly reports of student use
  of Lexia
• Found that Lexia reports provided information
  useful by teachers for individual student
  progress
• Found that Lexia did not offer data that would
  have changed practices or informed decision
  making

74
ATS AAC Integration Series Training
Did attending AAC integration training influence
the use of AAC with students/child?
75
Data Collection

• 126 people who participated in one or more AAC
  integration trainings held during the 2007-2008
  school year (43 completed survey)
• Design
• Survey sent to teachers, therapists, and parents
  via Survey Monkey
• Survey sent in Spring 2008 at the completion of
  entire series of trainings
• Survey developed in conjunction with Trainers
• Survey Monkey provides an analysis by percent for
  each question

76
Discussion and Implications
Attending one or more of these trainings has
changed how I integrate AAC device(s) into daily
activities? 62.2 responded yes Being new to AAC,
the trainings gave me an understanding of how to
support teachers and students who use it. It has
allowed for a more consistent use across the day,
more opportunities and for showing others how to
better integrate the devices into their
classrooms. It has provided ideas for eliciting
communication more naturally across
settings. The student or child has increased the
use of their AAC device(s) as a result of the
training. 62.2 responded yes
77
Discussion and Implications
What are the biggest challenges to implementing
the strategies learned in the training? TIME The
age old...enough time to get to know the device
and then to program it's use so that it is used
in the most effective way for each student.
having everybody on the same team onboard Not
being in the classroom on a consistent basis
means that I can only suggest ideas, rather than
implementing them Demographics of
Participants Teacher or SLP from elementary or
preschool program with student accessing AAC
device by direct select. 50 of participants
have students/child using 1, 2, 4, or 8 cell
devices. 50 of participants have students/child
using dynamic display device.
78
High Stakes Writing Assessments
79
Sample size

• Total sample size 426 students in 5th, 8th, and
  11th grade on ATS caseload.

80
2007-2008 5th Graders on Caseload Who Took
Writing SOLs
53 of students on ATS caseload passed the
Writing SOLs
81
2007-2008 5th Graders on Caseload Who Took
Writing SOLs with and without AT
Of the 53 of students on ATS caseload who passed
the Writing SOLs
If students used AT, more passed than failed.
If students did NOT use AT, a higher percentage
failed than passed.
If students did use AT, less failed than those
who failed but did NOT use AT.
82
2007-2008 8th Graders on Caseload Who Took
Writing SOLs
53 of the students on ATS caseload passed the
Writing SOLs
83
2007-2008 8th Graders on Caseload Who Took
Writing SOLs with and without AT
Of the 53 of the students on ATS caseload passed
the Writing SOLs
If students used AT, more passed writing SOLs
than failed.
The same number passed and failed when NOT using
AT.
84
2007-2008 11th Graders on Caseload Who Took End
of Course Test
76 of the students on ATS caseload passed the
Writing SOLs
85
2007-2008 11th Graders on Caseload Who Took End
of Year Test
Sample size is smaller, data was collected from a
total of 46 students in 11th grade
Of the 76 of the students on ATS caseload passed
the Writing SOLs
Percentage of students passing is more than 3
times higher when using AT
Less students failed using AT than NOT using AT
86
Measuring Outcomes of Assistive Technologies
87
What Weve Learned

• Takes time and effort
• Need to be organized
• Need to consider attrition of subjects
• Need to share outcomes with those involved to
  eliminate reticence and aid in the understanding
  of their involvement
• Need to share outcomes to inform decision making
• The studies are limited in generalization
• Need to clearly identify the purpose of
  collecting and analyzing the data and how the
  analysis can be used
• Difficult to isolate AT as a variable therefore
  cannot provide causal relationships
• Teacher scoring provided assessment validity

88
Overall Benefits of Measuring Outcomes

• Can use data to support and validate AT service
  delivery
• Provides information to guide AT program
  decisions
• AT Staff can use data in presentations to school
  staff and administrators to encourage AT use
• Supports reflective thinking about AT practices