Utilizing Life-like, 3D Animated SigningAvatar

1
Utilizing Life-like, 3D Animated SigningAvatar
Characters for the Instruction of K-12 Deaf
Learners

• Mr. Jason Hurdich
• Team Leader, Sign Language Projects
• Vcom3D, Inc.
• Orlando, Florida, USA

2
Company Background

• Vcom3D, Inc. is a leader in research and
  development related to the instruction of K-12
  Deaf and Hard-of-Hearing (Deaf/HH) students.
• Our SigningAvatar characters provide a striking
  increase in many childrens attention span, their
  level of engagement, and academic results.
• During an evaluation at the Florida School for
  the Deaf and Blind (FSDB), the increase in
  comprehension when shifting from text-only to
  text accompanied by sign language by use of the
  SigningAvatar technology was 17 to 67.

3
Sign Smith Development
4
Sign Smith Studio for Development

• Vcom3D Sign Language Team uses Sign Smith Studio
  to script content into ASL format.
• Helps Deaf and Hard-of-Hearing children
  understand the content in their native language.
• Provides learning opportunities outside the
  traditional classroom.
• Used to create Signing Science Dictionary/Pictiona
  ry and Signing Math Dictionary to provide STEM
  access to children of different age and levels.
• Promotes independent learning.
• Research and additional development features will
  eventually become upgrades to commercial product.
  

5
Gesture Builder for Development

• Vcom3D Sign Language team uses Gesture Builder
  to
• Create specific signs not available in the
  existing lexicon library.
• Create signs that match concepts more
  specifically (i.e. regional signs).
• Create signs needed for custom content.
• Create signs for foreign sign languages.

6

• Early Research
• Life-Like, Expressive Avatars

7
Early Research with TERC

• TERC, in collaboration with Vcom3D, used
  SigningAvatar accessibility software to create
  Signing Science Dictionary and Pictionary, and
  Signing Math Dictionary (in 2009).
• Illustrated interactive 3D sign language
  dictionaries for young Deaf/HH children.
• Funded in part by National Science Foundation,
  NEC Foundation of America, and the U.S.
  Department of Education.
• Resource for teachers at school and parents at
  home.
• Expanding product line!

8
Research Supporting Sign-to-Text Connections

• Wilson Hyde (1997) - comprehension
  significantly increased with signed pictures
  connected to text
• Sims (2000) - comprehension increased from 17 to
  67 after using sign-enabled content
• Vesel (2004) - pre-test D/HH students mean score
  lower than hearing students post-test they score
  higher

9
TERC Research Evaluations

• Overall Average (Vesel 2004)
• Deaf/HH Hearing
• Pre 1.37 1.99
• Post 3.55 3.38
• Pre .37 1.72
• Post 3.62 2.81

Weather -Elementary
Oxygen -Middle
10
Mobile Language Reference for Deaf
Hard-of-Hearing K-12 Students

• U.S. Department of Education Phase I and II
  funded research.
• Vcom3D showed evidence of the success in
  utilizing mobile devices to make ASL information
  accessible on the go for K-12 students.
• Content includes definitions and explanations of
  idioms, scientific terms, multi-sense and
  unfamiliar terms which Deaf/HH students often
  have trouble understanding.
• With Phase 2 funding, Vcom3D made Signing Science
  Dictionary accessible via Apple iPod Touch and
  iPhone as seen here.

11
Phase I Evaluation Results Mobile Language
Dictionary

• The following average evaluations by educators of
  Deaf/HH upper elementary and middle school
  students (Hurdich and Sims, 2007)
• Ease of Use 8.3 out of 10
• Motivation 10 out of 10
• Supported Reading Comprehension
• 7.7 out of 10

12
Phase II Mobile Language Dictionary

• In October 2007, Vcom3D, Inc. began the Phase II
  SBIR project (which runs through September 2009).
• The Clerc Center did a preliminary study designed
  to determine the ability of educators to use
  Authoring Tools for creating signed instructional
  materials for their students and the potential
  impact of these materials on teaching and
  learning.
• This reference tool includes animated
  translations from English to ASL of multi-meaning
  terms, idioms and difficult words that Deaf
  students often face in literacy.
• Technology Goal - Develop Mobile Signing Science
  Dictionary for Deaf/HH K-12 Students using the
  Apple iPod Touch.
• Educational Goal - Improve a students ability to
  learn ASL grammar and to reinforce a bilingual,
  bicultural educational approach.

13
Evaluation of iPod Touch at the Clerc Center

• This component of the evaluation is a preliminary
  study designed to begin to find out about the
  ability of educators to use Sign Smith Studio to
  create signed instructional materials for their
  students and the potential impact of these
  materials on teaching and learning.
• The evaluation focused on four areas
• 1) Ability of one educator to use Sign Smith
  Studio to create signed instructional materials
• 2) Accuracy of the signed materials
• 3) Ability of the educator to integrate the
  signed materials into the curriculum and
• 4) Impact of the signed materials on students
  ability to read and express ideas and on their
  attitude toward learning.

14
Students Reactions to iPod Touch

• The iPod helped me to do homework independently
  without needing to depend on parents.
• This iPod helped me to correlate what was taught
  in the classroom.
• It helped to reinforce my learning when parents
  were busy.
• The avatars are fun and cool!
• This helped to foster my independent learning.
• I would absolutely use this as a tool for
  learning.

15
Life-like, Expressive Avatars for the
Instruction of K-12 Deaf Learners

• National Science Foundation SBIR Phase I and II
  funded research.
• This project was conducted in partnership between
  Vcom3D and the Laurent Clerc National Deaf
  Education Center at Gallaudet University.
• Goal - To identify signing styles of both genders
  communicating in American Sign Language with
  young students.
• Based on the Shared Reading Project principles
  created by David Schleper (1997).

16
ASL readers go well beyond strictly interpreting
English text

• Based on Schlepers Principles for Reading to
  Deaf Children, Vcom3D identified several
  principles that could be addressed using our Sign
  Smith Studio sign language authoring tool
• Deaf readers re-read stories on a storytelling to
  story reading continuum.
• Deaf readers make what is implied explicit.
• Deaf readers adjust sign placement to fit the
  story.
• Deaf readers adjust signing style to fit the
  story.
• Deaf readers connect concepts in the story to the
  real world.
• Deaf readers use attention maintenance
  strategies.
• Deaf readers engage in role-play to extend
  concepts.

17

• During the same project, researchers found that
  appropriate use of facial expression is an
  important communicative skill for teachers
  working not only with Deaf/HH, but also with
  hearing students.
• Although the importance of facial expression is
  recognized, more research is required in order to
  develop a comprehensive language model of how the
  face is used to communicate in different contexts
  and in different registers.

18

• In order to enhance the expressiveness of our
  Virtual Tutor (also known as SigningAvatar
  characters) simulations, Vcom3D is researching
  the nuances of how the face and body are used in
  communication - in ASL, and teaching in general.
  
• The outcome of our research is to improve the
  quality of our Virtual Tutors in areas such as
  modeling of facial expression, connection of
  these expressions with their meanings, and to
  support the instruction of children and teachers
  using our Sign Smith Studio product.
• Our research began by videotaping teachers at the
  Clerc Center.
• Using the developed proof-of-concept system, we
  videotaped a short story signed by a male
  teacher, Jason Stewart, at Kendall Demonstration
  Elementary School (KDES).
• Vcom3D analyzed the video, wrote an English
  transcript, and animated the story.
• The animated version is 4 minutes of a story
  called The Forest.

19
Phase I Evaluation Process

• Students at KDES and American School for the Deaf
  participated in the evaluation process of our
  newly developed Life-like Expressive Avatars.
• The students were evaluated in their
  comprehension of the materials with and without
  the aid of the software.
• The students were asked to
• Read a few passages and answer comprehension
  questions without using the software.
• Approximately two weeks later, the same reading
  passages were shown with the aid of the 3D
  animated story.
• Take another comprehension test.

20
Initial reactions from KDES Students

• It helped me to correlate with the signing and
  reading English. It also helped me to reinforce
  what I was reading allowing me to compare
  stories. I thought the avatar looked like a real
  person.
• The avatar signed clearly and I was able to
  fully understand the story better. I felt I was
  able to learn more.
• I would much rather have the avatar sign to me
  over a Deaf person if I had an iPod based on the
  avatars ability to show actions and
  expressions.
• I would much rather view ASL than reading
  English, and would rather have avatars during
  testing. It did help me understand the questions
  better.
• I felt I was able to understand the story more
  in depth, helped me to read better, and the
  avatar made no mistakes in signing. It was so
  cool!

21
Initial reactions from KDES Teachers

• What I liked most about the avatar was he had
  smooth movements, clear signs, and the face was
  very expressive.
• My students liked the animation -- the story
  became more interesting and understandable.
• The signing of the avatar and facial expressions
  was clear.
• The avatar was identical to the actual signer.
• I think over time that life-like expressive
  avatars could support the development of English
  reading comprehension for unaided reading.
• The project was very useful for my students it
  helped a student who had trouble understanding
  the text during the post-test.

22
Educators Survey Data Summary- KDES and
American School for the Deaf
Question Average
1. Was it easy for your student to view the life-like expressive avatar(s) for reading comprehension? 9
2. How motivated were your students to use life-like expressive avatar(s) for reading? 9.8
3. How much do you think the life-like expressive avatar(s) supported the reading comprehension of the students who used it? 9.3
23
Phase II Research Lifelike, Expressive Avatars

• How to best identify range of human facial
  expressions in ASL.
• How to incorporate three levels of effective
  storytelling identified by Dr. Kristin Mulrooney
  (Gallaudet University) to make our characters
  more life-like
• (1) grammar
• (2) emotions
• (3) communication

24

• Significance and Implications of
• Research Projects

• The NSF-funded research will result in improved,
  computer-based reading instruction for the
  50,000 K-12 Deaf/HH students in the U.S. whose
  first language is ASL. Currently, Deaf children
  are delayed in developing language skills, to the
  extent that the average reading level of a Deaf
  high school graduate is no greater than 4th
  grade.
• The Dept. of Education-funded research will
  result in a Mobile Language Reference tool that
  can be taken anywhere to supplement the
  curriculum of the Deaf/HH student. It will
  contain a library of definitions that are often
  difficult for these students to master.

25

• Vcom3D Products

26
Sign Smith Studio 3.0The ONLY sign language
authoring tool that gives you the ability to add
SigningAvatar characters to your digital content
easily! The characters sign your content and
make it accessible to your students or anyone who
could benefit from sign language.
27
Sign Smith Studio 3.0

• Benefits
• Saves time and money
• Create curriculum-specific, platform independent
  content
• No software animation experience needed
• Make changes in seconds unlike video
• Features
• Ten characters
• Coordinated facial expressions, signs and
  mouthing
• Automatic translation of over 10,000 English
  words
• Disambiguation for multiple meaning words
• 5 concurrent behaviors to synchronized for
  natural signing
• Export to Apple QuickTime Microsoft Windows
  MediaTM
• Use the exported videos in literacy and
  multimedia programs that support these formats

28
Gesture Builder 2.0

• No animation experience is required!
• Applicable tool for creating specialized sign
  language content
• Full index of hand shapes based on the ASL manual
  alphabet
• Signs created using Gesture Builder can be
  imported into Sign Smith Studio and reused on any
  of the SigningAvatar characters.

29
ASL Animations Volumes 1-5
The ONLY ready-made animated sign language
resource that you can start using right away!
Now with more than 1,200 signs that represent
1,600 English words!

• FEATURES
• Animated GIFs and MOVs
• Compatible with Mac and PC
• Choose from single-play and multi-looped
  animations
• Use in multimedia and literacy programs
  (IntelliTools Classroom Suite, Clicker 5,
  Kurzweil 3000 for Windows, SoftTouch products),
  PowerPoint and more!

30
Sign Smith ASL on iTunes

• Introducing the best way to learn American Sign
  Language on the new iPhone and iPod Touch App
  Store!
• SSASL Lite is FREE to download and contains 20
  basic signs.
• SSASL Essential is 0.99 to download and contains
  100 basic signs including the manual ASL
  Alphabet.
• SSASL Ultimate is 4.99 and contains all 1,200
  signs from ASL Animations.
• Great reviews and more than 25,000 downloads in
  one month!

31
Sign4Me on iTunes

• Developed by our sign language team, which is
  comprised of 8 Deaf professionals, Sign 4 Me
  gives the user what they need to learn as little
  or as much sign language as they want!
• Sign 4 Me provides sign language instruction in
  3D! Our 3D character can be zoomed in or out and
  rotated to give you the best vantage point for
  every sign. You control the placement of the
  character and the speed of signing. Zoom in
  closely to see the hands and how they are shaped.
• The sign language library contains more than
  11,400 words. As you type in sentences, phrases,
  and words, they are saved in your History. This
  allows you to look up previous content you have
  entered and keeps you from having to retype again
  and again. Go back to find your content either by
  date or alphabetically. You also have the option
  to loop the signing over and over so you can
  practice or just see it one time for reference!

32
Signing Science Dictionary

• The only illustrated interactive 3D sign language
  science dictionary.
• Designed for students who are Deaf/Hard-of-Hearing
  and use American Sign Language (ASL) or Signed
  English (SE) in the classroom.
• Great resource for teachers and parents.
• The SSD supports access to standards-based
  science content among elementary and middle
  school students.
• FEATURES 1,300 science terms and definitions
  All text available in ASL and SE Most terms
  contain an illustration Use at school and at
  home

33
Thank You

• Contact
• Jason Hurdich, jasonh_at_vcom3d.com
• Cindy Barson, cindyb_at_vcom3d.com, 407-737-8488 x121

34
Thank You!
Thank you for attending this session at ATIA 2009
Chicago! Watch for us in Orlando this
January! Please help us improve the quality of
our conference by completing your session
evaluation form. Completed evaluation forms
should be submitted as you exit or to staff at
the registration desk.