EIA9806822:

1
EIA9806822 Automated Understanding of Captured
Experience Georgia Institute of Technology,
College of Computing Investigators Irfan Essa,
G. Abowd, C. Atkeson, K. Ramachandran
Project Goals To reduce substantially the human
input necessary for creating and accessing large
collections of multimedia, particularly
multimedia created by capturing what is happening
in an environment.
Captured Experience During a Lecture, displayed
as a WWW page
Outcomes (provide each where applicable) Ideas
Extended use of automated capture in
everyday environments (classroom/office/home)
over longer periods of time and the investigation
of providing automated speech and visual
transcription, with accompanying search,
retrieval and visualization techniques. Tools
Tools that support the inclusion of automatically
sensed context as meta-data to tag captured
information. This architectural approach has been
instantiated as a toolkit (the Context Toolkit)
and used to produce prototype capture
applications. Other TOOLs include INfrastructure
for Capture and Acess (INCA) and Stampede. People
The Classroom 2000/eClass system has been
presented on a variety of local, national and
international science and education programs as a
demonstration of how ubiquitous computing
technology can impact the future of education.
Many Post-Docs (2), PhD MS Students (gt10), and
Undergraduates (gt20) have helped develop and have
used the technology developed in part by this
funding are now working in academic, research,
and industrial positions.
2
EIA9806822 Automated Understanding of Captured
Experience Georgia Institute of Technology,
College of Computing Investigators Irfan Essa,
G. Abowd, C. Atkeson, K. Ramachandran
Vision-based Tracking of Activities
Recognition of Flipping Pages using HMMs
Annotation of Objects Interacted with
Interactive Web pages generated by the Classroom
2000 / Eclass System after a Lecture.
3
EIA9806822 Automated Understanding of Captured
Experience Georgia Institute of Technology,
College of Computing Investigators Irfan Essa,
G. Abowd, C. Atkeson, K. Ramachandran

• Impact
• We developed methodologies that supported
  extended use of captured experiences classrooms
  (primarily, with a few experiments in the home
  and office scenarios) over longer periods of
  time. To support this work, we researched and
  made significant contributions by
• Taking the existing Classroom 2000 software
  systems and extended it to use automatic speech
  transcription, with accompanying search,
  retrieval and visualization techniques.
• Software engineering principle were extensively
  used to provide infrastructure to support such
  large scale capture and access scenario and allow
  for easy interface between capture, sensing, and
  usabilty requirements (with the goal of reducing
  the load on the user). Resulted in Context
  Toolkit and the INfrastructure for Capture and
  Access (INCA) Toolkit
• Developed a distributed Programming Structure
  (Stampede) to support such high-end multimedia
  processing.
• Developed computer vision-based methods to track
  people and recognize their activities within a
  specific context.
• System in use for educational transcription. Used
  by many students and educators.

Barriers Opportunities Developed systems and
infrastructures allow for extensions to many
domains. Aware environments will serve towards
providing us with ability to sort through and
find relevant information as we go about our
daily activities. Complex environments like homes
and offices provide very interesting
opportunities. Further opportunities lie in the
support of challenges associated with cognitive
and/or age-related impairments.
Website http//www.cc.gatech.edu/cpl/projects/AUC
E