Multimedia Esperanto SDK

1
Multimedia Esperanto SDK A Free Software
Development Framework for multimedia
applications for people with disabilities
Games Accessibility Workshop November 2005,
Linz Philipp Emanuel Haindl University of Linz,
Austria
2
Overview

• SenseZero Multimedia Esperanto Project What
  is the Multimedia Esperanto Project ?
• Means of computation of Multimedia Esperanto
  How can Multimedia Esperanto be computed ?
• Multimedia Esperanto SDK for Multimedia
  Applications How can Game Developers make use
  of Multimedia Esperanto ?

Philipp Emanuel Haindl, University of Linz,
Austria
3
SenseZero Multimedia Esperanto Project

• Ambition Compute a Formal Multimedia
  Language (Video Sound) Multimedia rendered
  with respect to sensual and cognitive properties
• Color Pattern Enforcement / Color Substitution
  for easier perception
• Multimedia Esperanto Multimedia content
  uniformly perceivable independently on sensual
  or cognitive fitness
• Targeted user group Color Vision
  Deficiency Mild Cognitive Disabilities (e.g.
  Alzheimers Disease)
• Developers Students and recent graduates
  (computer science, linguistics, medicine,
  economics)

Philipp Emanuel Haindl, University of Linz,
Austria
4
SenseZero Multimedia Esperanto Project

• Ways of Participation
• See what you see Visual Archetype Tracer
• See what you see Measuring differences in
  color perception and predicting colors with
  Hidden Markov Models (Web Application)
• Visual Archetype Tracer
• Trace Visual Archetypes in arbitrary sets of
  images and exchange them with the community
  (Standalone Client Application)
• Prix Ars Electronica 2006 Project will be
  submitted to Net Vision Category
• Website http//www.sensezero.org

Philipp Emanuel Haindl, University of Linz,
Austria
5
Computation of Multimedia Esperanto

• Hidden Markov Models Analysis of the
  differences in color perception obtained from the
  results of the See what you see examination
  (for computation of color equivalents)
• Homology and Similarity Search (Fuzzy
  Smith-Waterman) Finding patterns in multimedia
  content unique for a certain meaning associated
  with the context (for computation of Visual
  Archetypes)
• k-Means-Clustering Avoid overfitting of the
  data being analyzed

Philipp Emanuel Haindl, University of Linz,
Austria
6
Computation of Visual Archetypes (Multimedia
Esperanto)

• What is a Visual Archetype Context-Aware
  Visual Pattern in a set of images
  Unique for the meaning being associated
  with the images in the analyzed context
  (sets of images with same semantics)
• Computation of Archetypes Searching for
  homologue and similar color patterns in
  multimedia content using Fuzzy
  Smith-Waterman Algorithm

Philipp Emanuel Haindl, University of Linz,
Austria
7
Multimedia Esperanto SDK for Multimedia
Applications

• Provides an easy to use API for multimedia
  applications / Image Renderers
• Based on Microsoft .NET 2.0 Framework
• Processing an image ? (Input) arbitrary
  image ? (Output) transformed image (doesnt
  differ much from the original image) Exchange
  colors and enforce structural patterns in an
  image with one single method invocation
• SenseZero Update Agent
• Keeps the color and pattern definitions on the
  clients up to date
• Applications Image Manipulation and Display
  Software, Streaming Services
  (compatible with
  Microsoft Media Encoder SDK), Computer Games

Philipp Emanuel Haindl, University of Linz,
Austria
8
Transforming images with the Multimedia Esperanto
SDK

• Prerequisites
• Assure the client has the SenseZero Update
  Agent installed on the local machine Add the
  .NET library SenseZeroColorEquivalents.dll to
  the projects references
• Transforming an image Create a new instance of
  the ColorEquivalents class Load the local
  matrix by calling LoadEquivalentsMatrix of the
  engines instance Calling TranslateBitmap
  with the bitmap, which should be transformed, as
  the only parameter will return an image
  with inforced patterns and exchanged colors

Philipp Emanuel Haindl, University of Linz,
Austria
9
Transforming images with the Multimedia Esperanto
SDK

• Now a more practical demonstration of the SDK

Philipp Emanuel Haindl, University of Linz,
Austria