A Wearable Gestural Interface SixthSense Technology

1
A Wearable Gestural Interface
SixthSense Technology

• Neha Kapoor

2
What is SixthSense ?

• SixthSense is a wearable gestural interface
  that augments the physical world around us with
  digital information and lets us use natural hand
  gestures to interact with that information.
• SixthSense bridges the gap between physical
  world and digital world, bringing intangible,
  digital information out into the physical world.
• SixthSense frees information from its confines
  by seamlessly integrating it with reality, and
  thus making the entire world your computer.

3
Why WUW ?
Microsoft Surface
Apple Iphone
Perspective pixel
Diamond Touch Tabletop
Oblong g-speak
Ubiq Window
4
Limitation of existing systems.

• Most of the Multi-touch based interactive system
  are not mobile.
• Small devices fail to provide the intuitive
  experience of full-sized systems.
• Information still resides on screens or dedicated
  projection surfaces.
• System depends on physical touch-based
  interaction.
• Oblongs g-speak have expensive hardware setup
  that requires calibration.
• Most devices lack Portability.
• Information not projected on variety of surfaces.

5
How WUW differ ?

• Portable.
• No Hardware interface required Information is
  projected onto the objects, walls, surfaces etc.
• No physical touch required User interacts with
  projected information using hand gestures.
• Tends to merge physical data with digital data.
• Prototype costs not more than 350.

6
Prototype

• Hardware
• A Pocket Projector (3M MPro110)
• A Camera (Logitech QuickCam)
• Colored Markers
• Software
• Microsoft Windows platform (C)
• WPF
• OpenCV

7
Working

• Camera recognizes and tracks users hand gestures
  and physical objects using computer-vision based
  techniques.
• The software program processes the video stream
  data captured by the camera and tracks the
  locations of the visual tracking fiducials at the
  tip of the users fingers.
• The projector projects visual information
  enabling surfaces, walls and physical objects
  around us to be used as interfaces.

8
Gestural Interaction of WUW

• Gestures supported by multi-touch
• based interactive products such
• as Microsoft Surface and Apple
• iphone.
• Examples include
• Zoom In
• Zoom Out
• Draw on surface using index finger
• Flip through documents/images
• Pan in a map application

Cont.
9

• Freehand gestures
• Framing Gesture used for taking pictures.
• Namaste Gesture for navigating to homepage
• Iconic gestures Draw icons/symbols in air using
  index finger.
• Star symbol launches weather application.
• Magnifying glass symbol for map application.
• _at_ symbol let the user check mail.
• X symbol undo an operation.

10
Future Work

• Plan to investigate more sophisticated
  computer-vision based techniques for gesture and
  object recognition.
• Technique that do not require the user to wear
  color markers.
• WUW prototype to be made more stylish to make it
  more socially acceptable.
• Intends to implement small solar panel that so
  that system can work with sustainable energy.

11

• My Research Work

12
Quickies

• Quickies are stickies that have some
  intelligence and the ability to remind us about
  the task we ought to perform or to provide us at
  the right time with the information we captured
  in the past.

Example of Quickies
13
Working of Quickies

• By the use of NPL, AI, and ConceptNet quickies
  can help us to seamlessly connect our physical
  and digital worlds.

• Information is captured as digital note with
  help of Digital pen hardware.
• Computer program transforms this note to
  computer understandable text.
• Commonsense knowledge engine processes and
  understands text to perform appropriate action.

Quick view of how quickies work
14
Tangible Public Map

• Tangible public map allows people to use their
  own belongings, objects they carry with them to
  access relevant, just-in-time information and to
  find locations of places.

Example of TaPuMa
15
Working of TaPuMa

• TaPuMa identifies the object, provides relevant
  options related to object and further augments
  the information (eg.location of objects) onto the
  interface.

• Camera tracks the object
• and its location.
• OpenCV identifies the object.
• Software program provides relevant options
  based on object.
• Projector projects location along with the
  map.

Quick view of how TaPuMa work
16
Language Translators

• This feature would give an easy way such that the
  user can interact with projected information in
  their own selected language.
• It can be another application in WUW, that gives
  an option to select and interact with information
  in their native language. Eg. Japanese person
  sees all instruction and projected information in
  his native language.

17
My Future Work

• Working of Sixth Sense System.
• Working of above explained applications.
• Integration of Quickies, Tangible Public Map and
  Language Transform applications with Sixth Sense
  system.

18

• Thank You