Trinetra Independence for the Blind and Visually Impaired

1
TrinetraIndependence for the Blind and Visually
Impaired
Phase 2 Detailed Design
2
HCI Rapid Prototyping Spring 06

• Sixteen of the total students in course
• Devoting one full semester
• Phase 1 Concept Exploration
• Phase 2 Detailed Design
• Phase 3 Prototype Implementation

3
Clients

• Priya Narasimhan
• ECE/CS Faculty
• Expertise embedded systems, distributed systems,
  fault tolerance, middleware
• Passionate about ensuring Trinetra leads to
  improvement in quality of life
• Dan Rossi
• Oracle system administrator
• Key driver and target user for Trinetra
• Key contributor in ensuring that the technology
  stays focused on the real needs and desires of a
  blind user

4
Presentation Order

• 1. HCI
• 2. Shopping
• 3. OCR
• 4. Transportation
• 5. Implementation Guidelines

5
Trinetra HCI Group

• Puneet Aggarwal
• Pablo Quinones
• Chris DeLeon
• Matt Lee
• Jim Lin

6
Visionary ScenarioLiving with Trinetra
7
Visionary Scenario 1 of 2

• Product Information (Shopping)
• Milk expiration date alert, medicine side effects
  check
• Object Identification (Shopping)
• Bottles in fridge, cereal boxes in pantry
• Navigation (Transportation)
• Knows how long before next shuttle will arrive
• Able to notify driver that hell be waiting
• Awareness of Location on Shuttle Ride
  (Transportation)
• Clients device privately calls out stops in
  earpiece
• Speaker on Bus Announces the stops

8
Visionary Scenario 2 of 2

• Sign Reading (OCR)
• Goes someplace new, can compare menu options to
  the posted special. Orders the special.
• Identifies bus stop, and which buses are on this
  route
• Product Information (Shopping)
• Identifies prices, safety warnings

9
Functional Capabilities

• Object Information
• Product prices, serving size, expiration date,
  cooking instructions, medication side effects
• Object Identification
• Cereal boxes on shelf, bottles in fridge
• Querying shuttle status
• Knows how long before next will arrive
• Able to notify driver that user will be waiting
• Awareness of location during shuttle ride
• Shuttle Speaker calls out stops
• Awareness of menu options
• In a restaurant, user can compare menu options
  to the whiteboard special
• Identifying bus signs and information
• Which buses stop at this stop?
• Shopping
• Navigates grocery store, hardware store
• Shelf Navigation

Shopping
Transportation
OCR
Retail-navigation
10
Product Identification (Home)

• Goal
• Finding the carton of milk in fridge
• Interaction Physical browsing
• - Reach into fridge and touch
• - Audio from system names the product

11
Product Information (Store)

• Goal
• Find price and additional
• information like servings, calories,
• additional ingredients
• Interaction
• - user touches or holds product in hand
• - user presses button corresponding to a value
• - user hears the price, number of servings

12
Product Information (Home)

• Goal
• Find preparation/use instructions,
• expiration date
• Interaction
• - Points camera at package label
• - Voice reads out the label via earpiece
• - In future, information is available on the
  phone

13
Reading Signs

• Goal
• User aware of sign information
• Interaction
• - If sign placement known, user points camera
  and voice reads text into earpiece
• - Else, camera used to pan environment
• - User receives an audio cue if sign found -
  User presses a button on phone to hear text

14
Catching/Riding the Shuttle

• Goal
• Determine shuttle arrival
• Interaction
• - Launches Shuttle Helper app on phone
• - Prompts, Which shuttle stop?
• - Request Notification to the driver
• - Audio indication of ETA until next shuttle and
• shuttle interval
• - Notification when shuttle is near

15
User Artifacts

• Cell Phone (camera, text-to-speech)
• Earpiece
• Barcode Pencil
• RFID Reader IDBlue

16

• Interaction Architecture

17
Integrated Interaction Architecture
18
Interaction Architecture-2

• The Interface shall include setting hot keys.
• It shall be configurable as per the user
  requirement.

19

• Overall Architecture

20
Trinetra Proxy (CerfCube 255)
Local Database
Bluetooth
Ethernet
Inventory
Optical Character Recognition
GPRS/EDGE
USB
Product Info from OCR or barcode Sync with PC
HTTP Server
Bluetooth/IP Gateway (Parani100)

• Text/Audio Message
• Voice Recognition (VoiceSignal)
• Text-to-Speech Engine (TALKS)
• Text Recognition (OCR), built-in camera

Cell Phone Functions
Nokia Cell Phone
Audio Message
Shopping
Earpiece
Transportation
Inventory (Shopping) Notification (Transportation)
ETA, Location, Notification (Transportation)
Capacity IR Sensor
Product Name, Expiration Date, etc
Localization GPS 802.11
Barcode Reader (Baracoda Pencil)
Capacity, Location
Notification
IPAQ (in bus)
RFID Reader (Baracoda IDBlue)
Bus Shuttle
HTTP Server, Global Database (Shopping) Applicatio
n Server (Transportation)
21
Trinetra - Shopping

• Paul Cheong
• Prateek Goenka
• Jun Han
• Rohan Kumar
• Patrick Lanigan
• Jim Lin

22
Trinetra Proxy (CerfCube 255)
Local Database
Bluetooth
Ethernet
Inventory
Optical Character Recognition
GPRS/EDGE
USB
Product Info from OCR or barcode Sync with PC
HTTP Server
Bluetooth/IP Gateway (Parani100)

• Text/Audio Message
• Voice Recognition (VoiceSignal)
• Text-to-Speech Engine (TALKS)
• Text Recognition (OCR), built-in camera

Cell Phone Functions
Nokia Cell Phone
Audio Message
Shopping
Earpiece
Transportation
Inventory (Shopping) Notification (Transportation)
ETA, Location, Notification (Transportation)
Capacity IR Sensor
Product Name, Expiration Date, etc
Localization GPS 802.11
Barcode Reader (Baracoda Pencil)
Capacity, Location
Notification
IPAQ (in bus)
RFID Reader (Baracoda IDBlue)
Bus Shuttle
HTTP Server, Global Database (Shopping) Applicatio
n Server (Transportation)
23
Overview

• User interface
• Product information retrieval
• Local cache
• Local network
• Global network
• Input handlers and device drivers
• Bluetooth Device Discovery

24
User Interface
25
UI Design -1

• If product information cannot be found in the
  cache, local database, or UPC database, then we
  allow the user to enter information to pair with
  the barcode or RFID tag unique ID.
• The first step of product information retrieval
  is for the user to select if he wants to scan a
  barcode or RFID tag

26
UI Design -2

• If product information is found,
• The user is prompted to add any additional
  information (Update Product Info).
• The user can either answer Yes and input
  Notes in addition to the already retrieved
  product name
• Or answer No and return back to the
  barcode/RFID selection menu.
• If product information is not found,
• The user is prompted to enter product
  information.
• The user can either answer Yes and enter
  product name and any additional information
  (Notes)
• Or answer No and return back to the
  barcode/RFID selection menu.

27
User Interaction
28
Information Retrieval
29
Local Cache
30
Cache Interface and Details

• Cache Interface
• Add Entry
• allows user to manually save product information
• can be used to save info when local/global
  network is not available
• Update Entry
• used to prevent redundant entries
• can be used by user to store additional
  information
• Retrieve Info
• used to retrieve previously saved product
  information

• Caching Details
• What to cache?
• Only cache global / static product information
  (ID, name, description)
• No localized / dynamic information (price, stock
  status, etc)
• Minimizes problems due to stale cache data
• How to cache it?
• Single binary file contains serialized cache
  objects
• Index stored in memory allows faster searching
• LRU cache replacement policy

31
Cache Entry and Cache Index

• A cache entry will have
• Unique ID
• 16 bytes
• unique produce idenfitier (barcode, tag ID)
• Product name
• 64 characters maximum
• Product description (notes)
• 512 characters maximum
• extended information (directions, warnings, etc)

• A cache index will
• Store metadata about cached items
• Allow efficient direct access to byte streams

TrinetraCacheIndexEntry
TStreamID StreamIDTBuf8lt16gt ItemIDTtime Created
Ttime ModifiedTtime Accessed
TrinetraCacheItem
TBuf8lt16gt ItemIDTBuflt64gt ItemNameTBuflt512gt ItemD
esc
32
Local Network
33
Components

• Bluetooth-to-Ethernet gateway (Parani100)
• Enables communication between Ethernet and
  Bluetooth-enabled devices
• Proxy server (Intrinsyc CerfCube)
• Responsible for filtering requests
• Queries database and returns replies
• MySQL database
• XML-RPC for communication

34
Parani100 Bluetooth-to-Ethernet Gateway

• Multi-port Bluetooth to IP Gateway
• Connects Bluetooth devices to LAN
• Testing
• Nokia 6620 detection and pairing successful
  (tested by configuring gateway to allow
  connections from phone and getting LAN access
  with phone via Parani100)

35
Proxy Server

• Programmed on Intrinsyc CerfCube
• Provides access to in-store local database
• Database Adapter
• hides DB specific details
• queries database for data
• Request Handler
• receives and filters query requests
• passes results back to mobile client
• Testing
• Intrinsyc booted and kept running for several
  days without failure
• Interfacing with CerfCube via terminal successful.

36
Global Network
37
Global Database

• www.upcdatabase.com/rpc
• Online UPC database provides access via XML-RPC
  communication protocol
• Same protocol as local network simplifies
  implementation on the phone
• Phone will use GPRS for Internet access and
  retrieve product information from upcdatabase.com
  via XML-RPC

38
System Integration
39
Input Handler

• Handles device service discovery
• Manages connections to multiple input devices
  (IDBlue, BaracodaPencil)
• Passes output from devices (barcodes, tag IDs) as
  input to request broker

40
Device Drivers

• Communicates with device over Bluetooth socket
• Implements device specific communication
  protocols
• Sets devices to correct operating modes
• Listens for and parses incoming data packets
• IDBlue Driver (shown right)

41
Bluetooth Device Discovery

• Existing code on Nokia Forum
• Need to integrate code
• Currently working target bluetooth device
  address hard coded into the Nokia phone.
• Problem Will not work with any other devices
  (same products will not work)
• Solution Device Discovery

42
Bluetooth Device Discovery

• Device Discovery Steps
• Phone searches for any acknowledgements from
  bluetooth devices within its proximity
• Device gets the initial packet from the phone and
  acks back to let the phone know its existence
• Phone opens up ports and starts communication
• Rewrite UI for the phone
• Easier access for the users especially for the
  visually impaired.

43
Trinetra Shopping(end)

• Paul Cheong
• Prateek Goenka
• Jun Han
• Rohan Kumar
• Patrick Lanigan
• Jim Lin

44
Trinetra OCR

• Tudor Dumitras
• Kunal Kapur
• Pablo Quinones
• Matt Lee

45
Trinetra Proxy (CerfCube 255)
Local Database
Bluetooth
Ethernet
Inventory
Optical Character Recognition
GPRS/EDGE
USB
Product Info from OCR or barcode Sync with PC
HTTP Server
Bluetooth/IP Gateway (Parani100)

• Text/Audio Message
• Voice Recognition (VoiceSignal)
• Text-to-Speech Engine (TALKS)
• Text Recognition (OCR), built-in camera

Cell Phone Functions
Nokia Cell Phone
Audio Message
Shopping
Earpiece
Transportation
Inventory (Shopping) Notification (Transportation)
ETA, Location, Notification (Transportation)
Capacity IR Sensor
Product Name, Expiration Date, etc
Localization GPS 802.11
Barcode Reader (Baracoda Pencil)
Capacity, Location
Notification
IPAQ (in bus)
RFID Reader (Baracoda IDBlue)
Bus Shuttle
HTTP Server, Global Database (Shopping) Applicatio
n Server (Transportation)
46
Overview - OCR

• Goal
• User interaction
• Recognition results
• Software/hardware architecture

47
Goal

• Reading text from signs labels

48
Approach

• Camera OCR software
• (OCR Optical Character Recognition)

49
User Interaction
USER PHONE
Selects OCR mode
Camera turns on
User points camera at sign
User presses button
Phone makes shutter noise
Camera turns off
Phone says processing image
ltsome delaygt
First line of text is read out
User presses next line button
Text from second line is read out to user
User presses repeat line button
Text from second line is read out (again) to user
User presses next line button
Phone makes end of text sound
User presses next line or stop button
Camera turns on
50
OCR Software

• Software packages considered
• - G-OCR
• - SimpleOCR
• - Textbridge Pro 8.0
• Final solution
• - ABBYY FineReader Engine 8.0
• - Informally regarded as the best

51
Recognition Results (1 of 3)
Trinetra Assist i\ e 11 r.uiii k E. Lanigan,
Aari M I

• n..!
• NtuaMltl
• i grocery needs assistance from the
• gt find items
• ler independence

52
Recognition Results (2 of 3)
Carnegie IV) ltlt- Planetary Robotics Building ltlt-
Wean Hall 4- Smith Hall 4/ Hamburg Hall 4gt
Newell-Simon Hall 4lt Facilities Management
Services Bu -gtgt 4902 Forbes
(nothing)
53
Recognition Results (3 of 3)
(nothing)
Half Half
54
High-level Architecture
remote server
phone
55
Architecture
56
Trinetra OCR(end)

• Tudor Dumitras
• Kunal Kapur
• Pablo Quinones
• Matt Lee

57
Trinetra Transportation

• Aaron M Paulos
• Joe Tadduni
• Jovan Williams
• Andrew Williams
• Chris DeLeon
• Puneet Aggarwal

58
Trinetra Proxy (CerfCube 255)
Local Database
Bluetooth
Ethernet
Inventory
Optical Character Recognition
GPRS/EDGE
USB
Product Info from OCR or barcode Sync with PC
HTTP Server
Bluetooth/IP Gateway (Parani100)

• Text/Audio Message
• Voice Recognition (VoiceSignal)
• Text-to-Speech Engine (TALKS)
• Text Recognition (OCR), built-in camera

Cell Phone Functions
Nokia Cell Phone
Audio Message
Shopping
Earpiece
Transportation
Inventory (Shopping) Notification (Transportation)
ETA, Location, Notification (Transportation)
Capacity IR Sensor
Product Name, Expiration Date, etc
Localization GPS 802.11
Barcode Reader (Baracoda Pencil)
Capacity, Location
Notification
IPAQ (in bus)
RFID Reader (Baracoda IDBlue)
Bus Shuttle
HTTP Server, Global Database (Shopping) Applicatio
n Server (Transportation)
59
System Architecture
60
Software Architecture
61
Shuttle Hardware

• Computing Platform
• Fujitsu Lifebook P-series Laptop
• GPS Coordinate Receiver
• Garmin GPS 18
• Communication
• PCMCIA GPRS Cellular Modem
• Capacity Sensing
• Phidget 8/8/8 USB sensor interface board
• 2 Paired IR Transmitter / Receiver Circuits

62
Capacity Sensing
63
Shuttle Software
64
End-user Hardware
End-user Hardware

• Supports high speed connectivity via GPRS.
• Supports CLASS-4 EDGE, which is the current
  standard for high-speed data transmission on
  GSM-based Networks and phones.
• Supports TALK software.
• Web interface was developed using XHTML for
  Mobile 1.0 .
• WML is being considered in order to increase
  on-phone application functionality like input
  validation

65
User Interaction
66
Demo
67
Server Hardware and Configuration

• Dell Optiplex GX620
• 150 GB
• BroadCom 57xx Gigabit Ethernet
• Tomcat 5.0.28
• MySQL 5.0.19
• Sun JRE

68
Server Software - SQL Tables
PerBusDataBase
int maxCapacity int busID int maxCapacity int currentCapacity float lastXCoord float lastYCoord int currentScheduleID date lastHeartbeat float lastAvgVelocityReceived
PerGPSPoint
int afterStopID boolean busTurn float bearing float distance float xCood float yCood int ptType int routeID string label
PerUserRequest
int requestPhase int fromStopID Int routeID
PerAdmin
string username string hashedPassword int hashSalt
PerPINUser
int pin int phoneNumber
Italics are relatively staticNon-italics are
dynamic
69
Assumptions

• End-user interaction
• No concurrency problems
• SMS gets to user in timely fashion
• No connection interruptions
• No invalid inputs
• Server
• Server is reliable and scalable
• Shuttle
• Perfect GPRS connection
• Application is fault free

70
Risk Assessment

• What Can go Wrong?
• Lack of Human Resources
• Task Complexity
• Lack of Cooperation from shuttle administrators
• Accuracy of Shuttle Positioning and Capacity
• Contingency Plan
• Prioritize or trim functionality

71
Trinetra Transportation(end)

• Aaron M Paulos
• Joe Tadduni
• Jovan Williams
• Andrew Williams
• Chris DeLeon
• Puneet Aggarwal

72
Trinetra Proxy (CerfCube 255)
Local Database
Bluetooth
Ethernet
Inventory
Optical Character Recognition
GPRS/EDGE
USB
Product Info from OCR or barcode Sync with PC
HTTP Server
Bluetooth/IP Gateway (Parani100)

• Text/Audio Message
• Voice Recognition (VoiceSignal)
• Text-to-Speech Engine (TALKS)
• Text Recognition (OCR), built-in camera

Cell Phone Functions
Nokia Cell Phone
Audio Message
Shopping
Earpiece
Transportation
Inventory (Shopping) Notification (Transportation)
ETA, Location, Notification (Transportation)
Capacity IR Sensor
Product Name, Expiration Date, etc
Localization GPS 802.11
Barcode Reader (Baracoda Pencil)
Capacity, Location
Notification
IPAQ (in bus)
RFID Reader (Baracoda IDBlue)
Bus Shuttle
HTTP Server, Global Database (Shopping) Applicatio
n Server (Transportation)
73

• 5. Implementation Guidelines

74
Implementation Guidelines-1

• ODC Orthogonal Defect Classification
• Provides a valuable diagnostics method
• Evaluating the various phases of the product life
  cycle (design, development, test and service)
• Estimates the maturity of the product.

75
Implementation Guidelines-2

• Aim for effortless independence
• Leverage existing technologies
• Complement basic tools, dont complicate
• Reliability for real-life usability
• Unobtrusive, low-profile use
• Make system configurable
• Consistency is key to usability
• These are our final guides leading into
• Phase 3 Prototype Implementation

76
Questions
77
Functional Capabilities

• Object Information
• Product prices, serving size, expiration date,
  cooking instructions, medication side effects
• Object Identification
• Cereal boxes on shelf, bottles in fridge
• Querying shuttle status
• Knows how long before next will arrive
• Able to notify driver that user will be waiting
• Awareness of location during shuttle ride
• Clients device privately calls out stops
• Awareness of menu options
• In a restaurant, user can compare menu options
  to the whiteboard special
• Identifying bus signs and information
• Which buses stop at this stop?

Shopping
Transportation
Optical Character Recognition
78
Description Slides
79
Server Software - ETA

• The ETA function acts as an intermediary between
  the User Scheduler and the Shuttle Input Handler
• The ETA function gets data from the Shuttle Input
  Handler through the
• GetData(BusID,) function.
• The call returns data from the selected shuttle
  including location, Capacity, velocity, and
  direction of travel.
• The ETA function has 4 public function calls.
• GPSCood GetBusLocation(int BusID)
• int ETA( int stop)
• int GetCapacity(int BusID)
• int GetCapacity(int Stop)

80
Server Software BL / APIs
81
OCR Interaction Architecture
user presses star-star
Processing Image Press to cancel
user takes picture
shutter sound camera turns off
user presses star-star
user presses 1 or 3
user presses 1
user presses 1 or 2
ltFIRST line of text from OCRgt Press 1 to repeat
line Press 2 for next line Press 3 for previous
line Press star-star to end
ltSECOND line of text from OCRgt Press 1 to
repeat line Press 2 for next line Press 3 for
previous line Press star-star to end
ltFINAL line of text from OCRgt Press 1 to repeat
line Press 2 for next line Press 3 for previous
line Press star-star to end
user presses 2
user presses 2
user presses 3
user presses 3
82
OCR Implementation Details

• Client-side
• programmatic access to camera
• sending a http POST request to server with image
• handing response
• allowing user to read OCR text line-by-line
• Server-side
• listen for http POST request (JSP/Tomcat)
• grab image from request (JSP/Tomcat)
• invoke OCR software (a C program)
• get output (unicode text) from OCR
• respond with text