Microbiology Tray and Pipette Tracking as a Proactive Tangible User Interface

1
Microbiology Tray and Pipette Tracking as a
Proactive Tangible User Interface

• Harlan Hile
• Jiwon Kim
• Gaetano Borriello

2
Outline

• Microbiology Lab Environment
• Related Work
• System Design
• Setup
• Tracking
• UI
• Evaluation
• Conclusion

3
Introduction Lab environment

• It is desirable to have information presented
  throughout the lab and in context
• Specialized work areas for different tasks
• Lab is a structured environment
• Tasks are planned before they are performed

4
Introduction - Microbiology

• Microbiology does sets of experiments in a tray
  containing many wells
• Pipettes are used to transfer material
• Extra care is necessary to document and avoid
  mistakes

5
Problem Statement

• Use biology instruments (well trays and pipettes)
  as i/o devices for database
• Query for information in the context of the
  experiment
• Make new entries when actions are performed to
  seamlessly document
• Do tracking using a video system
• Display information onto benchtop with projector

6
Related Work

• Hand Tracking
• Augmented surfaces, but not tangible hand
  replaces mouse
• Tangible UI
• Give physical form to virtual objects
• Augmented Reality
• Enhance existing tasks
• In a variety of domains, none know users goals

7
Design Overview

• Overhead camera locates specially marked tray and
  pipette
• Projector displays information onto bench

8
Vision Algorithm Steps

• Color segmentation to find separate out markers
• Group pixels into larger chunks and fit lines to
  them
• Lines forming rectangles are candidates for
  trays, also need to match color patterns
• Lines with appropriately spaced markers indicate
  pipette tip location

9
Vision Algorithm Video
10
Interface Display

• A precomputed homography between the camera and
  display allows information to be displayed
  directly on the detected objects
• Shows when interactions are recognized

11
Video Example
12
Speed and Accuracy

• Runs at about 4 frames per second
• In java on 2GHz P4
• 11 frames per second without UI display
• With unobstructed views of tray and pipette, tray
  found 99 of the time, pipette found 95 of the
  time
• Errors seem to be from color variations, unseen
  markers
• False positives are infrequent, false negatives
  can be smoothed over
• But may seem less responsive

13
User Tests

• We conducted tests with our target users and
  collaborators, workers in the CSI lab
• Their style caused occlusions with the pipette,
  confusion with neighboring wells
• Display feedback of recognized actions allowed
  them to quickly adapt and get acceptable
  recognition rates
• Without feedback, one user had only 2 of 16
  recognized correctly, and with it had 15 of 16

14
User Feedback

• Even at 4fps, they felt the system was useful
• Would help them keep track of operations and
  recover from interruptions

• Said a display to show status of all the wells at
  once would be useful
• Liked the idea of proactive prompting for next
  step or to flag errors
• When can we have it?

15
Future Work

• Robustness
• Better accuracy, explore using multiple cameras
• Speed
• Lots of room for optimization
• User Studies
• With more users, outside our group
• Integration
• With TeraLabs experiment plans
• Include information like tray and reagent
  identity with RFID, instrumented pipette for
  dispensation events

16
Conclusions

• We have built the beginnings of a useful system
  for the microbiology lab
• Despite limitations of our system, feedback
  display allowed users to quickly adapt
• Room for a variety of improvements
• To be published in Pervasive 2004

17
Questions?