Pervasive Pixels (Columbia University Dept. of Computer Science)

1
Pervasive Pixels(Columbia UniversityDept. of
Computer Science)

• Henning Schulzrinne (PI)
• Steven K. Feiner
• Gail Kaiser
• John Kender
• Kathleen McKeown

2
Proposed Research

• Goal seamless mobile multimedia collaboration
  across distance
• Integrate advances across fields
• Collaborative work
• Graphical and visual interfaces
• Spoken language understanding and generation
• Vision sensing and understanding
• Networking and security

3
Contributions

• Contextual information management
• use workflow to determine display content
• multimedia summaries of past and present sessions
• Harmonizing physical and virtual environments
• map changing virtual information onto physical
  displays
• map layout of physical environment onto virtual
  space
• Network services
• clear, flexible interface to common services
• authentication and privacy support
• infrastructure for persistent large displays

4
Features of Research Infrastructure

• Large numbers of instrumented multi-display
  workspaces
• Networked mobile devices of various capabilities
• Transparent and automatic adaptability to changes
  of place, platform or group
• Support for a wide range of hardware and
  software, from commercial to novel

5
Proposed Research Infrastructure

• Outfit informal areas for collaboration
• Public areas for walk-by interaction
• Multiple touch displays, cameras, audio
• Portable units
• Stationary setups
• Multiple displays, video cameras, audio
• Seminar room, meeting rooms
• 12 faculty offices
• User-based personalization user location
• Triangulation on mobile devices
• Visual tracking
• Standard methods (e.g., active badge)

6
Public areas walk by stations

• Multiple touch displays, video projectors and
  cameras, embedded computers, speakers and
  microphones

7
Design for Walk-by Collaboration Station
ceiling
network
PC
loudspeaker
camera
array
proj.
microphone
electronic
whiteboard
card reader
IR/RF badge
8
Public Areas informal gatherings
9
Meeting Room

• Remote-controlled pan-tilt video cameras and
  projectors, Omnicam, conference table
  microphones, automatic audio mixer, ceiling
  speakers

10
Faculty Office

• Mimio electronic whiteboard, XGA video projector,
  Ethernet speaker phone, wall-mounted pan-tilt
  video camera, PocketPCs

11
Seminar room

• Omnicam omnidrectional audience camera,
  high-resolution DV video camera, 2 pan-tilt
  speaker cameras, ceiling mounted microphones,
  electronic whiteboard, XGA high-brightness video
  projectors

12
Functionalities

• Conferencing
• Internet conferencing server to mix IP and PSTN
  audio streams
• Interconnection with analog phone
• Digital hybrid connects digital or analog sound
  to existing telephone system in classroom
• Network voice-over-IP interface attached to
  Nortel Meridian PBX for 20 simultaneous
  conversations
• Multi-processor servers and IA64 compute and
  database server
• File storage
• Face, speaker and fingerprint recognition
• Backup facilities 2 printers and tape library
  system

13
Initial results

• HCI gesture-based user interface for public
  kiosk
• mouse replacement for pointing and selecting
• uses frontal and side camera
• Security disCFS and WebDAVA secure file systems
• disCFS NFS with credentials instead of
  authorization
• WebDAVA grant restricted access to resources
  using HTTP and Java applets
• Web-based collaboration
• content on all kinds of devices
• pass DOM through a series of filters and
  transformations ? HTML
• Ubiquitous multimedia communications
  infrastructure
• being commercialized I2 demonstration
• input into standardization (IETF)

14
Ubiquitous Computing

• Traditionally, focus on closed environments
• proprietary protocols
• single (trusted) user class
• single site (room, lab, home, )
• stand-alone components (video conferencing)
• PP focuses on whole system and user experience
• Pervasive Pixels networking component
• standard protocols
• SIP for media configuration, event notification,
  instant multimedia messaging
• SLP for service discovery
• integration of presence and user context
• standardization in the IETF (RPID)
• location-based services
• user context
• user authorization
• service location

15
Mobility in Pervasive Pixels

• Terminal mobility
• application-layer mobility complements L3
  mobility
• Session mobility
• move active sessions to devices found in the
  environment
• ? service discovery
• Service mobility
• move configuration to new devices
• Personal mobility
• one user, many devices

16
Location-based services

• Traditionally, focus on geospatial location
  (e.g., GPS)
• But other aspects as important
• civil location (often more intuitive)
• type of place (home vs. office outdoors vs.
  theatre)
• behavioral distraction, privacy, appropriateness
• Experimenting with low-complexity location
  mechanisms
• IR/RF active badges with low installation cost
  (Ivistar)
• BlueTooth location beacons
• LAN backtracking and DHCP
• swipe cards and i-buttons

17
Some initial lessons learned

• Usage remote presence from UKy during sabbatical
• research group meetings
• departmental site visit
• thesis proposals and defenses
• Perception Multimedia collaboration is a mature
  field
• Reality It doesnt work much better than in 1992
• still fails in hard-to-diagnose ways
• quality better, but echo, feedback and level
  issues remain
• Integration between synchronous and asynchronous
  collaboration
• integrating documents, minutes,
• Transition from call-focused to presence-focused
• much larger use of asynchronous collaboration
  (email, bulletin boards, )
• Working with start-up company new IP-based
  departmental communication system to replace PBX

18
Columbia SIP servers (CINEMA)
Telephone switch
Local/long distance 1-212-5551212
rtspd media server
Quicktime
Single machine
RTSP
sipconf Conference server
RTSP clients
Department PBX
sipum Unified messaging
Internal Telephone Extn 7040
713x
sipd Proxy, redirect, registrar server
SQL database
SIP/PSTN Gateway
Web based configuration
SNMP (Network Management)
Extn 7134
H.323
Extn 7136
siph323 SIP-H.323 translator
NetMeeting
xiaotaow_at_cs
19
Larger lessons for multimedia systems research

• Software tool support for multimedia
  communications lacking
• most are applications, not building blocks
• cross-platform research media tools are getting
  very old and creaky (vic, rat, etc.)
• multi-party support very weak (multicast never
  happened)
• Components designed to be operated by humans
• IP phones only have HTTP/HTML interface
• video projectors just proprietary configuration
• Lots of components, but hard to evaluate in real
  use
• still mostly barely demo quality audio delay,
  echo, random failures
• people will fall back to good ol PSTN quickly

20
Conclusion

• Pervasive Pixels attempt to integrate multiple
  modalities into system, not just grouping of
  components
• Evaluation in real usage, not just demos
• Spread throughout the department, not just lab