Your Life as a Movie

1
Your Life as a Movie

• Sailesh Chutani
• Director, Worldwide University Relations
• Microsoft Research

2
What if..

• Everything you ever say, hear, observe, do, and
  sense is recorded
• Those recordings are stored permanently and can
  be accessed easily
• You could analyze or modify copies of these
  recordings and share them

3
Some Questions

• Is this scenario technically feasible?
• What would such a future look like?
• What new opportunities and risks will we be
  confronted with?
• Elements of such a future are already here but
  just not uniformly distributed (to paraphrase
  William Gibson)

4
Microsoft Research

• Founded in 1991
• Staff of over 700 in over 55 areas
• Internationally recognized research teams
• 200 WW university relationships
• Research lab locations
• Redmond, Washington
• San Francisco, California
• Cambridge, United Kingdom
• Beijing, Peoples Republic of China
• Mountain View, California

5
A University CS Department inside Microsoft

• University organizational model
• Flat structure, critical mass groups
• Open research environment
• Aggressive publication of results in
  peer-reviewed literature
• Frequent visitors, daily seminars
• Strong ties to University Research
• Nearly 15 of basic research budget directly
  invested in Universities
• Lab grants, research grants, fellowships, etc.
• Hundreds of interns and visitors

6
MSR Mission

• Expand the state of the art in each of the areas
  in which we do research
• Rapidly transfer innovative technologies into
  Microsoft products
• Ensure that Microsoft products have a future

7
Expanding the State of the Art

• Thousands of peer-reviewed publications
• 13 of papers at 2001 CHI (UI)
• 20 of papers at 1996 SIGGRAPH (graphics)
• 25 of papers at 1999 PLDI (prog languages)
• 30 of papers at 2001 PLDI
• More papers at SIGGRAPH 2003, SIGMOD 2004 , SIGIR
  2004 and other key conferences than any other
  organization
• Community leadership
• Professional societies
• Journals
• Conferences

8
Segmenting the Problem

• Information Capture
• Storage
• Query and Retrieval
• Analysis and Processing

9
SenseCamA Digital Memory for Everyone

• Lyndsay Williams Ken Wood _at_ MSRC and the
  MyLifeBits team _at_ BARC

10
SenseCam

• Essentially a Black Box data and image recorder
  for the human body
• Current wearable prototype can be worn for a day
  and captures up to 2000 images in that time

• Image capture triggered by sensors, e.g. motion,
  light, temperature, people in field of view, ...
• Sensor data is also recorded for later
  presentation, analysis, and correlation

11
Ultra Wide-Angle Lens

• Captures everything regardless of exact camera
  orientation

12
Image Stabilisation

• Accelerometer measures motion of the device
• When any sensor indicates image capture, we wait
  a few tens of milliseconds to capture image when
  movement is less than 10 degrees/second if
  possible
• Greatly reduces the number of blurred images

Before
After
13
A Short Walk Through Cambridge
14
How does SenseCam work?

• Accelerometers for motion detection
• Passive Infrared (PIR) sensors triggered by
  people or other living beings in front of camera
• Digital sensors for light level and temperature
• Images captured automatically on sensor events
  caused by, e.g.
• turning around, sitting down, standing up,
  running, falling over, ...
• moving between locations (rooms, car, bus,
  indoor/outdoor, etc.)
• meeting a friend in the street
• Simple hand gesture allows intentional image
  capture
• Accelerometers also allow image stabilisation

15
SenseCam Applications

• Personal memory enhancement
• Where did I leave my umbrella?
• What was the name of that wine we had last week?
• What was written on the whiteboard?
• Tourism
• Automatic photo journals or blogs of specific
  trips, e.g. to the Edinburgh Festival
• Clinical support for the memory-impaired
• We are working with a Cambridge neuropsychologist
  on a trial for Alzheimers patients

16
(No Transcript)
17
Future

• Now building next-gen SenseCams
• smaller, better power management, audio trigger,
  wireless comms

18
Storage Needs

• Number of seconds in 80 years 606024365802,5
  22,880,000
• Data recording rate for MPEG-4 is 4Mb/s
• Storage required for full motion video of 80
  years of someones life is 10k tera bits or 10
  peta bits (1015)
• Cost of such a storage in todays terms is
  approximately 10m
• Storage capacity is doubling every year, in 10
  years 10 peta bits may cost lt10k

19
Storage Trends
Source Ed Grochowski, IBM Research Almaden
20
Moores Law
A threshold event will take place early in the
21st century the emergence of machines more
intelligent than their creators. By 2019, a
1,000 computer will match the processing power
of the human brain - about 20 million billion
calculations per second. Organizing these
resources - the software of intelligence - will
take us to 2029, by which time your average
personal computer will be equivalent to 1,000
human brains. -- Ray Kurzweil
21
Human Scale Storage

• Terabyte
• 1600 today
• 400 in 2-3 years
• Terabyte can hold
• A lifetime of all your conversations
• A year of what you can see

22
Querying

• When did I meet the Dalai Lama?
• How long did I live in Lausanne?
• How many people did I encounter in my life with
  whom I spend more than 5 minutes in conversation?
• Where was I on my wifes birthday during the last
  10 years?
• What were the scariest moments in my
  mountaineering trips?

23
Stuff Ive Seen (SIS)

• Support information re-use

• Today
• Many locations
• Many interfaces

• With SIS
• Unified index
• Fast, flexible search
• Immediate update
• Rich UI possibilities for your content

24
SIS Usage Observations

• Very short queries (1.6 words), quick
  filtering/sorting
• Few searches for best matching item
• Know and use other attributes
• Age of items opened
• 50 more than one month old, some up to 8 years
• Importance of people
• 25 of queries involve people
• Importance of time
• Date by far the most common sort attribute
• Useful date varies by source

25
Contextualizing Search

• Search is not the end goal
• Goal is information management in the context of
  ongoing work activities

• Search always available
• Search from within apps
• Show results in context of apps

26
MemoryLens

• Harness models of human memory
• to assist with navigating large
• stores of personal content

27

• MemoryLens Search UI

28
Modeling the interface after the way people think
feel

• Cognition
• Memory
• associations vs. facts/details
• Deep History
• objects recede vs. disappear
• Dynamic organization
• multiple simultaneous categories vs. fixed
• Personality Emotion
• Know what people care about and use to help
  organize
• Watch users habits behavior to customize
  experience
• Context
• Many people, places and events

29

• MemoryLens LifeBrowser

Adjustable threshold for
memorability
30
Walkthroughs

• Re-live the past, explore the events as they
  happened at your own pace
• Analogous to action replay but at a large scale
• DEMO

31
WWMX (wwmx.org) Geography for Photography

• The World-Wide Media eXchange is
• a centralized index of
• the worlds image media
• tagged by location (and other standard
  metadata).

32
Why Location for Photos?

• Provides immediate context, hints at semantics
• E.g., photos taken at Disneyland, video shot in
  Hawaii
• Repeatedly appears in user studies regarding
  organization
• Family trips strongly associated with location
• Event time location
• Subject as person doing something somewhere
• Universal index (like date/time)
• Independent of ontology/taxonomy, language,
  culture, etc.
• Familiar visual browsing UI via maps
• Scales well
• Technology to acquire location becoming common
• GPS, e911 services, radio-tower-based,
  802.11-based

33
(No Transcript)
34
(No Transcript)
35
(No Transcript)
36
(No Transcript)
37
(No Transcript)
38
(No Transcript)
39
(No Transcript)
40
(No Transcript)
41
Applications

• Real estate tours
• Travel and tourism
• Shared travelogues
• Driving directions with photos of landmarks
• Centralized media depository for community events
• Etc.

42
Creating Art

• Recordings can serve as the raw material for
  creating art
• DEMO

43
Meta Questions

• Who would regulate the recording of this
  information?
• Who will own this information?
• Who would be able to access it?
• Should copies and modifications be allowed?
• What would be the legitimate uses of this
  information?

44
Meta Questions

• What would it mean for us as human beings?
• Will human behavior change in light of the
  permanent record?
• How will we substitute for forgetting if memory
  becomes perfect?
• Would we become more empathetic if we can
  literally see other points of view?
• Would resolving conflicts be just a matter of
  comparing recordings?
• Would we create a class of people who live
  vicariously?
• Would life be safer?
• Is video/experience editing the next big growth
  industry?

45
Clues for Answers

• Current usage in forms of
• video surveillance,
• credit histories, public records
• policy on picture phones,
• popularity of reality TV
• use of web search to get background
• Exploration in Art, movies that have touched the
  themes
• Being John Malkovich, After Life, Slacker

46
Conclusions

• Scientists and engineers create the intellectual
  and physical building blocks Artists imagine the
  possibilities, but
• It is the interplay of Society and Culture that
  shape the actual usage and norms