MyLifeBits: Realizing the Memex Vision Santa Clara University 13 May 2004

1
MyLifeBits Realizing the Memex VisionSanta
Clara University13 May 2004

• Gordon Bell,
• Jim Gemmell Roger Lueder
• www.MyLifeBits.com
• www.research.microsoft.com/gbell

2
Mylifebits collage
3
Outline MyLifeBits

• Backgroundfulfilling the Memex vision
• Cyberizing everything
• File to database transition
• Usebeyond search
• Working with Media Center for home use
• Long-term agenda and outlook
• Archiving persons and things.

4
MemexAs We May Think, Vannevar Bush, 1945

• A memex is a device in which an individual
  stores all his books, records, and
  communications, and which is mechanized so that
  it may be consulted with exceeding speed and
  flexibility
• Full-text search, text audio annotations, and
  hyperlinks

5
Capturing what you see
6
I am data
7
The guinea pig

• Gordon Bell is digitizing his life
• Has now scanned virtually all
• Books written (and read when possible)
• Personal documents (correspondence including
  memos and email, bills, legal documents, papers
  written, )
• Photos
• Posters, paintings, photo of things (artifacts,
  medals, plaques)
• Home movies and videos
• CD collection
• And, of course, all PC files
• Now recording phone, radio, TV (movies), web
  pages conversations and meetings to come
• Paperless throughout 2002. 12 scanned, 12
  discarded.
• Only 30 GB!!!

8
Capture and encoding
9
Quindi conference capture
10
I mean everything
11
Wearable interactive jewellery LEDs flash
according to sensor type triggered
12
Potentially useful trivia but not normally
photographed
13
GPS tells where and when

14
Kentaro Toyama wwmx.org
15
gbell wag 67 yr, 25Kday life
16
MyLifeBits organization time and space
Archival (time) Working
Timeline/ Context(space)
Personal (some s) GB Co.(angel, etc.)
Professional ACM, etc., _at_Microsoft.com, New
cos.
17
MyLifeBits Some Lives(t)

• CGB_at_ Microsoft
• MLB
• Clusters
• Telepresence
• WWW presence
• Computer History Museum
• BOD member
• Fund-raising
• CyberMuseum
• Startups boards
• Bell-Mason Director
• Diamond Vanguard Brds.

• Personal
• Parents, children, grandkids
• CGB himself
• GKB
• Close friends
• GB s
• Personal incl. several legal structures
• Properties autos, real estate,
• Investments contracts
• Past prof. companies/organizns
• DEC
• Carnegie-Mellon U.
• DEC, NSF, Encore, Ardent, Me Inc.,

18
Bell Lives timeline
1900 1910 1920 1930 1940 1950 1960 1970 1980 1990
2000 2010 C,L m
d d CGB... GB SR mB,L
KF SB Where KvMO B
ABosP B WCa 6-year
--GS-HS---MIT DEC---.--- Education
KV-----mit,F cmu Work Bell
Elec DECcmuDEC E,NSF MSFT ComputerMuseum
M B SiValley Books
BN SBN
HiTechVent Computers 4-6
11 VAX E A
19
Personal LifeLog Applications
Diary/Journal
Self
Tutor
Mentor
Advisor
Meeting Prep
Personal Assistant
Babysitter
Financial Manager
Companion Caretaker
Medical Manager
Parole Officer
Photo Album
Assistant for Elderly
Application used by
Pers Flight Recorder
Autobiography
Captains Log
Conservator
Biography
Baby Book
Trustee
Obituary
Personal Proxy
Executor
Others
Application controlled by
Others
Self
20
MyLifeBits Software
MyLifeBits store
database
21
MyLifeBits is

• Memex and more (audio and video)
• Universal store for all personal stuff
• Guiding principles for the system
• Full text search collections (gt than hierarchy)
• Visualizations for search, display, insight
• Annotations and links add value and essential
• Increase search ability and value of information.
• So make many kinds and them easy to create!
• Stories are the ultimate annotation
• Keep the links when you author transclusion

22
MLB database size and content?

• Database features are essential Consistency,
  Indexing, Pivoting, Queries, Speed/scalability,
  Backup, replication.
• Folders Files were the starting point gtgt
  database into sets aka collections that are
  identical to the folder structure
• Outlook (msgs, attachments, calendar, contacts)
• Web trails including voice message annotation
• Journal (Outlook), trails every document use
  transaction
• What about?
• Money (transactions, payees, etc.)is their
  lifelog/trail
• Streets and trips to cross-index to all docs
• Attributes for photos for retrieval? Location,
  time, settings
• Presentations as a report or trail. Each slide an
  object!

23
Why bother? An existence proof.The following
exist in abundance

• Shoeboxes full of photos
• Photo albums framed photos
• Creative Memories is a thriving business selling
  resources for created high-end photo albums that
  are well laid out and highly annotated, using
  long-lasting materials.
• Home videos
• Bookshelves and filing cabinets
• Old bundles of letters
• Professional video/photo companies do capture at
  kids sports events and sell content like
  hotcakes
• Probably not accessed very often but TREASURED
  (whats the one thing you would save in a fire?)

24
Why bother? ..more reasons

• To eliminate physical storage (paper, CDs)
• It costs more (in time) to delete than the cost
  the storage
• You may only want to retrieve one of many items
  in the future, but cannot predict which one
  (which is why you file many things now)
• For posterity and nostalgia
• For memory enhancement faster search(search
  your LifeBits rather than the web a single
  source to look for anything you have ever seen)
• Let content analysis and data mining discover
  trends and correlations in your life

25

• Extensible XML schemas
• Logical views
• Programmatic relationships
• Synchronization service
• Information agents

26
Annotation like this
Voice Annotation
27
(No Transcript)
28
Pivot to look at all of MLB(t)
Call, contact, pivot by time to find web page
29
Find brig, image, and look for 80
30
Here are the photos
31
Timeline view tells a story
32
Interface to xls
33
Statistics of use
34
Visualization

• Browsing searching. Get me what I wantneed!
• Help the user find things among possible items
  versus
• Waiting for an ideal system that can find what I
  want
• Publication Conventional web, presentations,
  etc.
• Helps understand the nature of the content e.g.
  histogram of objects in time
• Context Links to help understand the
  relationship between objects. Provides more
  search handles.
• Information density what is it? What is its
  relationship to others?
• Content important. Flash and form, less useful.

35
Value of media depends on annotations

• Its just bits until it is annotated

36
Getting the user to tell a story is the ultimate
in media value

• A story is a layout in time and space
• Most valuable content (by selection, and by being
  well annotated)
• Stories must include links to any media they use
  (for future navigation/search transclusion).
• Cf MovieMaker Creative Memories PhotoAlbums

We took him to lunch at our favorite Dim Sum
place to say farewell
Dapeng was an intern at BARC for the summer of
2000
At table L-R Dapeng, Gordon, Tom, Jim, Don,
Vicky, Patrick, Jim
37
Value of media depends on annotations
Its just bits until it is annotated

• Auto-annotate whenever possible e.g. GPS cameras
• Make manual annotation as easy as possible. XP
  photo capture, voice, photos with voice, etc
• Support gang annotation
• Make stories easy

38
Future work Visualizations

• Don't give me a little card image and say,
  "That's all you've got, because that's what I
  thought you should want for your virtual
  shoebox." There have got to be multiple
  modalities and the designers have to be able to
  deal with that. don't metaphor me in, don't
  give me only one way of looking at things.
• -Andy van Dam, Hypertext '87 Keynote Address

U. Maryland
IN-SPIRE
Next Media
39
LifeLines (Plaisant et al.)
www.cs.umd.edu/hcil/lifelines
University of Maryland
40
Rethinking collections files

• Date collections (summer 99)
• Much better as a query
• By Person (Photos of Bill)
• Better as links of type photo of to person
  Bill
• By Event (Trip to UCLA)
• Better as links to event in calendar
• Working set
• Better as query that figures it out for me so I
  dont need to maintain it

41
Facets and people

• Time ( stage of life). Events
• Location (lat/long vs home, vacation)
• Institution (relations including family, work,
  clubs,)
• Role (student, professional, parent, owner, etc.)
• Content type
• Audio, graphics, photo, video aka moving picture
• Document t type o(200) plus profession
  specificad, billwill, cards (calling, credit,
  grade, greeting), certificate (birthdeath),
  correspondence, diary, essay, forms, legal (6),
  instructions, lists, resume, reservation,
  scrapbook, transcript,
• Dissemination
• Book, electronic, serial, unpublished,
• Special collections (e.g. geology, stamps,
  species, places)

42
Facet Lists
43
Certificate facets
44
By region and by time should be facets!
45
Telephone, Television, and Radio in theHome of
the Future
46
Evolution of media in the home
Tomorrow
Today
Yesterday

• Analog storage and transmission on separate
  networks
• Physical space limitations
• Tedious management and manual search

• Digital storage (CDs, DVDs, PVRs, MPEG WMA/V)
• Digital cable, internet radio, but phone is
  mostly analog
• Still limitations on what we can store
• Different stores for different stuff

• All digital
• Everything connected
• Unlimited storage
• Everything in a database

SQL
47
stereo
Cassette
Receiver
Wfr
Spkr
Cables/links Speaker 51 Plasma 2 or 3 Cable/Enet
2 IR 8 Stereo 4 5.1 digital 2 Comp./S-video
3 Plasma panel 1 Power 10 Kbd/mse 2 Monitor II
(opt.) 4 Camera 2 Total 42 46 Things
18remotes
stereo
CD
5 speakers
Spkr
IR
stereo Video
VCR
5.1 digital comp.
DVD
stereo Video
Set top
Set top
Cable/ Satellite
Video
5.1 digital
Plasma Panel
Media Center Computer
Ethernet
SVHS-wide
Camera Mic
Video composite or S-video
Kbd
Mse
48
(No Transcript)
49
MyLifeBits use scenarios

• Acquire everything! (I mean everything!_
• Professional personal use at work!
• Home/personal Provide ambiance entertainment
  using Home Media Center
• Enhancing content through photo and video albums
• Events, places, trips, people, time
  intervals---------- Database land and
  authoring --------
• How I spend my time or an interval of time.
  Recall a trail What was I thinking about?
• Endless need for authoring reporting tools
• ISBQ Interactive Story By Query
• A Person (auto- or -biography web hosted time
  line
• Personal/web/org. hosted collections catalogs

50
The Agenda for the Tbyte(s), Lifetime, PCThe
killer app after office and mail.

• Guarantee that data will live forever! dear
  appy problem
• Cheap, easy, and data-rich (e.g. time, place)
  capture
• GPS and time everywhere
• Paper capture has to be as easy as discarding
  (scanner/shredder)
• Personal meeting capture...
• E-booke-magazines journals need to have
  critical mass!
• Telephony and audio capture with indexing
• Media Center compatible for entertainment
  (photos, video, TV, radio)
• Content analysis (critical for photo video!)
• Information control privacy, security,
  expunge/deniability,
• Having to be schizophrenic or have a lobotomy
  when leaving a life
• One dbase for everything (articles, books,
  conversations, ... financial transactions) vs.
  long-term use of hierarchical files. Is dbase
  intuitive?
• Annotations/meta-information add every-increasing
  valueEasy annotation for aiding search and it
  becomes the content
• The killer apps Alzheimer, immortality,
  surrogate memory?
• GUIs to improve use (e.g. time to learn, use,
  retention)

51
The dear appy problem

• Dear Appy,
• How committed are you? Please come back to me,
  Lost and forgotten data
• Whos responsible?
• media
• platform, file, and databases
• evolving standards and formats
• evolving and/or disappearing apps
•  

52
Problems Amnesia control deleting corporate
life bits

• Full sharing of bits that are mine
• I created them, OK to copy and distribute
• DRM purchased for my own use
• OK to look at, but I only own half the bits
• Controlling forgetfulness
• Private, do not demo
• Expunge forever... this never happened
• The bits belong to a corporation or org.

53
The Content Analysis Problem

• Cliplets Automatic segmentation of a pile of
  documents and video into individual documents and
  scenes.
• Item typing Would like a minimal Dublin Core for
  each item date, creator, title, source,
  abstract, and type
• Type classification articles, letters, memos,
  etc.
• Ontology creation for collections

54
The End
55
Archiving persons and things

• www.oac.cdlib.org for 0(1K) corporations, people,
  places, things.
• List of finders, usually -gt paper boxes!
• E.g. Apple collection at Stanford points to 600
  or say 1K/ft.
• www.AlbertEinstein.org Einsteins papers, etc.
• diva.library.cmu.edu/Newell/ for Allen Newell
• profiles.nlm.nih.gov/ Nobel Prize winners,
  Lederberg
• www.ComputerHistory.org computing artifacts
• www.MyLifeBits.com project to capture entire life

56
List of finding aids
57
Apple at Stanford
58
www.alberteinstein.info
59
Allen Newell page
60
Lederberg
61
Computer History Museum

• 1401 Shoreline, Mountain View

62
Archiving computing artifacts

• Charles Babbage Institute Smithsonian is similar
• 135 collections 8K cu.ft. (20 M pages 2 TB)
• 160 oral histories (30MB/hr 6000 MB)
• 150 K photos (_at_1MB, 150 GB)
• Computer history Museum
• 6 K physical objects worlds best artifact
  collection
• 10 K photos
• 2 K videos (lt1 TB) including recent DV taped
  interviews
• 12 M pages books, manuals, brochures, papers,
  (1.2 TB)
• ?? Of executable source object codes
• 200 volunteers many more world-wide Amateurs
  versus professionals.

63
Computer History MuseumArtifact Collecting the
world is bits

• Artifact (the machine)
• Dormant or operating
• Hardware or software
• Project, people, plan
• Timeline of project
• Plan, schedule
• Specification, manuals
• Design
• Organization
• Communication
• Articles, books
• Interviews, talks, etc.

• Business aspects
• Plan, sales, marketing
• Ads, brochures, etc.
• Competitors
• Use
• User experience
• Video about its use
• Accessibility
• Raw bits, finding aid
• Interpreted story
• Exhibit

64
ChM Software Acquisition