1 MyLifeBits: Attempting to realize the Memex Vision Jim Gemmell & Roger Lueder Gordon Bell

2 Outline … MyLifeBits Background…fulfilling the Memex vision Background…fulfilling the Memex vision Cyberizing everything Cyberizing everything File to database transition File to database transition Use…beyond search Use…beyond search Long-term agenda and outlook Long-term agenda and outlook

3 Memex Posited by Vannevar Bush in “As We May Think” The Atlantic Monthly, July 1945 “A memex is a device in which an individual stores all his books, records, and communications, and which is mechanized so that it may be consulted with exceeding speed and flexibility” Supports: Annotations, links between documents, and “trails” through the documents “yet if the user inserted 5000 pages of material a day it would take him hundreds of years to fill the repository, so that he can be profligate and enter material freely”

4 Sketch of memex

Bush’s camera on the head

6 Capturing what you see

7 Memory Overload As hard drives get bigger and cheaper, we're storing way too much. By Jim Lewis There's a famous allegory about a map of the world that grows in detail until every point in reality has its counterpoint on paper; the twist being that such a map is at once ideally accurate and entirely useless, since it's the same size as the thing it's meant to represent. There's a famous allegory about a map of the world that grows in detail until every point in reality has its counterpoint on paper; the twist being that such a map is at once ideally accurate and entirely useless, since it's the same size as the thing it's meant to represent.

8 "The PC is going to be the place where you store the information and really the center of control“ Billg 1/7/2001 MyLifeBits is a project to “cyberize” everything! What? Recall of all articles, books, CDs, photos, video, communication (e.g. mail, phone), meetings,and web What? Recall of all articles, books, CDs, photos, video, communication (e.g. mail, phone), meetings,and web Why? …“because we can” Why? …“because we can” Office: communicate, store, & work Office: communicate, store, & work Home & Media Center: ambiance &entertainment Home & Media Center: ambiance &entertainment Immortality for progeny. Memory aids Immortality for progeny. Memory aids Goal: understand the 1 TByte PC for Lonfor Longhorn need, utility, cost, feasibility and tools. Goal: understand the 1 TByte PC for Lonfor Longhorn need, utility, cost, feasibility and tools.

9 LifeLog: A potential research program LifeLog: A (sub)system that captures, stores, and makes accessible the flow of one person’s experience in and interactions with the world LifeLog Thrust: Capture the “story” of a human Living Content Ontology (format) The End of the Line… Biographies Sagas Family Bibles Home Movies Photo Albums Videos Cave Paintings Blogs LifeLog

10 Gordon: Researcher, consumer, computer system tester, nerd wanna-be, and average man Melissa: middle manager Patrick: Consultant Nicholas: Analyst Sondra: Office manager Knowledge worker scenarios

11 The guinea pig Gordon Bell is digitizing his life Gordon Bell is digitizing his life Has now scanned virtually all: Has now scanned virtually all: Books written (and read when possible) Books written (and read when possible) Personal documents (correspondence including memos and , bills, legal documents, papers written, …) Personal documents (correspondence including memos and , bills, legal documents, papers written, …) Photos Photos Posters, paintings, photo of things (artifacts, …medals, plaques) Posters, paintings, photo of things (artifacts, …medals, plaques) Home movies and videos Home movies and videos CD collection CD collection And, of course, all PC files And, of course, all PC files Now recording: phone, radio, TV (movies), web pages… conversations and meetings to come Now recording: phone, radio, TV (movies), web pages… conversations and meetings to come Paperless throughout ” scanned, 12’ discarded. Paperless throughout ” scanned, 12’ discarded. Only 30 GB!!! Only 30 GB!!!

12 I am data

13 Capture and encoding

14 Quindi conference capture

15 I mean everything

16 Input: tools, time, and cost Scanners: HP Digital Sender, flat beds with ADF, 2-HP photo, faxing. (Duplex, color, feed-thru, etc.) Scanners: HP Digital Sender, flat beds with ADF, 2-HP photo, faxing. (Duplex, color, feed-thru, etc.) A good commercial scanner costs 2K-10K A good commercial scanner costs 2K-10K Photos: $1 or min. Large posters: ~ 1-5 hr. Artifacts: ~ 10 min. including photo Photos: $1 or min. Large posters: ~ 1-5 hr. Artifacts: ~ 10 min. including photo Scanning to TIF, PDF: <1 min/page or.10/page Scanning to TIF, PDF: <1 min/page or.10/page OCR: for MODI or PDF: ~3-5 pages/min (old data) OCR: for MODI or PDF: ~3-5 pages/min (old data) OCR: to recreate an editable “original” 10 min/page! OCR: to recreate an editable “original” 10 min/page! OCR (Volume paper files): 400 pages/hr. 7 ppm. OCR (Volume paper files): 400 pages/hr. 7 ppm. Books: scanned at CMU ($ /book) in 1997 Books: scanned at CMU ($ /book) in 1997 Videos: tbd Videos: tbd

17 Music 6.9 GB 1.8K files 180 CDs Working 2.3 GB 432 folders 2.9K files Archive 5.1 GB 477 folders 18.7 K files Video 2.6 GB 10 hours Low res My Books 98 MB 27.1K files & 42K.msg 17.7 GB (by size) Files (by number).xls.jpg.doc/html.pdf.ppt/ppt albums.tif CyberAll Nov.1, 2001 Mail.7 GB 43K msgs.doc/html.jpg.gif.xls.pdf.ppt.tif.gif

18 gbell wag: 67 yr, 25Kday life

19 gbell wag: 67 yr, 25Kday life

20 MyLifeBits organization: time and space Timeline/ Context (space) Personal (some $s) GB Co. (angel, etc.) Professional ACM, etc., New co’s. Archival (time) Working

21 MyLifeBits: Some Lives(t) Personal Personal Parents, children, grandkids Parents, children, grandkids CGB himself CGB himself GKB GKB Close friends Close friends GB $s GB $s Personal incl. several legal structures Personal incl. several legal structures Properties: autos, real estate, Properties: autos, real estate, Investments & contracts Investments & contracts Past prof. companies/organiz’ns Past prof. companies/organiz’ns DEC DEC Carnegie-Mellon U. Carnegie-Mellon U. DEC, NSF, Encore, Ardent, Me Inc., DEC, NSF, Encore, Ardent, Me Inc., Microsoft Microsoft MLB MLB Clusters Clusters Telepresence Telepresence WWW presence WWW presence Computer History Museum Computer History Museum BOD member BOD member Fund-raising Fund-raising CyberMuseum CyberMuseum Startups & boards Startups & boards Bell-Mason Director Bell-Mason Director Diamond & Vanguard Brds. Diamond & Vanguard Brds.

C,L m d d CGB... GB SR mB,L KF SB Where KvMO B ABos P B WCa 6-year --GS-HS---MIT DEC Education KV-----mit,F cmu Work Bell Elec DECcmuDEC E,NSF MSFT ComputerMuseum M B SiValley Books BN SBN HiTechVent Computers VAX E T Awards..

23 Personal LifeLog Applications Conservator Baby Book Companion Caretaker Babysitter Advisor Mentor Tutor Autobiography Photo Album Personal Assistant Diary/Journal Biography Financial Manager Medical Manager Executor Obituary OthersSelf Assistant for Elderly Application controlled by: Others Self Application used by: Personal Proxy Parole Officer Pers Flight Recorder Meeting Prep Captain’s Log Trustee

24 How LifeLog Fits Physical Cameras Microphone GPS, IMU Biomedical Others Transactions Other Cyber Phone, Vmail Fax Money, etc. Media TV, Radio Hardcopy Softcopy Ref. Data Others Data Capture and Distillation LifeLog Representation & Abstraction (Ontology) Access Modes Autobiography Search Assist Teach Biography Monitor Analyze Predict Multibiography Correlate Statistical Sources Applications Synbiography Generate Predict

25 MyLifeBits is: Memex and more (audio and video) Memex and more (audio and video) Universal store for all personal stuff Universal store for all personal stuff Guiding principles for the system: Guiding principles for the system: 1. Full text search & collections (> than hierarchy) 2. Visualizations for search, display, insight 3. Annotations and links add value and essential Increase search ability and value of information. Increase search ability and value of information. So make many kinds and them easy to create! So make many kinds and them easy to create! Stories are the ultimate annotation Stories are the ultimate annotation 4. Keep the links when you author: “transclusion”

26 MLB database: size and content? Database features are essential: Consistency, Indexing, Pivoting, Queries, Speed/scalability, Backup, replication. Database features are essential: Consistency, Indexing, Pivoting, Queries, Speed/scalability, Backup, replication. Folders &Files were the starting point >> database into sets aka “collections” that are identical to the folder structure Folders &Files were the starting point >> database into sets aka “collections” that are identical to the folder structure Outlook (msgs, attachments, calendar, contacts) Outlook (msgs, attachments, calendar, contacts) Web trails including voice message annotation Web trails including voice message annotation Journal (Outlook), trails: every document use & transaction Journal (Outlook), trails: every document use & transaction What about? What about? Money (transactions, payees, etc.)…is their lifelog/trail Money (transactions, payees, etc.)…is their lifelog/trail Streets and trips to cross-index to all docs Streets and trips to cross-index to all docs Attributes for photos for retrieval? Location, time, settings Attributes for photos for retrieval? Location, time, settings Presentations as a report or trail. Each slide an object! Presentations as a report or trail. Each slide an object!

27 Searching: the most useful app? Challenge: What questions for useful results? Challenge: What questions for useful results? Lots of ways to look at what you retrieve Lots of ways to look at what you retrieve Need for breaking the returns into segments Need for breaking the returns into segments Searching for an indexer and search engine: index service, Enfish, dtSearch Searching for an indexer and search engine: index service, Enfish, dtSearch Stuff I’ve Seen MSR’s index & search… evolving in the right direction. Stuff I’ve Seen MSR’s index & search… evolving in the right direction. Productizing would remove the pressure for Longhorn Productizing would remove the pressure for Longhorn

Internet MyLifeBits store database files Voice annotation tool Text annotation tool Legacy applications MAPI interface Legacy client Radio EPG tool PocketPC transfer tool Telephone capture tool Radio capture tool TV capture tool TV EPG download tool Browser tool PocketRadio player MyLifeBits Shell

29 Annotation like this… Voice Annotation

30 Pivot to look at all of MLB(t) Call, contact, pivot by time to find web page

31 Find brig, image, and look for 80

32 Here are the photos

33 Timeline view tells a story

34 Finding scatological works

35 Statistics of use

36

37

38 Detail view

39 Resource explorer Ancestor (collections), annotations, descendant & preview panes turned on

40 Interface to xls

41

42

43 Synchronized timelines with histogram guide

44 Visualization Browsing & searching. “Get me what I want|need!” Browsing & searching. “Get me what I want|need!” Help the user find things among possible items versus Help the user find things among possible items versus Waiting for an ideal system that can find “what I want” Waiting for an ideal system that can find “what I want” Publication: Conventional & web, presentations, etc. Publication: Conventional & web, presentations, etc. Helps understand the nature of the content e.g. histogram of objects in time Helps understand the nature of the content e.g. histogram of objects in time Context: Links to help understand the relationship between objects. Provides more search handles. Context: Links to help understand the relationship between objects. Provides more search handles. Information density: what is it? What is its relationship to others? Information density: what is it? What is its relationship to others? Content important. Flash and form, less useful. Content important. Flash and form, less useful.

45 Value of media depends on annotations “Its just bits until it is annotated” “Its just bits until it is annotated”

46 System annotations provide base level of value Date 7/7/2000 Date 7/7/2000

47 Tracking usage – even better Date 7/7/2000. Opened 30 times, ed to 10 people (its valued by the user!) Date 7/7/2000. Opened 30 times, ed to 10 people (its valued by the user!)

48 Get the user to say a little something is a big jump Date 7/7/2000. Opened 30 times, ed to 10 people. “BARC dim sum intern farewell Lunch” Date 7/7/2000. Opened 30 times, ed to 10 people. “BARC dim sum intern farewell Lunch”

49 Getting the user to tell a story is the ultimate in media value A story is a “layout” in time and space A story is a “layout” in time and space Most valuable content (by selection, and by being well annotated) Most valuable content (by selection, and by being well annotated) Stories must include links to any media they use (for future navigation/search – “transclusion”). Stories must include links to any media they use (for future navigation/search – “transclusion”). Cf: MovieMaker; Creative Memories PhotoAlbums Cf: MovieMaker; Creative Memories PhotoAlbums Dapeng was an intern at BARC for the summer of 2000 We took him to lunch at our favorite Dim Sum place to say farewell At table L-R: Dapeng, Gordon, Tom, Jim, Don, Vicky, Patrick, Jim

50 Value of media depends on annotations Auto-annotate whenever possible e.g. GPS cameras Auto-annotate whenever possible e.g. GPS cameras Make manual annotation as easy as possible. XP photo capture, voice, photos with voice, etc Make manual annotation as easy as possible. XP photo capture, voice, photos with voice, etc Support gang annotation Support gang annotation Make stories easy Make stories easy “Its just bits until it is annotated”

51

52

53 CD VCR Cassette Plasma Panel DVD Media Center Computer Set top KbdMse Wfr Spkr IR Cable/ Satellite Ethernet SVHS-wide 5.1 digital 5 speakers stereo Video* 5.1 digital comp. stereo Video* Cables/links Speaker 5+1 Plasma 2 or 3 Cable/Enet 2 IR 8 Stereo digital 2 Comp./S-video 3 Plasma panel 1 Power 10 Kbd/mse 2 Monitor II (opt.) 4 Camera 2 Total 42 – 46 Things 18+remotes *Video = composite or S-video Camera Mic Receiver L egacy R edundant

54

55 Media center 2

56 Photos

57 Caneel Bay Vacation Jan Gordon, Gwen, Brig, Pam, Fiona, Bob, Laura and Kolbe

58 MyLifeBits use scenarios 1. Acquire everything! (I mean everything!_ 2. Professional personal use at work! 3. Home/personal: Provide ambiance & entertainment using Home Media Center 4. Enhancing content through photo and video albums Events, places, trips, people, time intervals Database land and authoring How I spend my time or an interval of time. Recall a “trail“… What was I thinking about? 6. Endless need for authoring & reporting tools ► ISBQ: Interactive Story By Query ► A Person (auto- or -biography web hosted time line ► Personal/web/org. hosted collections & catalogs

59 The Agenda for the Tbyte(s), Lifetime, PC: The killer app after office and mail. 1. Guarantee that data will live forever! “dear appy” problem 2. Cheap, easy, and data-rich (e.g. time, place) capture: GPS and time everywhere Paper capture has to be as easy as discard (scanner/shredder) Personal meeting capture... E-book…e-magazines & journals need to have critical mass! Telephony and audio capture with indexing Media Center compatible for entertainment (photos, video, TV, radio) 3. Content analysis (critical for photo & video!) 4. Information control: privacy, security, expunge/deniability,… 5. One dbase for everything (articles, books, conversations,... financial transactions) …vs. long-term use of hierarchical files. Is dbase intuitive? 6. Annotations/meta-information add every-increasing value Easy annotation for aiding search and it becomes the content 7. The “killer apps”: Alzheimer, immortality, surrogate memory? 8. GUI’s to improve use (e.g. time to learn, use, retention)

60 The “dear appy” problem Dear Appy, How committed are you? Please come back to me, Lost and forgotten data Who’s responsible? Who’s responsible? media media platform, file, and databases platform, file, and databases evolving standards and formats evolving standards and formats evolving and/or disappearing apps evolving and/or disappearing apps

61 The Amnesia Control Problem Full sharing of bits that are mine Full sharing of bits that are mine I created them, OK to copy and distribute I created them, OK to copy and distribute DRM: purchased for my own use DRM: purchased for my own use “OK to look at, but I only own half the bits” “OK to look at, but I only own half the bits” Controlling forgetfulness Controlling forgetfulness Private, do not “demo” Private, do not “demo” Expunge forever... “this never happened” Expunge forever... “this never happened”

62 The Content Analysis Problem 1. “Cliplets”: Automatic segmentation of a pile of documents and video into individual documents and scenes. 2. Item typing: Would like a minimal Dublin Core for each item: date, creator, title, source, abstract, and type 3. “Type” classification: articles, letters, memos, etc. 4. Ontology creation for collections

63 The End