1 MyLifeBits: Realizing the Memex Vision Santa Clara University 13 May 2004 Gordon Bell, Jim Gemmell & Roger Lueder

2 Mylifebits collage

3 Outline … MyLifeBits Background…fulfilling the Memex vision Background…fulfilling the Memex vision Cyberizing everything Cyberizing everything File to database transition File to database transition Use…beyond search Use…beyond search Working with Media Center for home use Working with Media Center for home use Long-term agenda and outlook Long-term agenda and outlook Archiving persons and things. Archiving persons and things.

4 Memex As We May Think, Vannevar Bush, 1945 “A memex is a device in which an individual stores all his books, records, and communications, and which is mechanized so that it may be consulted with exceeding speed and flexibility” Full-text search, text & audio annotations, and hyperlinks Full-text search, text & audio annotations, and hyperlinks

5 Capturing what you see

6 I am data

7 The guinea pig Gordon Bell is digitizing his life Gordon Bell is digitizing his life Has now scanned virtually all: Has now scanned virtually all: Books written (and read when possible) Books written (and read when possible) Personal documents (correspondence including memos and , bills, legal documents, papers written, …) Personal documents (correspondence including memos and , bills, legal documents, papers written, …) Photos Photos Posters, paintings, photo of things (artifacts, …medals, plaques) Posters, paintings, photo of things (artifacts, …medals, plaques) Home movies and videos Home movies and videos CD collection CD collection And, of course, all PC files And, of course, all PC files Now recording: phone, radio, TV (movies), web pages… conversations and meetings to come Now recording: phone, radio, TV (movies), web pages… conversations and meetings to come Paperless throughout ” scanned, 12’ discarded. Paperless throughout ” scanned, 12’ discarded. Only 30 GB!!! Only 30 GB!!!

8 Capture and encoding

9 Quindi conference capture

10 I mean everything

11 Wearable & interactive jewellery LEDs flash according to sensor type triggered

12 Potentially useful trivia – but not normally photographed

13 GPS: tells where and when

14 Kentaro Toyama wwmx.org

15 gbell wag: 67 yr, 25Kday life

16 MyLifeBits organization: time and space Timeline/ Context (space) Personal (some $s) GB Co. (angel, etc.) Professional ACM, etc., New co’s. Archival (time) Working

17 MyLifeBits: Some Lives(t) Personal Personal Parents, children, grandkids Parents, children, grandkids CGB himself CGB himself GKB GKB Close friends Close friends GB $s GB $s Personal incl. several legal structures Personal incl. several legal structures Properties: autos, real estate, Properties: autos, real estate, Investments & contracts Investments & contracts Past prof. companies/organiz’ns Past prof. companies/organiz’ns DEC DEC Carnegie-Mellon U. Carnegie-Mellon U. DEC, NSF, Encore, Ardent, Me Inc., DEC, NSF, Encore, Ardent, Me Inc., Microsoft Microsoft MLB MLB Clusters Clusters Telepresence Telepresence WWW presence WWW presence Computer History Museum Computer History Museum BOD member BOD member Fund-raising Fund-raising CyberMuseum CyberMuseum Startups & boards Startups & boards Bell-Mason Director Bell-Mason Director Diamond & Vanguard Brds. Diamond & Vanguard Brds.

18 Bell Lives timeline C,L m d d CGB... GB SR mB,L KF SB Where KvMO B ABosP B WCa 6-year --GS-HS---MIT DEC Education KV-----mit,F cmu Work Bell Elec DECcmuDEC E,NSF MSFT ComputerMuseum M B SiValley Books BN SBN HiTechVent Computers VAX E A

19 Personal LifeLog Applications Conservator Baby Book Companion Caretaker Babysitter Advisor Mentor Tutor Autobiography Photo Album Personal Assistant Diary/Journal Biography Financial Manager Medical Manager Executor Obituary OthersSelf Assistant for Elderly Application controlled by: Others Self Application used by: Personal Proxy Parole Officer Pers Flight Recorder Meeting Prep Captain’s Log Trustee

20 MyLifeBits Software MyLifeBits store database Voice annotation tool Text annotation tool Telephone capture tool TV capture tool TV EPG download tool Radio capture tool Radio EPG tool PocketPC transfer tool PocketRadio player Import files MyLifeBits Shell files Legacy applications Browser tool Internet IM capture MAPI interface Legacy client

21 MyLifeBits is: Memex and more (audio and video) Memex and more (audio and video) Universal store for all personal stuff Universal store for all personal stuff Guiding principles for the system: Guiding principles for the system: 1. Full text search & collections (> than hierarchy) 2. Visualizations for search, display, insight 3. Annotations and links add value and essential Increase search ability and value of information. Increase search ability and value of information. So make many kinds and them easy to create! So make many kinds and them easy to create! Stories are the ultimate annotation Stories are the ultimate annotation 4. Keep the links when you author: “transclusion”

22 MLB database: size and content? Database features are essential: Consistency, Indexing, Pivoting, Queries, Speed/scalability, Backup, replication. Database features are essential: Consistency, Indexing, Pivoting, Queries, Speed/scalability, Backup, replication. Folders &Files were the starting point >> database into sets aka “collections” that are identical to the folder structure Folders &Files were the starting point >> database into sets aka “collections” that are identical to the folder structure Outlook (msgs, attachments, calendar, contacts) Outlook (msgs, attachments, calendar, contacts) Web trails including voice message annotation Web trails including voice message annotation Journal (Outlook), trails: every document use & transaction Journal (Outlook), trails: every document use & transaction What about? What about? Money (transactions, payees, etc.)…is their lifelog/trail Money (transactions, payees, etc.)…is their lifelog/trail Streets and trips to cross-index to all docs Streets and trips to cross-index to all docs Attributes for photos for retrieval? Location, time, settings Attributes for photos for retrieval? Location, time, settings Presentations as a report or trail. Each slide an object! Presentations as a report or trail. Each slide an object!

23 Why bother? An existence proof. The following exist in abundance: Shoeboxes full of photos Shoeboxes full of photos Photo albums & framed photos Photo albums & framed photos Creative Memories is a thriving business selling resources for created high-end photo albums that are well laid out and highly annotated, using long-lasting materials. Creative Memories is a thriving business selling resources for created high-end photo albums that are well laid out and highly annotated, using long-lasting materials. Home videos Home videos Bookshelves and filing cabinets Bookshelves and filing cabinets Old bundles of letters Old bundles of letters Professional video/photo companies do capture at kids’ sports events and sell content like hotcakes Professional video/photo companies do capture at kids’ sports events and sell content like hotcakes Probably not accessed very often but TREASURED (what’s the one thing you would save in a fire?) Probably not accessed very often but TREASURED (what’s the one thing you would save in a fire?)

24 Why bother?..more reasons To eliminate physical storage (paper, CDs…) To eliminate physical storage (paper, CDs…) It costs more (in time) to delete than the cost the storage It costs more (in time) to delete than the cost the storage You may only want to retrieve one of many items in the future, but cannot predict which one (which is why you file many things now) You may only want to retrieve one of many items in the future, but cannot predict which one (which is why you file many things now) For posterity and nostalgia For posterity and nostalgia For memory enhancement & faster search (search your LifeBits rather than the web … a single source to look for anything you have ever seen) For memory enhancement & faster search (search your LifeBits rather than the web … a single source to look for anything you have ever seen) Let content analysis and data mining discover trends and correlations in your life Let content analysis and data mining discover trends and correlations in your life

Extensible XML schemas Logical views Programmatic relationships Synchronization service Information agents Extensible XML schemas Logical views Programmatic relationships Synchronization service Information agents application specific data system people application specific data user application specific data infrastructure application specific data

26 Annotation like this… Voice Annotation

27

28 Pivot to look at all of MLB(t) Call, contact, pivot by time to find web page

29 Find brig, image, and look for 80

30 Here are the photos

31 Timeline view tells a story

32 Interface to xls

33 Statistics of use

34 Visualization Browsing & searching. “Get me what I want|need!” Browsing & searching. “Get me what I want|need!” Help the user find things among possible items versus Help the user find things among possible items versus Waiting for an ideal system that can find “what I want” Waiting for an ideal system that can find “what I want” Publication: Conventional & web, presentations, etc. Publication: Conventional & web, presentations, etc. Helps understand the nature of the content e.g. histogram of objects in time Helps understand the nature of the content e.g. histogram of objects in time Context: Links to help understand the relationship between objects. Provides more search handles. Context: Links to help understand the relationship between objects. Provides more search handles. Information density: what is it? What is its relationship to others? Information density: what is it? What is its relationship to others? Content important. Flash and form, less useful. Content important. Flash and form, less useful.

35 Value of media depends on annotations “Its just bits until it is annotated” “Its just bits until it is annotated”

36 Getting the user to tell a story is the ultimate in media value A story is a “layout” in time and space A story is a “layout” in time and space Most valuable content (by selection, and by being well annotated) Most valuable content (by selection, and by being well annotated) Stories must include links to any media they use (for future navigation/search – “transclusion”). Stories must include links to any media they use (for future navigation/search – “transclusion”). Cf: MovieMaker; Creative Memories PhotoAlbums Cf: MovieMaker; Creative Memories PhotoAlbums Dapeng was an intern at BARC for the summer of 2000 We took him to lunch at our favorite Dim Sum place to say farewell At table L-R: Dapeng, Gordon, Tom, Jim, Don, Vicky, Patrick, Jim

37 Value of media depends on annotations Auto-annotate whenever possible e.g. GPS cameras Auto-annotate whenever possible e.g. GPS cameras Make manual annotation as easy as possible. XP photo capture, voice, photos with voice, etc Make manual annotation as easy as possible. XP photo capture, voice, photos with voice, etc Support gang annotation Support gang annotation Make stories easy Make stories easy “Its just bits until it is annotated”

38 Future work: Visualizations Don't give me a little card image and say, "That's all you've got, because that's what I thought you should want for your virtual shoebox." There have got to be multiple modalities and the designers have to be able to deal with that. … don't metaphor me in, don't give me only one way of looking at things. Don't give me a little card image and say, "That's all you've got, because that's what I thought you should want for your virtual shoebox." There have got to be multiple modalities and the designers have to be able to deal with that. … don't metaphor me in, don't give me only one way of looking at things. -Andy van Dam, Hypertext '87 Keynote Address Next Media Web Scout U. Maryland IN-SPIRE

39 LifeLines (Plaisant et al.) University of Maryland

40 Rethinking collections & files Date collections (“summer 99”) Date collections (“summer 99”) Much better as a query Much better as a query By Person (“Photos of Bill”) By Person (“Photos of Bill”) Better as links of type “photo of” to person “Bill” Better as links of type “photo of” to person “Bill” By Event (“Trip to UCLA”) By Event (“Trip to UCLA”) Better as links to event in calendar Better as links to event in calendar Working set Working set Better as query that figures it out for me so I don’t need to maintain it Better as query that figures it out for me so I don’t need to maintain it

41 Facets and people Time (& stage of life). Events… Location (lat/long vs home, vacation) Institution (relations including family, work, clubs,…) Role (student, professional, parent, owner, etc.) Content type –Audio, graphics, photo, video aka moving picture –Document t type o(200) plus profession specific ad, bill…will, cards (calling, credit, grade, greeting), certificate (birth…death), correspondence, diary, essay, forms, legal (6), instructions, lists, resume, reservation, scrapbook, transcript, Dissemination –Book, electronic, serial, unpublished, Special collections (e.g. geology, stamps, species, places)

42 Facet Lists

43 Certificate facets

44 “By region” and “by time” should be facets!

45 Telephone, Television, and Radio in the Home of the Future

46 Evolution of media in the home Yesterday: Today: Tomorrow: Analog storage and transmission on separate networks Analog storage and transmission on separate networks Physical space limitations Physical space limitations Tedious management and manual search Tedious management and manual search Digital storage (CDs, DVDs, PVRs, MPEG & WMA/V) Digital storage (CDs, DVDs, PVRs, MPEG & WMA/V) Digital cable, internet radio, but phone is mostly analog Digital cable, internet radio, but phone is mostly analog Still limitations on what we can store Still limitations on what we can store Different stores for different stuff Different stores for different stuff All digital All digital Everything connected Everything connected Unlimited storage Unlimited storage Everything in a database Everything in a database SQL

47 CD VCR Cassette Plasma Panel DVD Media Center Computer Set top KbdMse Wfr Spkr IR Cable/ Satellite Ethernet SVHS-wide 5.1 digital 5 speakers stereo Video* 5.1 digital comp. stereo Video* Cables/links Speaker 5+1 Plasma 2 or 3 Cable/Enet 2 IR 8 Stereo digital 2 Comp./S-video 3 Plasma panel 1 Power 10 Kbd/mse 2 Monitor II (opt.) 4 Camera 2 Total 42 – 46 Things 18+remotes *Video = composite or S-video Camera Mic Receiver L egacy R edundant

48

49 MyLifeBits use scenarios 1. Acquire everything! (I mean everything!_ 2. Professional personal use at work! 3. Home/personal: Provide ambiance & entertainment using Home Media Center 4. Enhancing content through photo and video albums Events, places, trips, people, time intervals Database land and authoring How I spend my time or an interval of time. Recall a “trail“… What was I thinking about? 6. Endless need for authoring & reporting tools ► ISBQ: Interactive Story By Query ► A Person (auto- or -biography web hosted time line ► Personal/web/org. hosted collections & catalogs

50 The Agenda for the Tbyte(s), Lifetime, PC: The killer app after office and mail. 1. Guarantee that data will live forever! “dear appy” problem 2. Cheap, easy, and data-rich (e.g. time, place) capture: GPS and time everywhere Paper capture has to be as easy as discarding (scanner/shredder) Personal meeting capture... E-book…e-magazines & journals need to have critical mass! Telephony and audio capture with indexing Media Center compatible for entertainment (photos, video, TV, radio) 3. Content analysis (critical for photo & video!) 4. Information control: privacy, security, expunge/deniability,… 5. Having to be schizophrenic or have a lobotomy when leaving a “life” 6. One dbase for everything (articles, books, conversations,... financial transactions) …vs. long-term use of hierarchical files. Is dbase intuitive? 7. Annotations/meta-information add every-increasing value Easy annotation for aiding search and it becomes the content 8. The “killer apps”: Alzheimer, immortality, surrogate memory? 9. GUI’s to improve use (e.g. time to learn, use, retention)

51 The “dear appy” problem Dear Appy, How committed are you? Please come back to me, Lost and forgotten data Who’s responsible? Who’s responsible? media media platform, file, and databases platform, file, and databases evolving standards and formats evolving standards and formats evolving and/or disappearing apps evolving and/or disappearing apps

52 Problems: “Amnesia” control & deleting corporate “life” bits Full sharing of bits that are mine Full sharing of bits that are mine I created them, OK to copy and distribute I created them, OK to copy and distribute DRM: purchased for my own use DRM: purchased for my own use “OK to look at, but I only own half the bits” “OK to look at, but I only own half the bits” Controlling forgetfulness Controlling forgetfulness Private, do not “demo” Private, do not “demo” Expunge forever... “this never happened” Expunge forever... “this never happened” The bits “belong” to a corporation or org. The bits “belong” to a corporation or org.

53 The Content Analysis Problem 1. “Cliplets”: Automatic segmentation of a pile of documents and video into individual documents and scenes. 2. Item typing: Would like a minimal Dublin Core for each item: date, creator, title, source, abstract, and type 3. “Type” classification: articles, letters, memos, etc. 4. Ontology creation for collections

54 The End

55 Archiving persons and things… for 0(1K) corporations, people, places, things. –List of finders, usually -> paper boxes! –E.g. Apple collection at Stanford points to 600’ or say $1K/ft. Einstein’s papers, etc. diva.library.cmu.edu/Newell/ for Allen Newelldiva.library.cmu.edu/Newell/ profiles.nlm.nih.gov/ Nobel Prize winners, Lederbergprofiles.nlm.nih.gov/ computing artifactswww.ComputerHistory.org project to capture entire lifewww.MyLifeBits.com

56 List of finding aids

57 Apple at Stanford

58

59 Allen Newell page

60 Lederberg

61 Computer History Museum 1401 Shoreline, Mountain View

62 Archiving computing artifacts Charles Babbage Institute …Smithsonian is similar –135 collections 8K cu.ft. (20 M pages; 2 TB) –160 oral histories (30MB/hr =6000 MB) –150 K photos 150 GB) Computer history Museum –6 K physical objects: world’s best artifact collection –10 K photos –2 K videos (<1 TB); including recent DV taped interviews –12 M pages books, manuals, brochures, papers, (1.2 TB) –?? Of executable source & object codes –200 volunteers & many more world-wide Amateurs versus professionals.

63 Computer History Museum Artifact Collecting… the world is bits Artifact (“the machine”) –Dormant or operating –Hardware or software Project, people, plan –Timeline of project –Plan, schedule –Specification, manuals –Design –Organization –Communication –Articles, books –Interviews, talks, etc. Business aspects –Plan, sales, marketing –Ads, brochures, etc. –Competitors Use –User experience –Video about it’s use Accessibility –Raw bits, finding aid –Interpreted story –Exhibit

64 ChM Software Acquisition