Challenges in Using Lifetime Personal Information Stores based on MyLifeBits Gordon Bell Alpbach Forum 26 August 2004

The 1 TB Life 1TB gives you 65+ years of: 1TB gives you 65+ years of: 100 messages a day (5KB each) 100 messages a day (5KB each) 100 web pages day (50KB each) 100 web pages day (50KB each) 5 scanned pages a day (100KB each) 5 scanned pages a day (100KB each) 1 book every 10 days (1 MB each) 1 book every 10 days (1 MB each) 10 photos per day (400 KB JPEG each) 10 photos per day (400 KB JPEG each) 8 hours per day of sound - e.g. telephone, voice annotations, and meeting recordings (8 Kb/s) 8 hours per day of sound - e.g. telephone, voice annotations, and meeting recordings (8 Kb/s) 1 new music CD every 10 days (45 min each at 128 Kb/s) 1 new music CD every 10 days (45 min each at 128 Kb/s) It will take you 5 years to fill up your 80 GB drive It will take you 5 years to fill up your 80 GB drive Want video? Buy more cheap drives (1 TB/year lets you record 4 hours/day of 1.5 Mb/s video) Want video? Buy more cheap drives (1 TB/year lets you record 4 hours/day of 1.5 Mb/s video)

Everything goes in a database You need all the features of a database (Consistency, Indexing, Pivoting, Queries, Speed/scalability, Backup, replication) You need all the features of a database (Consistency, Indexing, Pivoting, Queries, Speed/scalability, Backup, replication) If you dont use one, you will find yourself creating one! If you dont use one, you will find yourself creating one! Files as blobs, also sync with file system for legacy apps Files as blobs, also sync with file system for legacy apps SQL

MyLifeBits Software MyLifeBits store database Voice annotation tool Text annotation tool Telephone capture tool TV capture tool TV EPG download tool Radio capture & EPG PocketPC transfer tool PocketRadio player Import files MyLifeBits Shell files Legacy applications Browser tool Internet IM capture MAPI interface Legacy client GPS import & Map display SenseCam Screen saver

Memex As We May Think, Vannevar Bush, 1945 A memex is a device in which an individual stores all his books, records, and communications, and which is mechanized so that it may be consulted with exceeding speed and flexibility Full-text search, text & audio annotations, and hyperlinks Full-text search, text & audio annotations, and hyperlinks

I am data

The guinea pig Gordon Bell is digitizing his life Gordon Bell is digitizing his life Has now scanned virtually all: Has now scanned virtually all: Books written (and read when possible) Books written (and read when possible) Personal documents (correspondence including memos and , bills, legal documents, papers written, …) Personal documents (correspondence including memos and , bills, legal documents, papers written, …) Photos Photos Posters, paintings, photo of things (artifacts, …medals, plaques) Posters, paintings, photo of things (artifacts, …medals, plaques) Home movies and videos Home movies and videos CD collection CD collection And, of course, all PC files And, of course, all PC files Now recording: phone, radio, TV (movies), web pages… conversations and meetings to come Now recording: phone, radio, TV (movies), web pages… conversations and meetings to come Paperless throughout scanned, 12 discarded. Paperless throughout scanned, 12 discarded. Only 44 GB, incl. 10 wma, 14 SQL!!! Video: o(100) mov Only 44 GB, incl. 10 wma, 14 SQL!!! Video: o(100) mov

Capture and encoding

I mean everything

50+ year old newspaper clippings

400 year old books

O(100s) tapes from videotape black hole

Personal LifeLog Applications Conservator Baby Book Companion Caretaker Babysitter Advisor Mentor Tutor Autobiography Photo Album Personal Assistant Diary/Journal Biography Medical Manager Executor Obituary OthersSelf Assistant for Elderly Application controlled by: Others Self Application used by: Personal Proxy Parole Officer Pers Flight Recorder Meeting Prep Captains Log Trustee Financial Manager

Personal Search is not Professional or Web search System sees every entry & access System sees every entry & access Everything, not just a professional life Everything, not just a professional life Limited to SIS, not an infinite amount, covers a profession & personal life Limited to SIS, not an infinite amount, covers a profession & personal life Web as seen by search engines MyLifeBits Knowledge breadth e.g. Dewey classification Depth e.g. information item types & coverage Professional user

Why bother?..some reasons Technologist: we can an opportunity e.g. 1 TB disks Technologist: we can an opportunity e.g. 1 TB disks For all of us with new media: a need e.g. jpg. Mp3 For all of us with new media: a need e.g. jpg. Mp3 Environmentalist: eliminates atoms (paper, CDs…) Environmentalist: eliminates atoms (paper, CDs…) For business--memory enhancement & faster search: Let content analysis and data mining discover trends and correlations in our lives…that even we dont know. For business--memory enhancement & faster search: Let content analysis and data mining discover trends and correlations in our lives…that even we dont know. Business: It costs more to delete than it costs to store Business: It costs more to delete than it costs to store Preservationist: decays or disappears unless its saved Preservationist: decays or disappears unless its saved For the human pack rat: I may need it some day. For the human pack rat: I may need it some day. For posterity and nostalgia: Maybe others will want it. For posterity and nostalgia: Maybe others will want it. Stories and ambience: basis for creating content Stories and ambience: basis for creating content For the aging & failed memory: surrogate memory For the aging & failed memory: surrogate memory

So youve got it – now what do you do with it? A record if it is to be useful … must be continuously extended, it must be stored, and above all it must be consulted The difficulty seems to be, not so much that we publish unduly … but rather that publication has been extended far beyond our present ability to make real use of the record - Vannevar Bush

Using my life bits: beyond folders #1: Folders One item. One place. It worked for 1000s of years.

My docs and archive S SelfSelf E E X- Employer Employer X-Employer Project Employer Library/file cab Active Employer Library/file cab <1995 Library/file cab Library/file cab Project Business Invests, family $s, & Legal Personal, including Medical Library/file cab

Freedom from hierarchy c:\my documents\talks\MyLifeBits.ppt ID=location=organization=display string c:\my documents\talks\MyLifeBits.ppt ID=location=organization=display string Dont make me invent unique names Dont make me invent unique names Dont make me file everything Dont make me file everything Or let me pick multiple folders Or let me pick multiple folders

multiple categorization not only improves organization and retrieval times but also matches more closely with the way users naturally think about organizing their information – Quan et al (MITs Haystack) multiple categorization not only improves organization and retrieval times but also matches more closely with the way users naturally think about organizing their information – Quan et al (MITs Haystack) MyLifeBits collection dialog Of course Aliases and Shortcuts can be used albeit painfully to file by time and/or event, subject, location, type.

Using my life bits: easily adding valuable content #2: Text annotations Making bits more valuable and retrievable.

Its just bits until it is annotated

Getting the user to tell a story is the ultimate in media value A story is a layout in time and space A story is a layout in time and space Most valuable content (by selection, and by being well annotated) Most valuable content (by selection, and by being well annotated) Stories must include links to any media they use (for future navigation/search – transclusion). Stories must include links to any media they use (for future navigation/search – transclusion). Cf: MovieMaker; Creative Memories PhotoAlbums Cf: MovieMaker; Creative Memories PhotoAlbums Dapeng was an intern at BARC for the summer of 2000 We took him to lunch at our favorite Dim Sum place to say farewell At table L-R: Dapeng, Gordon, Tom, Jim, Don, Vicky, Patrick, Jim

Annotation like this… Voice Annotation

Annotation when you feel like it, how you feel like it Screensaver is the killer app! Screensaver is the killer app!

Using my life bits: the value of time & time posts #3: I remember when… The 1 st or 2 nd most important retrieval handle.

MyLifeBits time overlap

MyLifeBits on-the-fly time clustering

F: father F: mother F: self F: Sister Education F: spouse F: son F: grandChild F: daughter F: grandchild F: Significant Other W/Education Work Organization

M Stewart Lifeline v2 Mark Stewarts Lifeline Copyright Mark Stewart, 2004

MSR Next Media Team

Using my life bits: Where, an essential attribute #4: I remember where Just essential.

Using my life bits: pivoting on data to aid recall #5: Relationships (links) Using something near it, to find it.

MyLifeBits Entities & Links Annotates Caller in Phone Call Photo of Event Transcludes

The entity-based mode of interaction is also promising for use as a personal database -Cattell

Is traditional classification required? … at OCLC (c1997) there was unanimous agreement among faculty and participants that access to electronic resources requires controlled vocabulary and classification

The results convincingly demonstrate that the category interface is superior to the list interface -Dumais, Cutrell & Chen

Mark Stewarts first page Copyright Mark Stewart, 2004

The Stew family tree Copyright Mark Stewart, 2004

PhotoFinder - Schneiderman and Kang

Using my life bits: never enough meta-data … but, can you afford it?b #6: more meta-data (properties) I remember something about the content (understanding a persons work)

Lederberg Finder page

Dublin core of a given item

Using my life bits: classification of everything #7: classification Is any gain from non-automated classification worth the cost and pain?

Is traditional classification required? … at OCLC there was unanimous agreement among faculty and participants that access to electronic resources requires controlled vocabulary and classification OCLC Institute, Knowledge Access Management: Tools and Concepts for Next Generation Catalogers, November 1997, Dublin, Ohio.

I have watched as hundreds of millions of dollars have been invested to re-invent the wheel - often badly. -Marcia Bates

Professional Life: Organizations Administrivia Projects Library

Lederberg papers official reports Number of document segments

Lederberg Artifact types Abstracts Agendas not Agendas Announcements m; Announcements Application forms Articles m Articles Autobiographies m Autobiographies Bibliographies m Bibliographies Biographies m Biographies Brochures m Brochures Certificates m Certificates Correspondence m Correspondence Diaries m Diaries Drafts (documents) Drawings m Drawings Electronic images m Electronic images Essays m Essays Eulogies Excerpts Grant proposals Interviews m Interviews Invitations Laboratory notebooks m Laboratory notebooks Laboratory notes Lecture notes Lectures m Lectures Legal documents m Legal documents Legislative records Lists Manifestoes Memoirs m Memoirs Minutes Monographs m Monographs Narratives Newsletters Newspaper columns m Newspaper columns Notebooks m Notebooks Notes Obituaries Official reports Oral histories m Oral histories Petitions Photographic prints m Photographic prints Press releasesPress releases m Procedures ProceedingsProceedings m ProgramsPrograms m ProposalsProposals m Questionnaires Reminiscences ReportsReports m Resolutions Resumes ReviewsReviews m School records SpeechesSpeeches m Summaries Tables (documents) Technical reportsTechnical reports m TranscriptsTranscripts m Typescripts Video recordingsVideo recordings m

Species: Animals: Chordata: Vertebrata: bony fish

Computer structures: digital computer: minicomputer

Computer structures: digital computer: minicomputer (refined: Digital Equipment Corp.)

Computer structures taxonomy: computers

Classification wish list Download classifications rather than build them Download classifications rather than build them Definitions & synonyms should help find what I want Definitions & synonyms should help find what I want Today it is too expensive to manually classify my scanned paper. E.g. right time meta-data is critical! Today it is too expensive to manually classify my scanned paper. E.g. right time meta-data is critical! Next year we hope the system can classify papers and other documents e.g. bills Next year we hope the system can classify papers and other documents e.g. bills In 10 years we expect all documents to appear electronically & classified with a little help from me In 10 years we expect all documents to appear electronically & classified with a little help from me

Using my life bits: Ontologies… useful? or fools errand? #8: ontology??? Succumbing to the ontology fallacy -Bates

Company 1 1.Generic organization: Correspondence, financial, manuals, notebooks, org chart, plans, products, stocks, etc.. Facets: doc type, dissemination, institution type 2.Generic org. plus projects x roles; facets: financial; legal 3.Generic organization for club, foundation, museum, professional org, religious, sport, etc. 4.Books, CDs, papers, videos Facets: media type Employer 2 Non-profit 3 Library 4 HealthLegal Organizations Academic Inst. 2 Financial Assets Family & related social Ancestors, Parents, Siblings Media ArtifactsComm. Library & archives: info & records. Personal archives (Ambiance…) Children Spouse | Significant Other Friends Articles, bio, books, interviews, talks, …web pages Auto, home& other things Property Diaries Family Business 2 Self Family ($,property, legal, health) potentially private … Institution type: academic,… companies, family, other organizations…self vs. complex contact??

MyLifeBits: Some Lives(t) Personal Personal Parents, children, grandkids Parents, children, grandkids CGB himself CGB himself GKB GKB SSF SSF Close friends Close friends GB $s; Legal entities GB $s; Legal entities Personal incl. several legal structures Personal incl. several legal structures Properties: autos, real estate, Properties: autos, real estate, Investments & contracts Investments & contracts Past prof. companies/organizns Past prof. companies/organizns DEC DEC Carnegie-Mellon U. Carnegie-Mellon U. DEC, NSF, Encore, Ardent, Me Inc., Bell-Mason DEC, NSF, Encore, Ardent, Me Inc., Bell-Mason Bell-Mason Director Bell-Mason Director Diamond & Vanguard Brds. Diamond & Vanguard Brds. Startups & boards Startups & boards Microsoft Microsoft MLB MLB Clusters Clusters Telepresence Telepresence WWW presence WWW presence Computer History Museum Computer History Museum BOD member BOD member Fund-raising Fund-raising CyberMuseum CyberMuseum

F: father F: mother F: self F: Sister Education F: spouse F: son F: grandChild F: daughter F: grandchild F: Significant Other W/Education Work Organization

Roles & Institutions? Versus an elaborated contact I …. I Brigham, Laura I Brigham, Laura I MIT I MIT I DEC I DEC I ACM ACM, IEEE,… NAE I Computer Museum… Dust Inc.

Things Can everything be part of the model? Can everything be part of the model? Family Family Companies Companies Other organizations Other organizations Pets Pets Houses Houses Cars Cars Assets Assets

Using my life bits: Providing insight, including… Where did I spend my time? What has been by output? #9: logging & reports

Interface to xls

TV Usage

MyLifeBits Log of a video file

Using my life bits: Recording everything! #10: CARPE Continuous archival recording of personal experiences

The A/V/real time data Future: new capture modes/devices SenseCam Deja View Body Media Quindi

Sensecam & Interactive jewellery

Open Problems

The Agenda for the Tbyte(s), Lifetime, PC: The killer app after office and mail.searching 1. Guarantee that data will live forever! dear appy problem 2. Cheap, easy, and data-rich (e.g. time, place) capture: GPS and time everywhere Paper capture has to be as easy as discarding (scanner/shredder) Personal meeting capture...perhaps by the room E-book…e-magazines & journals need to have critical mass! Telephony and audio capture with indexing (telephonic speech-to-text needed) Media Center compatible for entertainment (photos, video, TV, radio) 3. Content analysis (critical for photo & video!); doable for text. 4. Information control: privacy, security, expunge/deniability,… 5. Having to be schizophrenic or have a lobotomy when leaving a life or being a part of some other persons life recording 6. One dbase for everything (articles, books, conversations,... financial transactions) …vs. long-term use of hierarchical files. Is dbase intuitive? 7. Annotations/meta-information add every-increasing value at high cost! Easy annotation for aiding search and it becomes the content 8. Other killer apps: Alzheimer, immortality, surrogate memory? 9. GUIs to improve use (e.g. time to learn, use, aid in retention)

The dear appy problem Dear Appy, How committed are you? Please come back to me. Forever yours truly, Lost and forgotten data Whos responsible? Whos responsible? Media or 8 track cassette, 8 floppy Media or 8 track cassette, 8 floppy Evolving platform, file, and database Evolving platform, file, and database Evolving, incompatible standards & formats for legacy data that disregard ancestors Evolving, incompatible standards & formats for legacy data that disregard ancestors Evolving and/or disappearing apps Evolving and/or disappearing apps

A Storocratic Oath 1. Do no harm to dates (File creation, Photo taken) 2. Do no harm to device created & other meta-data. Camera data & location data are sacred. Camera data & location data are sacred. 3. Support & aid the creation of critical meta- data. When/how the user feels like it When/how the user feels like it Auto-magically! Auto-magically! 4. Maintain user confidentiality

The killer app?? Input, File, Classify, and Find… Input, File, Classify, and Find… Operational Operational Stuff Ive Seen (e.g. msg, name, paper, fact, birthday, phone call, photo Stuff Ive Seen (e.g. msg, name, paper, fact, birthday, phone call, photo Observe every action… Observe every action… Time & motion (routing, communicating, scheduling … thinking) Time & motion (routing, communicating, scheduling … thinking) Archival ones self Archival ones self Finder aka Table of Contents aka Site Map Finder aka Table of Contents aka Site Map Story telling. Story telling. Screen saver & personal ambience Screen saver & personal ambience