Passive Capture & Ensuing Issue for a Personal Lifetime Store Jim Gemmell, Lyndsay Williams, Ken Wood, Roger Lueder & Gordon Bell CARPE Workshop Oct 15, 2004

The Price of Active Capture “When I had my first child, I bought a camera and took many pictures. But eventually I realized I was living behind the camera and no longer taking part in special events. I gave that up – now I don’t have nearly as many pictures of my second child.” “When I had my first child, I bought a camera and took many pictures. But eventually I realized I was living behind the camera and no longer taking part in special events. I gave that up – now I don’t have nearly as many pictures of my second child.”

Passive Capture Make capture passive Make capture passive Let the user experience! Let the user experience! Save a rich record of personal experience Save a rich record of personal experience Possible application as memory aid Possible application as memory aid The challenge: we were already swamped in media… won’t this just make it worse? The challenge: we were already swamped in media… won’t this just make it worse?

Outline What is SenseCam What is SenseCam What is MyLifeBits What is MyLifeBits Demo Demo Conclusion Conclusion

SenseCam

sensecam a photographic memory for everyone Ken Wood & Lyndsay Williams Microsoft Research Cambridge 15 Oct 2004

sensecam Essentially a “Black Box” data and image recorder for the human body Essentially a “Black Box” data and image recorder for the human body Current wearable prototype can be worn for a day and captures up to 2000 images in 12 hours Current wearable prototype can be worn for a day and captures up to 2000 images in 12 hours Image capture triggered by sensors, e.g. motion, light, temperature, people in field of view,... Image capture triggered by sensors, e.g. motion, light, temperature, people in field of view,... Sensor data is also recorded for later presentation, analysis, and correlation Sensor data is also recorded for later presentation, analysis, and correlation

a short walk through Cambridge

Accelerometer measures motion of the device Accelerometer measures motion of the device When any sensor indicates image capture, we wait a few tens of milliseconds to capture image when movement is less than 10 degrees/second if possible When any sensor indicates image capture, we wait a few tens of milliseconds to capture image when movement is less than 10 degrees/second if possible Greatly reduces the number of blurred images Greatly reduces the number of blurred images BeforeAfter image stabilisation

Captures everything regardless of exact camera orientation Captures everything regardless of exact camera orientation ultra wide-angle lens

Sensor readings Corresponding images in rapid serial viewer dealing with sensecam data: MyLifeBits

status and plans 12 devices built; adding audio trigger and GPS 12 devices built; adding audio trigger and GPS But hardware is just the beginning But hardware is just the beginning Trials and studies are the interesting bit Trials and studies are the interesting bit domestic probe domestic probe give it to people and see what they do: personal memory enhancement? family fun? give it to people and see what they do: personal memory enhancement? family fun? clinical trials with memory-loss patients clinical trials with memory-loss patients ethics approval for 20 Cambridge patients over 3 years ethics approval for 20 Cambridge patients over 3 years tourism trials tourism trials Edinburgh Festival, museums,... Edinburgh Festival, museums,... cross-cultural privacy study cross-cultural privacy study UK, China, India, Egypt,... UK, China, India, Egypt,...

fridgecam

MyLifeBits

MyLifeBits is: An experiment in lifetime storage An experiment in lifetime storage Digitizing Gordon Bell’s past Digitizing Gordon Bell’s past Capturing more of his future Capturing more of his future A software system A software system Capture Capture Storage & retrieval Storage & retrieval Organization & annotation Organization & annotation Minimum requirement: fulfill Vannevar Bush’s 1945 “Memex” vision Minimum requirement: fulfill Vannevar Bush’s 1945 “Memex” vision

The guinea pig Has now scanned virtually all: Has now scanned virtually all: Books written (and read when possible) Books written (and read when possible) Personal documents (correspondence including memos and , bills, legal documents, papers written, …) Personal documents (correspondence including memos and , bills, legal documents, papers written, …) Photos Photos Posters, paintings, photo of things (artifacts, …medals, plaques) Posters, paintings, photo of things (artifacts, …medals, plaques) Home movies and videos Home movies and videos CD collection CD collection And, of course, all PC files And, of course, all PC files Now recording: phone, radio, TV (movies), web pages… conversations and meetings to come Now recording: phone, radio, TV (movies), web pages… conversations and meetings to come Paperless throughout ” scanned, 12’ discarded. Paperless throughout ” scanned, 12’ discarded. Only 44 GB, incl. 10 wma, 14 SQL!!! Video: o(100) mov Only 44 GB, incl. 10 wma, 14 SQL!!! Video: o(100) mov

So you’ve got it – now what do you do with it? “A record if it is to be useful … must be continuously extended, it must be stored, and above all it must be consulted” “The difficulty seems to be, not so much that we publish unduly … but rather that publication has been extended far beyond our present ability to make real use of the record” - Vannevar Bush

Everything goes in a database You need all the features of a database (Consistency, Indexing, Pivoting, Queries, Speed/scalability, Backup, replication) You need all the features of a database (Consistency, Indexing, Pivoting, Queries, Speed/scalability, Backup, replication) If you don’t use one, you will find yourself creating one! If you don’t use one, you will find yourself creating one! Files as blobs, also sync with file system for legacy apps Files as blobs, also sync with file system for legacy apps SQL

Entities & Links Annotates Caller in Phone Call Photo of Event Transcludes

MyLifeBits Software MyLifeBits store database Voice annotation tool Text annotation tool Telephone capture tool TV capture tool TV EPG download tool Radio capture & EPG PocketPC transfer tool PocketRadio player Import files MyLifeBits Shell files Legacy applications Browser tool Internet IM capture MAPI interface Legacy client GPS import & Map display SenseCam Screen saver

DEMO

A Storocratic Oath 1. Do no harm to dates (File creation, Photo taken) 2. Do no harm to device created & other meta-data. Camera data & location data are sacred. Camera data & location data are sacred. 3. Support & aid the creation of critical meta- data. When/how the user feels like it When/how the user feels like it Auto-magically! Auto-magically! 4. Maintain user confidentiality

research.microsoft.com/hwsystems

BONUS SLIDES

Collaborators Other MSR Collaborators: Kentaro Toyama, Ron Logan, Steve Drucker, Curtis Wong, Mary Czerwinski, Brian Meyers Other MSR Collaborators: Kentaro Toyama, Ron Logan, Steve Drucker, Curtis Wong, Mary Czerwinski, Brian Meyers Interns: Josh Blumenstock, Evan Salomon, Aleks Aris Interns: Josh Blumenstock, Evan Salomon, Aleks Aris

Memex As We May Think, Vannevar Bush, 1945 “A memex is a device in which an individual stores all his books, records, and communications, and which is mechanized so that it may be consulted with exceeding speed and flexibility” Full-text search, text & audio annotations, and hyperlinks Full-text search, text & audio annotations, and hyperlinks

The AI Challenge: Content analysis & Data Mining “Creative thought and essentially repetitive thought are very different things. For the latter there are, and may be, powerful mechanical aids” – Vannevar Bush Is MyLifeBits just enough rope to hang yourself with? Is MyLifeBits just enough rope to hang yourself with? MyLifeBits must become MyPersonalAssistant MyLifeBits must become MyPersonalAssistant Content analysis and data mining Content analysis and data mining Doc similarity & “clean living” Doc similarity & “clean living” Document meta-data extraction Document meta-data extraction

Moving forward: new capture modes/devices SenseCam Deja View Body Media Quindi

Future work: UI Focus to date on functionality – now we need to make it friendlier Focus to date on functionality – now we need to make it friendlier User studies with SFSU User studies with SFSU

Future work: Visualizations Don't give me a little card image and say, "That's all you've got, because that's what I thought you should want for your virtual shoebox." There have got to be multiple modalities and the designers have to be able to deal with that. … don't metaphor me in, don't give me only one way of looking at things. Don't give me a little card image and say, "That's all you've got, because that's what I thought you should want for your virtual shoebox." There have got to be multiple modalities and the designers have to be able to deal with that. … don't metaphor me in, don't give me only one way of looking at things. -Andy van Dam, Hypertext '87 Keynote Address Next Media Web Scout U. Maryland IN-SPIRE

Dear Appy, How committed are you? Signed, Lost and Forgotten Data Dear Appy, I'm having trouble with long-term commitment -- not on my end, heaven knows, but from the apps that created me and with whom I like to associate. Over time, these pesky apps evolve and they simply don't recognize the data that they once helped create! But, we data progeny -- and there are lots of us -- feel that as our creators, these apps should be responsible for eternal support. But the little problem with recognition isn't the worst of it – sometimes the apps even disappear altogether. I ask you, is it expecting too much for 20-something year old data like me to be interpretable by my app (e.g. Acrobat, DB2, Draw, Eudora, Office, Quicken, or RealNetworks), or am I just associating with irresponsible apps? If things continue on their current path, it seems I will be completely un-interpretable within 20 to 50 years! My apps will move to other platforms, or evolve to be more Internet- or Next-Big-Thing-centric... By Gordon Bell

Classification wish list Download classifications rather than build them Download classifications rather than build them Definitions & synonyms should help find what I want Definitions & synonyms should help find what I want Today it is too expensive to manually classify my scanned paper. E.g. “right time” meta-data is critical! Today it is too expensive to manually classify my scanned paper. E.g. “right time” meta-data is critical! Next year I hope “the system” can classify papers and other documents e.g. bills Next year I hope “the system” can classify papers and other documents e.g. bills In 10 years I expect all documents to appear electronically & classified with a little help from me In 10 years I expect all documents to appear electronically & classified with a little help from me

Personal Search is not Professional or Web search System sees every entry & access System sees every entry & access Everything, not just a professional life Everything, not just a professional life Limited to SIS, not an infinite amount, covers a profession & personal life Limited to SIS, not an infinite amount, covers a profession & personal life Web as seen by search engines MyLifeBits Knowledge breadth e.g. Dewey classification Depth e.g. information item types & coverage Professional user