EnsemBlue: Integrating Distributed Storage and Consumer Electronics Daniel Peek and Jason Flinn University of Michigan

2 Bringing Distributed Systems Home Distributed systems ignore Consumer Electronic Devices (CEDs) Much user interaction is through CEDs Integrate CEDs into distributed systems

University of Michigan3 CED Integration Difficult Application-specific resources Heterogeneous interfaces Closed platforms –Cannot execute our code

University of Michigan4 Today

University of Michigan5 Hypothesis: Distributed file systems can help manage CEDs and multimedia EnsemBlue My Data My Devices

University of Michigan6 EnsemBlue Based on BlueFS [Nightingale 04] –Client-server design –Single namespace –Supports mobile clients –Designed for small group of users File Server

University of Michigan7 Outline Motivation Device-Specific Namespaces Device-Specific Functionality Disconnected Collections of Devices Evaluation Conclusion

University of Michigan8 Device-Specific Protocol Connecting CEDs CEDs cannot perform DFS protocol DFS Protocol? Distributed File System Device-Specific Protocol Attach: computer speaks for CEDWorks with disconnected clients DFS Protocol

University of Michigan9 Device-Specific Namespaces User prefers one organization CED requires another organization Classical Jazz F01 F02 F03 F04

University of Michigan10 Where to store the mapping? Device-Specific Namespaces Cannot store in DFS due to partial caching Make CEDs self-describing

University of Michigan11 Receipts What should mapping look like? Object F2 ~dpeek/classical/song.mp3/iPod/f27/ABCD.mp3 Path lookups may fail with partial caching Receipts bypass path resolution

University of Michigan12 Device-Specific Execution Updating indexes Transcoding Type-specific caching Organizers

University of Michigan13 Device-Specific Execution Cannot execute on CED Leverage general computers –Take action when files change Problem: Need notification of file changes

University of Michigan14 Notification Via Persistent Queries Don’t need new mechanism Leverage cache consistency mechanism –Structure notifications as file system object –Robust to crash –Handles disconnected operation updatestruncate readprocess *.mp3

University of Michigan15 MP3 M4A Persistent Queries Example: M4A to MP3 transcoder File Server M4A to MP3 Transcoder *.m4a M4A MP3 *.m4a MP3

University of Michigan16 Disconnected Devices Server gives safety, availability, consistency Server mediates communication File Server

University of Michigan17 Disconnected Devices Disconnected devices cannot interact! File Server

University of Michigan18 Disconnected Devices Peer-to-Peer better supports mobility CEDs store and forward updates File Server

University of Michigan19 Ensembles Ensembles can avoid these problems Select a pseudo file server –Consistent view of all files –Improved data access –Update propagation Pseudo File Server ? ?

University of Michigan20 Outline Motivation Device-Specific Namespaces Device-Specific Functionality Disconnected Collections of Devices Evaluation Conclusion

University of Michigan21 Microbenchmarks Persistent query execution is fast –<200 ms for 1,000s of matches, GBs of data Persistent queries have low overhead –Negligible even for 10,000 queries Ensemble formation as fast as 20 ms –Proportional to data transfer size

University of Michigan22 Organizing My Photos Photo Organizer iCal *.jpg File Server

University of Michigan23 Managing My Music iPod DB Updater MP3 DB MP3DB MP3DB *.mp3 File Server MP3

University of Michigan24 Conclusion Added CEDs to distributed systems Device-specific data organization Device-specific functionality Supports TiVo, iPod, digital cameras, media players, cell phones, PDAs

University of Michigan25 Questions ?