1. A Micro-Services-Based Approach for Curation and Preservation Solutions Stephen Abrams Patricia Cruse John Kunze Perry Willett University of California Curation Center California Digital Library Global Oracle PASIG User Group Meeting, Redwood Shores, May 10-12, 2011

2. Who are we?

3. What do we do? Innovative curation solutions (preservation and access) for the UC community and external partners – EZID, Merritt repository, Web Archiving Service (WAS) – Guidelines and best practices – Integration with other CDL programs (e.g., OA publishing, special collections) and external initiatives (e.g., DataCite, DataONE, HathiTrust, etc.)

4. Thats simple, right? Ever increasing number, size, and diversity of content

5. Thats simple, right? Ever increasing number, size, and diversity of content Expanded use by new constituencies Information Center for the Environment Minnesota Digital Library

6. Thats simple, right? Ever increasing number, size, and diversity of content Expanded use by new constituencies Ongoing (potentially disruptive) changes in technology and user expectation Increasing obligations, shrinking budgets http://www.flickr.com/photos/krbuchholz/3278516200/http://www.flickr.com/photos/mildlydiverting/32286893

7. Merritt repository How can I meet the data management requirements of my grant? I know my desktop content is at risk; what should I do? Whats a good way to share the data underlying a recent publication? How can I ensure persistent availability?

8. Merritt repository Preservation back-end for existing discovery services Dark archive for preservation masters Integration with distributed data grids Bright archive for preservation and end-user access

9. Managerially friendly

10. Model free application/msword 342.5 KB

11. Strongly versioned

12. Easy submission

13. Merritt micro-services Merritt is built from a micro-services toolkit –IdM/Authn/Authz LDAP –Persistent identifiers EZID –Persistent storage CAN/Pairtree/Dflat/Checkm/ReDD –Fixity Fixity –Replication Replication –Catalog Inventory/4store –Ingest Ingest/Zookeeper –Characterization JHOVE2 –Discovery XTF –Transformation –Notification –Annotation Version 2 GhOST/Shibboleth

14. Micro-service choreography User agent Ingest Inventory Fixity EZID Node Store Node SIPAIP DIP Notification Identifier

15. Micro-services The Unix philosophy Make each program do one thing well To do a new job, build afresh rather than complicate old programs by adding new features Expect the output of every program to become the input to another, as yet unknown, program Design and build software … to be tried early Don't hesitate to throw away the clumsy parts and rebuild them L. McIlroy et al. Unix time-sharing system forward Bell System Technical Journal 57:6, part 2 (1978): 1902 Complex emergent behavior Low barrier, low maintenance, low commitment Policy neutral, protocol/platform independent The file system is the database http://www.flickr.com/photos/oskay/265899811

16. Curious Oysters, http://www.flickr.com/photos/thecuriousoysters/4458657148/ Questions?

17. For more information UC Curation Center http://www.cdlib.org/uc3 uc3@ucop.edu Merritt repository http://merritt.cdlib.org/ UC3 Stephen AbramsMargaret Low Lisa ColvinDavid Loy Patricia Cruse Mark Reyes Scott Fisher Tracy Seneca Alex GenadinikJoan Starr Erik Hetzner Marisa Strong Greg Janée Perry Willett John Kunze Abrams, Cruse, Kunze, & Minor, Curation micro-services: A pipeline metaphor for repositories, Journal of Digital Information 12:2 (2011) Abrams, Kunze, & Loy, An emergent micro-services approach to digital curation infrastructure International Journal of Digital Curation 5:1 (2010): 172-186