Curation Micro-Services “It’s a Series of Tubes” Curation Micro-Services “It’s a Series of Tubes”

The Unix philosophy “Make each program do one thing well” “To do a new job, build afresh rather than complicate old programs by adding new features” “Expect the output of every program to become the input to another, as yet unknown, program” “Design and build software … to be tried early” “Don't hesitate to throw away the clumsy parts and rebuild them” — D. L. McIlroy et al., “Unix time-sharing system forward,” Bell System Technical Journal 57:6, part 2 (1978): 1902

The micro-services “philosophy” CC CC

Curation micro-services MetaphorsAssumptionsPrinciplesPreferencesPractices Pipeline Safety through redundancy Modularity The small and simple over the large and complex Focus on outcomes, not means Lego bricks Meaning through context Granularity The minimally sufficient over the feature laden Complexity through composition, not addition Utility through service Orthogonality The configurable over the prescribed Policy neutral, platform and protocol independent Value through use (and reuse) Emergence The proven over the (merely) novel Approach sufficiency through incrementally necessary steps Stewardship is a relay Evolution Early prototyping, frequent refactoring ParsimonyCode to interfaces

Curation micro-services ModeFocusValueServiceValenceVisibility Curation Value Accretion Annotation UI / Access control / Message queuing Interoperation User-facing Visibility Notification Utility Accessibility Access Application Derivation Transformation Selectivity Search Actionability Index Stewardship Ingest Preservation Context Epistemology Characterization Interpretation Provider- facing Ontology Inventory State Reliability Replication Protection Fixity Stability Storage Identity

Design goals Principle of least surprise Multiple interface modalities – RESTful HTTP – Command line – Procedural (Java, Perl, Ruby, …) Linked data Stable URL references The file system is the database State or content Storage node ObjectVersionFile default/1234/3/xyz state/ Storage service

“You say micro, I say macro…” Access ANVL ARK BagIt CAN Checkm Dflat ERC EZID GhOST Ingest Inventory LockIt N2T Namaste Noid Pairtree ReDD RUU Storage Access ANVL ARK bagit.plBagIt CAN checkm.plCheckm Dflat ERC EZID GhOST Ingest Inventory LockIt N2T namaste.plNamaste Noid Pairtree ReDD RUU Storage ServiceToolConvention

Development roadmap First waveSecond wave Third waveFourth wave Fifth waveSixth wave IdentityInventoryIndexSearchNotificationAnnotation StorageIngest / AccessFixityReplication CharacterizationTransformation IDm / Authn / AuthzMetadata standards Object / collection modelingSemantic interoperability Policy / business model development

Ingest process flow Submitting user agent Ingest Inventory Storage Node Identity Submit Create identifier Identifier Add version Get version metadata Version metadata Notification Version metadata Get version metadata Add version

Ingest implementation Submitting user agent Submitter Consumer Ingester Storage Queue HTML form Servlet Implicitly multi-threaded Servlet Implicitly multi-threaded Dæmon Explicitly multi-threaded Zookeeper dæmon Job metadata Job payload Submission notification Ingest notification Batch or single object

Questions? silverpipes.jog / firstpresmacomb.org

More information UC Curation Center (UC3) Micro-service specifications Digital curation group UC3 Stephen AbramsErik Hetzner Margaret Low Mark Reyes Perry Willett Patricia CruseGreg Janée David Loy Tracy Seneca Scott FisherJohn Kunze Isaac Rabinovitch Marisa Strong