Cyberinfrastructure Status July, 2011

2 NSF reverse site visit Refactoring and cleanup after review preparations Coordinating Node technology changes (reducing latency, generalizing) Authentication Identity management Access control TeraGrid Compute Node design 2011 – First half

3 Public release of cyberinfrastructure Synchronization, replication, authentication, authorization, resolution, discovery, retrieval Coordinating Nodes UNM, UCSB, ORC Member Nodes KNB, Dryad, ORNL-DAAC, AKN, NBII, CUAHSI, replication targets, Merritt, other Metacat, other Mercury Investigator Toolkit Libraries, Web interface, Zotero, R, Morpho, DataONE Drive Goals for 2011

4 The Major Components Investigator Toolkit Web InterfaceAnalysis, VisualizationData Management Client Libraries JavaPythonCommand Line Member Nodes Service Interfaces Bridge to non-DataONE Services Data Repository Coordinating Nodes Service Interfaces Object StoreIndex Resolution RegistrationReplication Discovery Identifiers MonitorPreservation Catalog IdentityAuthen

5 Investigator Toolkit Software SearchPortalR ClientMorpho Client Libraries JavaPythonCommand Line Member Node Software Metacat Coordinating Node Software Service Interfaces Object StoreIndex Software Delivered at Public Release ZoteroDataONE FS(Excel) Dryad GMN(CUASHI) (Merritt) PreservationMonitor CatalogIdentifiers ReplicationDiscovery ResolutionRegistration Mendeley … DataONE Service Programming Interface (SPI)

6 Tier 1: Public read, synchronization, search, resolve Tier 2: Read with access control Tier 3: Write using client tools Tier 4: Able to operate as a replication target Some Member Nodes with Tier 4 support and corresponding Coordinating Node functionality required for 2011 Member Node Tiers

7 NodeTierStatusPriorityRelease KNB ORNL DAAC Dryad2 ? CUAHSI AKN Merritt ? Fedora? Repl. Targets (GMN) TeraGrid Mercury based ? Metacat based ? Member Nodes Status: 0 = Not implemented, 1=partial, 2=functional, 3=complete Priority:1 = High, 2= medium, 3= low

8 Coordinating Node Services ServiceTierStatusPriority Node registry121 Object storage132 Identifier resolution132 Synchronization111 Discovery111 Log retrieval102 Identity mapping222 Session state222 Identifier reservation333 Replication402 Status: 0 = Not implemented, 1=partial, 2=functional, 3=complete Priority:1 = High, 2= medium, 3= low

9 Coordinating Node Components cn_service cn_restcn_rest_proxy Metacat identity_ manager nodes_ registry synchro- nization replic- ation indexer SOLR Postgres JMSJMS LDAP Mercury Search

10 ComponentTierStatusPriorityRelease Libclient_python Libclient_java R plugin2 (3) D1Drive - FUSE D1Drive - Dokan D1Drive - WebDAV Mercury UI Morpho Excel Zotero, Mendeley Investigator Toolkit Status: 0 = Not implemented, 1=partial, 2=functional, 3=complete Priority:1 = High, 2= medium, 3= low

11 CollectAssureDescribeDepositPreserveDiscoverIntegrateAnalyze Data Lifecycle 11 Morpho

12 Other Pieces ComponentStatusPriorityRelease Documentation Member Node test service Integration testing Monitoring Hardware Continuous build2 Issue tracking3 Repository3 Conferencing2 Status: 0 = Not implemented, 1=partial, 2=functional, 3=complete Priority:1 = High, 2= medium, 3= low

13

14 Infrastructure will change over time. Message between services Behavior of services, components Message semantics How to accommodate change while maintaining operations? Versions

15 Infrastructure Version: Tag identifying the overall functional capabilities, types, messages supported by the DataONE infrastructure Component Version: Tag of a specific component indicating the current revision of the component Example: Version XXX of Metacat supports version YYY of the DataONE infrastructure Infrastructure + Component Versions

16 1.Changes to Types Schema 2.Update test data 3.Build libcommon Pyxb / JIBX 4.Propagate changes through libclient 5.Propagate changes through remainder of components 6.Deploy CN test 7.Update test Member Nodes 8.Deploy staging 9.Update / synchronize content to match production 10.Switch staging and production Where to update all Member Node stacks? Where to update all Investigator Tools DataONE Infrastructure Release

17 Release Coordinating Nodes Touches everything deployed – MNs, ITK, monitor,... Likely to require content migration, synchronization Expensive Update Member Nodes Vendor driven updates (e.g. core changes to Metacat) DataONE driven updates (e.g. access policy fix) Known contact points for updates Update client tools Automated version checks? How to notify users? How to notify third party developers? Component Updates

18 For an infrastructure upgrade: What is the acceptable latency between change announcement and infrastructure back in sync? Should DataONE maintain old versions of service interfaces? If so how many revisions? Some changes may not be backward compatible Option to make old version CNs readonly while content migrated to new systems? Version Latency and Acceptable Delta

19 New package available with clear indication of affected components, signed off by component developers Affect CNs: Install on staging, process metadata updates, lock production, switch. Affect MNs: Notify instance administrators of update Install updates and verify operation Registry picks up updated services Affect ITK: Post update information Notify active clients of update Notify third party developers of changes Administrative Process for Updates

20

21 July New hardware deployed at UNM, UCSB, ORC CN development with focus on Auth and synchronization Integration test services August CN development focus on indexing, replication Deploy CN staging to new hardware ITK libraries updated Full Tier 1 functionality deployed September Identity mapping, authentication, authorization Evaluation of Tier 1 functionality Support Tier 2 operations October Support Tier 3 and 4 operations on specific Member Nodes CNs support replication, load testing of CNs Evaluation of Tier 2 and initial tier 3 support November Final evaluation of functionality Incorporate or schedule suggestions Final stress and usability testing December Final documentation, testing Announcement of initial release General Timeline

22