1. ORION CI 28 Mar 2006 ORION CI Conceptual Architecture Team Progress Report Agenda Context Presentation (Options) Discussion

2. ORION CI 28 Mar 2006 Context

3. ORION CI 28 Mar 2006 ORION Context

4. ORION CI 28 Mar 2006

5. ORION CI 28 Mar 2006 Expanding Role of Environmental Observing From individual expedition to collaborative observation. – Long-term persistent presence, continuous 24/7 time series – Increasing numbers and diversity of sensors – Multiple, complex data streams – Many more types of users and applications – Ability to control instruments, observing systems, and large-scale experiments. Ubiquitous data requiring greater differentiated expertise. – From kilobits to terabytes in a matter of hours. – Collaborative discovery by groups of researchers with different vocabularies and expertise looking at complex data streams. A need for cooperating networks of data/processing/analysis systems that are different than in the past.

6. ORION CI 28 Mar 2006 Team Members Matthew Arrott (UCSD, Chair) Alan Chave (WHOI) John Graybeal (MBARI) Eric Guillemot (NEPTUNE Canada) Ingolf Krüger (UCSD) Benoît Pirenne (NEPTUNE Canada)

7. ORION CI 28 Mar 2006 Team Mandate Develop a set of information system capabilities. Develop conceptual level design options to support the Ocean Observatories Initiative (OOI) systems operations and RFP development. Develop a CI Work Breakdown Structure (WBS). Create resource estimates for each line in the WBS to prepare a Rough Order of Magnitude (ROM) cost and timeline for the project.

8. ORION CI 28 Mar 2006 What do you want us to talk about?

9. ORION CI 28 Mar 2006 Presentation Options How will using ORION be different from the way science is done today? What will CI do for us as individuals and as a community? How has CI been implemented in other environments? What is the current architectural concept? How can we be sure this is a viable, implementable infrastructure? How did you go about putting this together?

10. ORION CI 28 Mar 2006 How will using ORION be different from the way science is done today?

11. ORION CI 28 Mar 2006 Getting A Book: The Way It Used To Be I need to special order a book. What book do you want? It’s by Rachel Carson, Silent Spring. Yes, we can order that. It’s 23.95. When will you have it? Maybe Tuesday, but if it’s not in stock, not for a month. Can you send it to me? No, you’ll need to pick it up. OK, can I pay by phone? Yes, we’ll need your credit card and address. OK, here’s my card info and address… OK, we’ll call you when it’s in.

12. ORION CI 28 Mar 2006 Getting A Book: The Way It Is Now

13. ORION CI 28 Mar 2006 Doing Science: The Way It Has Been Hey Francisco, you still running that mooring in Monterey? Yeah, John, what do you need? Is it collecting current data? Yeah, you looking for profiles? Yup, are you doing that with an ADCP? Yes, we’ll probably start profiling soon, too. No, don’t need that. How do I get the data? You want the raw data or QCd? Real-time?. Oh, the QC’d, delayed mode data, definitely, last 3 years worth. That’d be off our dods site, here’s the address.. Documentation come with that? We got some, I’ll have to get one of my techs to send it to you…

14. ORION CI 28 Mar 2006 Doing Science: The Way It Is and Will Be

15. ORION CI 28 Mar 2006 Doing Science: Another Option

16. ORION CI 28 Mar 2006 Or…

17. ORION CI 28 Mar 2006 Interfacing with ORION Working with the system will be at least as easy as it is now. Building an instrument? –Instrument interfaces are defined –Libraries and adapters are provided to help you. –Core instruments will be taken care of. Discovering data? –Lots of options, most of which actually work. –Just “get the data” (as you want it) once you find it. –Not just ORION data, data from lots of places. Running an observatory? –Access to system status. –Ability to coordinate the entire system as needed.

18. ORION CI 28 Mar 2006 Return to menu Forward to science scenarios

19. ORION CI 28 Mar 2006 A Science Scenario What we have here –An example for illustration –Only one of many applicable science scenarios The example –“Developing a complex instrument for ORION” –Actually a platform, but we’ll call it an instrument here –Covers most of the lifecycle of the instrument And furthermore… –Accessing and using data from the instrument

20. ORION CI 28 Mar 2006 Developing a Complex Instrument for ORION Platform carrying multiple sensors Commandable Not always connected Multiple non-trivial data sets Data transport varies –may be stored and copied out of the instrument later –may be streamed in (near) real time

21. ORION CI 28 Mar 2006 Steps Along The Way

22. ORION CI 28 Mar 2006 Develop: Create Instrument What does my instrument have to do? In any case, I have to get money (may involve research). Finally, I’m building the instrument. ORION just sent me instructions on testing my instrument. What does ORION say my instrument must do? Can I really justify building this? (May call for research.) ORION IO may give me funds (e.g., for core systems)! I guess I’d better test it. Darn, need to fix that./ Yay, it’s ready!

23. ORION CI 28 Mar 2006 Commission: Test and Validate Instrument Does system know about my instrument? Does my instrument talk to system correctly? Put instrument in the water and hook it up. Make sure the instrument is “behaving.” Make sure my instrument is working correctly. Tell system about my instrument. Record the transaction.

24. ORION CI 28 Mar 2006 Publish: Announce Availability of Instrument Declare what is the final version of my instrument Group this with the other instruments of similar products. Tell everyone this instrument is available. Describe my instrument (using ORION forms). Make sure this description is valid and complete. Enter my instrument into the approved list.

25. ORION CI 28 Mar 2006 Discover: Find the Instrument What kind of an instrument am I looking for? Son of a gun, I didn’t know they have a forbitz! Let’s look over here, this list looks relevant. Describe the details I’m looking for. What’s the best way to find it? (How & where?) OK, I’ll browse/ search using this interface. Start the search already. Darn, need to try again./ Yay, found it!

26. ORION CI 28 Mar 2006 Acquire: Get OK to Command Instrument Please let me control the frobnitz (private) instrument. “Authorizing access, please wait.” “Instrument documentation is in the mail.” “Can user control this thing?” Never mind, I didn’t want to use that instrument anyway. Your instrument access list includes: frobnitz ID#2365 Hi, Pat? Jan here. Of course. Do you agree to call the provider now and give credit later? “Sure, if user calls me and credits me.”

27. ORION CI 28 Mar 2006 Use: Command Instrument This is an instrument I can command. I’d better cite that scientist like I agreed to. Thank you for giving me credit on your paper. “set sample rate on ‘instrument 23493’ =10 Hz for 3600 secs” Changing sample rate to 10 Hz for 1 hour. I’m getting the data back, I can start my analysis now. Do I need to do more? OK, don’t need to command it any more.

28. ORION CI 28 Mar 2006 Govern: Restrict Instrument Control Is it still OK to use the instrument (so much)? Define access rights and policies for instrument “Is it, or is it not, OK to use instrument right now? Is it OK for this user to use the instrument?

29. ORION CI 28 Mar 2006 Release: Don’t Need Instrument I am done with this instrument, thanks. “Anyone else need this instrument?” “Is there a default use for this instrument?” “OK, it’s done with you too. Your connection is broken.” “Go back to default schedule.” “Instrument is available.”

30. ORION CI 28 Mar 2006 Retract: Instrument Unavailable Uh-oh, this old instrument model doesn’t meet our standards. Someone is using it, send a warning. No one is using it now, declare it deprecated. Anyone still using it? Provide deployment statistics. No one is using it, declare it deprecated.

31. ORION CI 28 Mar 2006 What About the Data? Return to menu Forward to data access scenarios

32. ORION CI 28 Mar 2006 Publish: Announce Data Available Decide on the final format for data stream(s). Group this data with the other similar products. Tell everyone this data product is available. Describe data stream (using ORION forms). Make sure this description is valid and complete. Enter my stream into the list of available data.

33. ORION CI 28 Mar 2006 Discover: Find the Data What kind of an data am I looking for? Son of a gun, I didn’t know they have carbon data! Let’s look over here, this data set relevant. Describe the data sets I want in detail. What’s the best way to find it? (How & where?) OK, I’ll browse/ search using this interface. OK, it knows me and is letting me see the data set. Start the search already. Darn, need to try again./ Yay, found it!

34. ORION CI 28 Mar 2006 Acquire: Obtain the Data Please give me access to the frobnitz data stream. Yes. “Authorizing access, please wait.” Oh look, the download button’s enabled. Can user have data? Do you accept this data is not quality controlled? Oh, I didn’t want the data if it isn’t Quality Controlled. Click on link, watch data stream in. (done) Sure, if user agrees to disclaimer

35. ORION CI 28 Mar 2006 Use: Modify Data, Present Results This is just data. On we go. I’d better cite that scientist like I agreed to. Thank you for giving me credit on your paper. I’m getting data, my software’s running, I detected upwelling! I write my paper. Can I turn this stream off? Yup, don’t need to access it any more.

36. ORION CI 28 Mar 2006 Return to menu

37. ORION CI 28 Mar 2006 What will CI do for us as individuals and as a community?

38. ORION CI 28 Mar 2006 For the individual Provide workflow and resource management tools to automate experimental design and execution Provide a secure work environment that automatically mediates conflicts Generalize and automate query and publish/subscribe processes for retrospective and real-time data

39. ORION CI 28 Mar 2006

40. ORION CI 28 Mar 2006 Dynamic Data Driven Assimilation System (DDDAS) use case Workflow binds diverse fixed and mobile instruments to assimilation model and data repository on shore Resource conflict mediation is automated

41. ORION CI 28 Mar 2006

42. ORION CI 28 Mar 2006 Distributed remote multi-mission laboratories distributed on an RCO Resource intensive, shared use operations require automated brokering Quality of service policy constraints are important

43. ORION CI 28 Mar 2006 For the community Facilitates collaborative experimentation and communication Automated integration of ORION policies and external obligations Ability to archive data with reliable discovery services for future use

44. ORION CI 28 Mar 2006

45. ORION CI 28 Mar 2006 Automated tracking and coordination of the state of observatory resources External resources can be integrated with observatory assets and operations Science use case becomes the observatory use case

46. ORION CI 28 Mar 2006 Return to menu

47. ORION CI 28 Mar 2006 How has CI been implemented in other environments?

48. ORION CI 28 Mar 2006 CI: Evolved from Grid Computing Grid Computing is a term for loosely coupled distributed computing across a diverse community of resources, owners Many Grid Computing efforts and products –Computational Grid (TeraGrid, Open Science Grid) Decomposition of large problems into many small atomic tasks –Data Grid (Storage Resource Broker, OGSA-DAI) Federation & cataloguing of distributed data repositories –Service Grid (On-Demand and Location-based service models—IBM, Microsoft) Generalization of computational and data grids as “Service” patterns Emergence of Service Oriented Architectures –Autonomic Grid (DMTF, WSDM from HP, IBM, CA, Oracle, Opsware) Autonomous resource management, load balancing and fault detection isolation and recovery applied to networks of coupled resources Grid computing contributes to overall cyberinfrastructure

49. ORION CI 28 Mar 2006 ORION CI Building on Other CI Projects GriPhyN, Atlas, Ligo, CMS –Data distribution and shared computational grid NVO (National Virtual Observatory) –Community data model, and shared data repositories and applications BIRN (Biomedical Information Resource Network) –Federated data repositories of disparate data models into a common meta-catalog –Resolution of disparate data models through mediation GEON (GEOsciences Network) –Extends data mediation model with ontologies TeleScience and NEES –Developed real-time control of remote instrumentation and the coupling of remote physical and simulated systems

50. ORION CI 28 Mar 2006 Environmental Cyberinfrastructures like ORION What other projects are most like this? –LEAD (Linked Environments for Atmospheric Discovery) –NEON (National Ecological Observatory Network) What makes them similar? –Significant increase in the number and diversity of instruments and data products –Real-time stream processing and inquiry –Interaction with the sensing environment –Semantic heterogeneity (mixed vocabularies)

51. ORION CI 28 Mar 2006 Return to menu

52. ORION CI 28 Mar 2006 What is the current architectural concept?

53. ORION CI 28 Mar 2006 External Architecture

54. ORION CI 28 Mar 2006 External Interaction Model

55. ORION CI 28 Mar 2006 Lifecycle of ORION Activities (v.B)

56. ORION CI 28 Mar 2006 Internal Architecture

57. ORION CI 28 Mar 2006 System Components

58. ORION CI 28 Mar 2006 Internal Interaction Model

59. ORION CI 28 Mar 2006 Deployment Architecture

60. ORION CI 28 Mar 2006 Enterprise Service Bus

61. ORION CI 28 Mar 2006 Example of Activity

62. ORION CI 28 Mar 2006 Example of Activity

63. ORION CI 28 Mar 2006 Return to menu Go to Scenarios

64. ORION CI 28 Mar 2006 Viable Architecture How can we be sure this is a viable, implementable architecture?

65. ORION CI 28 Mar 2006 Reasons for Confidence Open and transparent design process Ongoing review by external and internal experts and stakeholders Iterative functional deployments reflecting use priorities and system risks A rapidly maturing field –Major subsystems have multiple implementations in existence –Science efforts on which to build (BIRN, GEON) –Many sophisticated commercial collaboration architectures, particularly in web domain (Flikr, eBay, MySpace) –Pioneering efforts in ocean cyberinfrastructure (LOOKING on research side, NEPTUNE Canada in operation) Innovation will be focused on increased usability, effectiveness and reliability

66. ORION CI 28 Mar 2006 Concerns Some technical areas deserve extra attention –Incorporation of streaming data, a new data type, into scientific process –Specification and adoption of instrument services –Effective operation of systems of systems that will scale with the anticipated growth of ORION –Interfacing with external organizations and resources Implementing Organizations take this on –Evaluation and response to the most challenging tasks –Collaborative development encouraged to maximize success

67. ORION CI 28 Mar 2006 Return to menu

68. ORION CI 28 Mar 2006 How did you go about putting this together?

69. ORION CI 28 Mar 2006 Methodology & Timeline DoDAF model Development Process –Requirements –Operational activities –Organizational and operational entities Dictionary –Domain model –System components –WBS –Use domain experts for final costing Timeline –May 15, 2006

70. ORION CI 28 Mar 2006 Characterize the System Identify the main areas of Research Activity, Internal/External lines of Operations, and/or Support Functions. Describe the operations that are normally conducted in the course of achieving an experiment or research goal. –Description includes: Capabilities, operational activities, input/output flows between activities, input/output flows from/to activities outside the scope of the architecture Characterize the command, governance, and business relationships among the human, organizations, and organizational type roles that are the key participants/stakeholders in the architecture. Depict the operational nodes, including organizations, with need lines between those nodes that indicate a need to exchange information.

71. ORION CI 28 Mar 2006 Entity Activities

72. ORION CI 28 Mar 2006 Lifecycle of ORION Activities

73. ORION CI 28 Mar 2006 Example of Activity

74. ORION CI 28 Mar 2006 Internal Interaction Model

75. ORION CI 28 Mar 2006 Reference: NOSA Logical Data Model

76. ORION CI 28 Mar 2006 Return to menu

77. ORION CI 28 Mar 2006 Internal Interaction Model v1

78. ORION CI 28 Mar 2006 Concepts of Operations

79. ORION CI 28 Mar 2006 Discussion: Questions, Comments, Suggestions