1. 06/30/2003 1 Data Product Service (DPS) Packaging and Context Dan Crichton Steve Hughes Ron Joyner Chris Mattman Paul Ramirez Peter Shames

2. 06/30/2003 2 Acknowledgements Ed Greenburg Larry Preheim Steve Levoe Tom McVittie Tim M Erik Barkley

3. 06/30/2003 3 Data Product Service (DPS) Packaging and Context DPS definition: a service that provides reliable delivery of well documented data products that can be directly related to science observation requests DPS mission system context –Functions and high level data flows Data packaging specification –Generic approach –Applied to DPS problem –Evolving content during processing Prototype & deliverables Open issues & next steps

4. 06/30/2003 4 Data Product Service Functional Elements Science Sequence Development Acquisition Plan Development Sequence Plan Observation Requests Obs. Definition Obs. Planned Products Obs. Uplink Files Science Mission Planning Objective-Campaigns Instruments Observation sets Integrated Sequence Development Sequence Plan Development Sequence Plan Sequence Uplink Files Activity Requests Activity Definition Activity Planned Products Activity Uplink Files Spacecraft Activities Sequence Execution Activity Sequence Uplink Files Sequence Events Product Creation Product Notification Product Transfer Service Management Service Scheduling Service Possibilities Service Requests Service Schedule Predicts Service Monitoring Receive Reports Capture Data Provide User Views Performance Assessment Commitments Operational Visibility Tracking Services Scheduled Activities (General) Service Reports Service Notifications Performance Reports Telemetry Service Frame Accounting Report Command Service Radiation Reports DSMS File Transfer Svc Telemetry Service Product Accounting/Reports Product Assembly/Delivery Command Service Product Accounting/Reports DSMS Science Product Service Science Product File(s) Assembly Received Product Product Accounting Cataloging Data Ancillary Data Reporting Processing History?? Observation Planning Integrated Obs. Descriptions Observation Templates Product Types Product Type Relationships Product Type Definition Ancillary Specification Engineering Sequence Development Spacecraft Plan Development Sequence Plan Engineering Requests Eng. Definition Eng. Planned Products Eng. Uplink Files Information Infrastructure XML Schemas Interface Specifications Dictionary & Repository svc

5. 06/30/2003 5 Data Product Service Flows Science Sequence Development Acquisition Plan Development Sequence Plan Observation Requests Obs. Definition Obs. Planned Products Obs. Uplink Files Science Mission Planning Objective-Campaigns Instruments Observation sets Integrated Sequence Development Sequence Plan Development Sequence Plan Sequence Uplink Files Activity Requests Activity Definition Activity Planned Products Activity Uplink Files Spacecraft Activities Sequence Execution Activity Sequence Uplink Files Sequence Events Product Creation Product Notification Product Transfer Service Management Service Scheduling Service Possibilities Service Requests Service Schedule Predicts Service Monitoring Receive Reports Capture Data Provide User Views Performance Assessment Commitments Operational Visibility Tracking Services Scheduled Activities (General) Service Reports Service Notifications Performance Reports Telemetry Service Frame Accounting Report Command Service Radiation Reports DSMS File Transfer Svc Telemetry Service Product Accounting/Reports Product Assembly/Delivery Command Service Product Accounting/Reports DSMS Science Product Service Science Product File(s) Assembly Received Product Product Accounting Cataloging Data Ancillary Data Reporting Processing History?? Observation Planning Integrated Obs. Descriptions Observation Templates Product Types Product Type Relationships Product Type Definition Ancillary Specification Engineering Sequence Development Spacecraft Plan Development Sequence Plan Engineering Requests Eng. Definition Eng. Planned Products Eng. Uplink Files Information Infrastructure XML Schemas Interface Specifications Dictionary & Repository svc

6. 06/30/2003 6 XML Packaging for Science/Observation Data Objects* An end-to-end data product production pipeline requires the packaging of data objects, ancillary data, and documentation to address mission support, science analysis, and data archive requirements. This task proposes the development of a generalized standards specification to: –describe the data objects, ancillary data, and documentation required to support science planning, mission operations, science analysis, and the packaging and cataloging of an archive quality packaged data product –prescribe the process used to produce a packaged data product. * A science data object is defined as any digital data file(s) collected by a science instrument or compiled by a scientist that has science content. A data product is a searchable, retrievable, and orderable package consisting of one or more data objects.

7. 06/30/2003 7 XML Packaging for Science/Observation Data Objects XML will be used to specify, collect, and validate package content A simple prototype will demonstrate the ability to create, search, and retrieve data product packages.

8. 06/30/2003 8 Core Data Product Service Data Product Creation DOM Transaction Product Metadata Product Accounting Observation Product Inst. State S/C State Traj State Product Accounting AMMOS’ CFDP Service Transaction Product N.B. Multiple instr Multiple modes Multiple products All in XML Anc & Eng Data Spec Meta Data Specs Input Data Spec DPS Process Spec Data Product Spec Eng & Anc Data Observation File Obs tag DPS tag S/C Data Sources Standard Repository Product Retrieval Product Search Users Data Dictionary

9. 06/30/2003 9 Core Data Product Service (in Context) Data Product Creation DOM Transaction Product Metadata Product Accounting Observation Product Inst. State S/C State Traj State Product Accounting AMMOS’ CFDP Service Transaction Product N.B. Multiple instr Multiple modes Multiple products All in XML Anc & Eng Data Spec Meta Data Specs Input Data Spec DPS Process Spec Data Product Spec Eng & Anc Data Including Observation Tag S/C Data Sources Operational Repository Product Retrieval Product Search Users Data Dictionary Science Campaign Spec Instrument Spec Observation Template ( multi-inst, multi-product,) Observation Requests Observation Set (Sequence) Expected Data Products Observation tag Science Processing Metadata Including Observation tag

10. 06/30/2003 10 Prescriptive XML Package Schema Package Specification –Structure/format of package components –Content of package data object(s) data object accounting data spacecraft, instrument, and target state information ancillary data including but not limited to –calibration data –navigation data other useful documentation and metadata

11. 06/30/2003 11 Product Service XML Schema Package Specification –Structure/format of package components –Content of Delivered Science Data Product package received telemetry data product/object(s) received data product accounting data ancillary data including but not limited to –spacecraft, instrument, and target state information from S/C engineering telemetry –calibration data –navigation data cataloging information extracted from observation request metadata –observation type and template –campaign associations processing history other useful documentation

12. 06/30/2003 12 Post Processing XML Schema Package Specification –Structure/format of package components –Content of package Spacecraft and ancillary files (see Product Service XML Schema) Process Specifications –Preprocessors –Engineering Analysis –Science Level Processors –Science Analysis and Quality Reporting –Archive Quality Packaging –Subscription, Notification, and Distribution Control Specifications –Process monitoring and control –Instrument Commanding –Data Verification Metadata providing identification, descriptive, relational, and location information for each package component

13. 06/30/2003 13 Information Infrastructure XML Schema Data Dictionary Operational Repository APIs for program / subsystem integration Infrastructure is used by all subsystems for … –Access / location of schema, other resources –Reference to data dictionary for interoperability –Optional storage of instances of XML schema –Optional exchange of data and state information

14. 06/30/2003 14 Deliverables XML schemas Active data dictionary Java class library that provides basic get and put functions for all components of the package Profile server that provides a standard system level search interface for packaged products within a standard repository Product server that provides a standard system level interface for the retrieval of the identified package from the repository Prototype Web interfaces for: –Data Dictionary –Product Service –Monitor/Performance Reports –Schema Display and Population –User Viewpoint Web Page Construction

15. 06/30/2003 15 Issues / Questions Observation: existing huge chasm between uplink and downlink What are driving scenarios / requirements? Is this just for MRO or does it meet a broad variety of JPL (and other) mission needs? Do packages embed raw data or point to it, or both? What about calibration data? Which parts of this need standardization? How complete a view of the context do we need, is Nav / predicts included? Packaging is suitable for more than just science / observation data objects What is nature of “Observation Tags”, how are they created and tracked, how do they handle the one-many, many-one problems? How is processing history handled, are program artifacts archived w/ data, referenced, or just named? What kinds of information infrastructure are required, what technologies, how many instances of what? Are DPS needs for info infra different from ICIS, SMS, monitoring or other DSMS needs? Is “data dictionary” just that or does it relate to an ontology or taxonomy? How is it created and maintained? What is relationship between DPS, monitoring, and data accountability? Are they separate but related?

16. 06/30/2003 16 Next Steps Develop representative science / mission scenarios (DC, SH, EG) –Orbiter –Observatory –Rover Evaluate approach vs scenarios (team) Ensure handling of typical sequence operations vs MDS goal oriented process (TM, EG) Develop User Viewpoint Use Cases, at least … (DC, SH, EG) –Science team –MOS or S/C team –DSMS operations team Update prototype plan to match agreed scenarios & Use Cases