1. 1 DSP Ground Data System, Data Processing and Data Management (For Discussion) Jiankui Shi Chinese DSP Data Center Centre for Space Science and Applied Research, Chinese Academy of Sciences 11/14/2002 GRAZ

2. 2 Content I. Introduction II. Chinese Ground Data System III. VPN link between China and ESA IV. DSP Data Processing V. Chinese DSP Data Centre VI. Software, Testing, Management and Schedule

3. 3 I. Introduction 1. Concerning Acronyms CDDCChinese DSP Data Centre DDMSDSP Data Management System DOMCDSP Operations and Management Centre DODASDSP Orbit Design and Analysis Subsystem DPSSDSP Payload Science Sub-system DSASDSP Science Application System DSDSDSP Science Data System DSOCDSP Scientific Operations Centre DSSSDSP Science Strategy Subsystem EDSDCEuropean DSP Science Data Centre EPOSEuropean Payload Operations Service SDPStandard Data Products

4. 4 2. DSP Data Levels Level 0 Raw telemetry data received at the ground station Level 1 Instrument raw data files, time-tagged in UT, sorted in time, and containing one calendar day of data from 00:00 UT to 24:00UT. There are separate files for each instrument and various auxiliary data. This level corresponds to the data on the Cluster raw data CD- ROMs. Level 1p The L1p data are for instrument health checking. L1p data file is of the same as the L1 data but for each groundstation pass, not for one calendar day. L1p covers all period as the coverage of the L0 to produce it (TBD). It is also time-tagged in UT, sorted in time and instrument-specific. Level 2 Calibrated high resolution data Level 3 Standard data products of the DSDS, mainly PP and SP data. The PP and SP data are validated data.

5. 5 3. Auxiliary Data Long Term Orbit File (LTOF) Short Term Orbit File (STOF) Long Term Event File (LTEF) Short Term Event File (STEF) Spacecraft Attitude and Spin Rates (SATT) Time Calibration File (TCAL) Command History File (CMDH) Housekeeping Parameter Definition (HPD ) 4. DSP Standard Data Products PP (Prime Parameter, spin-averaged, for each spacecraft) in CDF SP (Summary Parameter, minute-averaged, for each spacecraft in CDF Aux (Cluster-like EPOS Parameters, e.g. predicted event times ) in CDF

6. 6 II. Chinese Ground Data System The bodies concerning the DSP data processing and management in China are DSAS and DOMC. DSAS DOMC DSOC DPSSDODAS DSSS CDDC The bodies concerning the DSP data process

7. 7 1. DOMC (DSP Operation and Management Centre Concerning data process, DOMC will be responsible for: (1) to receive the L0 data from the Groundstations those are Beijing station, Shanghai Station and VILSPA station. (2) to generate L1p data and transfer them to DSOC for instrument health checking. (3) to generate L1 data and transfer them to DPSS for shipment to data center(s). (4) to generate hostkeeping data, some Auxliary data files ((SATT, TCAL, CMDH, HPD) ), and others. The responsible person in DOMC is Mr. Li Xuzhi / Mr. Meng Xin.

8. 8 2. DSAS (DSP Science and Application System) In DSAS, the each sub-bodies responsibility concerning data process are as follows. (1) DSOC (DSP Scientific Operation Center) will be responsible for: (A) to put the L1p on network for PI to check the instrument health (B) others The responsible person in DSOC is Dr. Shen Chao. (2) DODAS (DSP Orbit Design and Analysis Subsystem) will be responsible for: (A) to generate some Aux. data files (LTOF, STOF, LTEF, STEF). (B) others The responsible person in DODAS is Mr. Li Qiang.

9. 9 (3) DPSS (DSP Payload Science Sub-system) will be responsible for: (A) to make completed CD-ROM with L1 and Aux. data files, and to ship the CD-ROM to data center(s). (TBC) (B) to put L1 and aux. files on network for transfer to EDDS (TBC) (C) to generate the plot files for Chinese instrument science parameter for Quicklook. (D) the Chinese instrument science data calibration and others The responsible person in DPSS is Dr. Chen Tao. (4) CDDC (China DSP Data Center) will be responsible for: (A) to produce and validate the PP/SP for chinese instruments (B) SDP production and exchange with European side (C) SDP data dissemination within China (D) Web-based quicklook data facility (E) Data Archiving (including L0,1,2,3, and Aux. Data) (F) Data Management tasks and data serve. The responsible person in DPSS is Dr. Shi Jiankui. (5) DSSS (DSP Science Strategy Subsystem) will be responsible for scientific research. The responsible person in DSSS is Prof. Z. X. Liu.

10. 10 III. VPN link between China and ESA We suggest the VPN link as follows. VPN DSOC DPSS CDDC VILSPA ADDC EPOS VPN link between China and ESA

11. 11 The tasks of the VPN link suggested as: (1) DSOC will use the VPN link to connect VILSPA for the level 0 data received in VILSPA Ground Station for EDDS with the level 1p for instrument health checking by ePIs (TBC) (2) DPSS will use the VPN link to connect EDDC for level 1 data exchange to connect EDDC for level 2 (high resolution) data exchange(TBD) (3) CDDC will use the VPN link (TBD) to connect ADDC for level 3 data exchange to connect ADDC for Quicklook data exchange (TBD) The capacity of the VPN link (TBD)?The VPN link tasks are ok (TBC)?

12. 12 L0 L1p L1 L2 L3 AUX QLF CDDC (Quick-look, Dissemination, DDMS) DSOC DPSS CAL for China ADDC (Quick-look, Dissemination, DDMS) ePI QC, CAL DOMC VILSPA Beijing Shanghaii VPN cPI QC DODAS VPN IV. DSP Data Processing 1. Procedure of the DATA processing

13. 13 (1) Level 0 data (a) L0 data ( the telemetry data) will be received by DOMC from the ground stations in China. For the L0 data from VILSPA, DSOC will get it by the VPN link and transfer it to DOMC in as soon as possible. (b) After processing, DOMC will transfer the L0 data to DSAS. (2) Level 1p data (a) The processing of raw telemetry (L0) data into the equivalent of DDS data (L1p) will be done by DOMC. The DOMC will send the L1p to DSOC In 12 hours after each groundstation pass or receiving the L0 from DSOC. DSOC will make the L1p available for PI to access by the VPN link and check the instrument health as soon as possible (maximum within 24 hours after the downlink in Beijing or Shanghai Station). (b) The L1p data includes instrument science data and housekeeping data. (c) The PIs use the L1p to quick check his/her instrument health as soon as the L1p is available.

14. 14 (3) Level 1 data The responsibility for production of the daily raw data files (L1) is DOMC. DOMC will generate the one-day file of L1 (BJ0800-BJ0800) and transfer it to DSPP in 24 hours after the last BJ 0800. As soon as the L1 file is completed and all other necessary files for the CD-ROM are available. The DPSS will produce the daily raw data CD-ROM (L1). One CD-ROM will contain one-day data for all instruments (more than one day TBD). The DPSS will deliver the L1 data that are the same as in the CD-ROM (L1) by the VPN link between ESA and China as soon as possible, and mails the CD-ROMs (L1) once a week (number is TBD) to EDDC (TBC) and calibrates the data for Chinese instruments. (4) Level 2 Data ( High resolution data ) The responsibility for calibration the L1 data to produce the L2 data for European instruments is with the PI, for Chinese instruments it is DPSS. The format of the L2 is TBD. The L2 data will be sent back to CDDC in 2 months (TBD) after received the L1 data. European PIs should provide the high resolution data and updates thereof to CDDC as soon as they become available.. The L2 data will be kept in CDDC, access authority to the L2 data will be strictly restricted.

15. 15 (5) Level 3 Data The responsibility for validation the L2 data is the PI. The PI should validate his/her L2 data in 6 weeks. After validation, in 1 week, Chinese PIs will transfer his validated data to CDDC, and Europe PIs will send the validation files to his/her associated data centre: After receiving the validated data, in one week, the CDDC will produce the PP/SP for Chinese instruments (instrument NUADU is TBD) and the European Data Centres will produce the PP/SP for their associated instruments The Data Centres will exchange the Standard Data Products using the mechanisms provided by DDMS. (6) Auxiliary Data LTOF, STOF, LTEF and STEF will be produced by DODAS and then send them to DPSS as soon as possible SATT, TCAL, CMDH, HPD will be produced by DOMC. These files will be sent to DPSS as soon as possible.

16. 16 (7) Quicklook plot files Quick-look data plot files for European instruments, except NUADU, will be produced in Europe. DPSS will produce the Quick-look data plot files for Chinese instruments and NUADU, then transfer them to CDDC as soon as possible. The quicklook plot files between European and Chinese facility will be exchanged by CDDC and ADDC within three days. A full collection of data products from the DSP mission will be held in CDDC, including level 0, 1, 2, 3, and Aux. data.

17. 17 V. Chinese DSP Data Centre 1. Description of CDDC Task The CDDC main tasks are: Definition for data level including auxiliary data (contents of SDP to be agreed within IWG) SDP production and exchange with Europe, dissemination SDP data with in China Web-based Quicklook data facility Data serving with DSP Data Management System Support for data system development and implementation with Europe Data Archiving Contact point for data issues between Europe and China Data Management tasks, DSP homepage, News or information

18. 18 2. Detailed Task Description (1) Definition for data level including auxiliary data The data formats used in the DS PROGRAM shall in general be identical to the Cluster data formats. However, due to different boundary conditions and other constraints, limited modifications are envisaged. The baseline for L1 data is the format described in the Cluster DDID. Key elements are the use of Standard Formatted Data Units (SFDU), also for off-line data delivery, and the use of standard telemetry packet headers. Files to be generated by the DOMC/DSAS include instrument raw telemetry files for each instrument, data type, and spacecraft, with packet headers. Data types include Normal Science Data (NSD), Housekeeping Data (HKD), Auxiliary Data (AUX), Housekeeping Parameter Definition Data (HPD) and Catalogue Data (CAT). The level 1 data CD-ROMs also contain a VOLDESC file and a cumulative index file. The task includes The co-ordination of data formats and naming conventions The definition of interface control documents The verification of the implementation

19. 19 (2) SDP production, exchange and dissemination The CDDC will produce the PP/SP for Chinese instruments (and NUADU, TBD) and the European data centres will produce the PP/SP for European Instruments. Then the two sides data centres will exchange the Standard Data Products. The CDDC will dissemination SDP data with in China (3) Web-based quicklook data facility The web-based interface software for quicklook data facility DSDSWeb (similar to CSDSWeb) will also be in CDDC. DSDSWeb will include panels for all instruments. (4) Data serve with DSP Data Management System CDDC will install the DSP Data Management System for scientific user. The access authority will be strictly restricted on the rule. (5) Support for data system development and implementation with Europe Whereas in the Cluster CSDS the Chinese data centre plays a passive role as recipient of data, the DSDS involves the Chinese side actively as source of data products. Therefore it is necessary for the integrity of the system that the interfaces between Europe and China and the Chinese data ingested into the system comply with defined standards.

20. 20 (6) Data Archiving Archiving of mission related data is a top level requirement. CDDC envisages to establish an archive plan in collaboration with the PIs and ESA. A full collection of data products from the DSP mission will be hold in CDDC, including level 0, 1, 2, 3, and Aux. data. (7) Contact point for data issues between Europe and China In order to facilitate the communication between Europe and China, CDDC envisages to operate as contact for data-related issues in the framework of DSP. (8) Data Management, DSP homepage, News or information issue With Austrian Data Centre, CDDC will responsible for the data management The DSP homepage will be installed in the CDDC host Web-based news and information concerning the DSP and relations will be on the CDDC host

21. 21 3. CDDC Facility Work-Station: DSP data Archiving Level 0-1-2-3 ， Aux data Server 2 ( F4800 ) (Back up of Server 1) Server 1 ( F4800 ) ． DSP homepage ． News or information issue ． Quicklook Web ． DSP Data Management System ． SDP Data exchange ． PP/SP production ． Data dissemination ． Public Data and others RAID ESA/Austrian DC Data Exchange Scientific User CDDC Facility The work station for data archiving: Sun Blade 2000: CPU (900MHz) Memory (4G) HD (73G). The server 1, and 2 are the same: Sun Fire 4800: 4 CPUs (900MHz) HD (36G) Memory (8G). RIAD: Sun Storage T3: 1000G.

22. 22 VI. Software, Testing, Management & Schedule 1. Software Software for the L1 and L1p data, responsibility: Chinese side EDDS with L1p for instrument health, responsibility: EDDC Software to produce Quicklook interface files, responsibility: European PIs for European instruments, CDDC for Chinese instruments. Software for Quicklook system and to produce quicklook plot files, responsibility: TBD Software for the calibration of the data from Chinese instrument, responsibility: Chinese side Software for the validation of the data from Chinese instrument, responsibility: Chinese side Software for the production of the PP/SP files for Chinese instruments (pipeline production software), responsibility: European PIs with their data centres for European instruments, CDDC for Chinese instruments DDMS with all instrument including all Chinese instruments, responsibility: EPOS Software for data archiving, responsibility: Chinese and European sides

23. 23 2. Testing The data from all sub-systems shall flow without problem immediately after launch. In order to avoid difficulties after launch, if possible the complete system should be tested before launch. Ideally, there should be end-to-end tests a few months before launch, so that in case of errors there is enough time for correction and re-test. (TBD) 3. Management In order to ensure a good progress and a good visibility of the situation, several reviews should be held, at least a Design Review, an Implementation Review, and Readiness Review (between now and the launch date in roughly equal time intervals). The schedule and the scope for these reviews are (TBD). 4. Schedule A schedule of all major tasks (TBD).