Swarm PDGS Status & Outlook Antonio de la Fuente & Swarm PDGS team Swarm 7th Data Quality Workshop 24th – 27th October 2017, Delft

Summary Swarm PDGS Status Swarm PDGS Outlook Conclusions

Swarm PDGS Status Swarm PDGS Architecture Swarm PDGS Architecture Evolution (Jan 2017) Swarm PDGS Configuration Processing Baseline Status Swarm Data Access & Dissemination Statistics Swarm Dissemination FTP Structure Change (Jul 2017) Personal credentials for Mission Users (Aug 2017)

Swarm PDGS Architecture The Swarm PDGS functions are distributed among different locations: the Archiving, Processing and Dissemination Facility (APDF) is hosted in Newport and operated from Guildford (UK) by Airbus D&S. In the APDF, satellite science telemetry coming from FOS and auxiliary data are ingested, L1b & L2-Cat2 systematically processed, archived, and disseminated. the Level 2 Processing System (L2PS), also known as SCARF, is a cluster of institutions where scientific Level 2 Cat-1 products are generated, and sent back to the APDF for archiving and distribution. the Support to Operations, E2E performance monitoring, and calibration and data quality assurance subsystems (CVQ) are located in ESRIN All PDGS functions are running without major problems since Launch

PDGS Architecture Evolution completed The architecture of the Swarm PDGS is always evolving to respond to new operational and science requirements, and to improve the robustness, maintainability, and efficiency. During the transfer to operations (TTO) the following major evolution projects were identified: APDF Archive back-end reengineering (APDF – CSA deployment) APDF Reference Platform Virtualization. Evolution of the Horus E2E mission monitoring tool. Implementation of the Full CDF Level-1b and Level-2 Cat-2 Operational Chain The Swarm PDGS has been completely reengineered, and all these changes were deployed in Operations without any disruption in January 2017. The Swarm PDGS is now prepared for the Mission Extension (2021 and beyond), and we have a firm foundation for the new activities to come (Mission Reprocessing.

PDGS Operational Configuration Component Rev Installation Date APDF@UK PAC   SARA SW: 2.5.4 CFG: 1.1 (SW-CP-AR-350) 04th May. 2017 19th Aug. 2015 SODA 1.2.0 12th Jul. 2017 CVT 1.0.6 19th Jan. 2017 APDF APDF SW 9.0.4 APDF CFG 3.0.0 10th Aug. 2017 L0GF V1.9 05th April. 2017 L1B V 3.18 p2 L2 CAT2 V 1.16 p2 10th Jul. 2017 DQC V3.05 CCDB Version 0015 (SW-A) Version 0015 (SW-B) Version 0016 (SW-C) 18th Oct. 2017 30th Jun. 2017 18th Oct. 2017 EFI Post Calibration Files Version 3 8th April 2014 Component Rev Installation Date MPPF@ESRIN   MPPF-DVT V3.0.2p3 19th Jan. 2017 MPPF-ICM Horus 2.3.5 10th Aug. 2017 CCDBTool 2.3 Almost all PDGS components have been updated since last Swarm DQW (in red). Major Evolution of the Swarm PDGS deployed in January 2017.

APDF Processing Baseline Status The table beside summarizes the L1 and L2 Cat-2 Current baseline status. The Product Baseline is a counter associated to a specific product type and satellite. It is increased when changes in a processor or calibrations file lead to significant improvements in the data quality. The Product Baseline is identified by the first two of the four digits placed at the end of the file name, while the last two represent an incremental File Counter which is increased in case of a regeneration of the product. SW_OPER_MAGC_LR_1B_20171001T000000_20171001T235959_0408.CDF.ZIP Since Sep. 2015, all L1 and L2 Cat-2 processors are running nominally. No Processing Baseline Changes since last IOPR https://earth.esa.int/web/guest/swarm/data-access/quality-of-swarm-l1b-l2cat2-products

Swarm Data Access Swarm data products are routinely made available to the Mission Teams and Registered Users immediately after production, through the Swarm ftp Dissemination Server (ftp://swarm-diss.eo.esa.int) hosted on the EO-GT ELVIS virtual infrastructure in Amsterdam with high availability, large bandwidth, and very good connectivity. Validated Level 1b and Level 2 products are openly, and freely available in accordance to the ESA EO data policy, after completion of the Fast Registration Procedure. More info at https://earth.esa.int/web/guest/swarm/data-access In total, 720 Swarm routine users and 191 Mission users/Partners from more than 63 countries have been registered until Oct 2017. 213 new routine users since September 2016. Steady growth.

Swarm Data Dissemination Statistics Since the beginning of the mission about 110 TB of data, and more that 6.7 millions of files have been distributed to Mission teams and registered users via the Swarm ftp dissemination server. Currently, around half a million of products are downloaded on a quarterly basis by users around the world. About 37 TB of data and 2.7 millions of files were downloaded during last year. Steady growth

New FTP Layout: Routine Users Following a Swarm community recommendation, several changes were introduced in the Swarm Dissemination FTP (swarm-diss.eo.esa.int) directory layout in early July 2017. All users were informed in advance both via e-mail and with news publication on the Swarm Web portal. Routine Users Under the “Level1b” and the “Level2daily” folders, the Swarm data is now presented through the set up of two main corresponding folders: The “Entire_Mission_Data” folder which contains the full data coverage of the entire mission, regardless of any consideration of interoperability among the same product type. May include mixed Product Baselines. Handle with caution! The “Latest_Baselines” folder which contains interoperable products generated in a consistent way with the application of all significant data quality improvements, but not necessarily covering the entire mission. The Orbit Counter file is now classified as a Level 1B product instead of an Auxiliary File. The “Auxiliary” and “Previous” Folders were removed

New FTP Layout: Mission Users Under the “Level0”, “Level1a”, “Level1b” and the “Level2daily” folders, the Swarm data is now presented through the set up of three main corresponding folders: The “Entire_Mission_Data” folder which contains the full data coverage of the entire mission, regardless of any consideration of interoperability among the same product type. May include mixed Product Baselines. Handle with caution! The “Latest_Baselines” folder which contains interoperable products generated in a consistent way with the application of all significant data quality improvements, but not necessarily covering the entire mission. The “Older_Baselines” folder which contains data that are not interoperable with the latest existing products. Such products are identified with an older Product Baseline number. The Orbit Counter file is now classified as a Level 1B product instead Aux

Personal credentials for Mission Users On August 9th all Mission users have received personal credentials to access the Swarm Dissemination server, to replace the current group mission accounts (i.e. ‘calval2’, ‘cnes’, ‘csa’, ‘prime’, ‘project’, etc.) which need to be dismissed due to the ESA security policy. A reminder email asking to complete the registration process and to start using the personal credentials provided has been sent to all Mission users on October 24th. The group accounts will be maintained active up until December 1st 2017 to ensure a smooth transition. Please, switch to your personal ftp account!

Swarm PDGS Outlook L1b & L2 Cat-2 Full Mission Reprocessing Campaign Swarm APDF Parallel Reprocessing Chain Integration of new data products Swarm Web Information & News

L1b & L2 Cat-2 Full Mission Reprocessing In Nov. 2017, new versions of the L1b v3.19 (ORBATT, ACCELE, MAGNET, PLASMA) & L2-Cat2 v1.17 operational processors (FAC, FAC_Comb, EEF, IBI, TEC) will be released. These new versions will include the latest improvements in the algorithms and numerous bug fixes (L1b: 25 bug fixes, 13 Evolution changes; L2-Cat2: 5 bug fixes, 7 Evolution changes) which need to integrated and validated carrefully in the APDF Reference Platform. The deployment of the new L1b/L2-Cat2 processors in Operations is planned in early Q1 2018. It is expected to start the Full Mission L1b/L2-Cat2 reprocessing in Q1 2018, being completed early Q2 2018. The Swarm Full Mission reprocessing will provide full consistent product datasets with complete coverage.

APDF Parallel Reprocessing Chain An Swarm APDF Parallel Processing chain is being implemented to carry out the coming Full Mission L1b/L2-Cat2 Reprocessing. This will ensure that the nominal processing activity is not impacted by the full reprocessing activities and that the dedicated chain can be designed in order to satisfy faster throughput requirements. The Swarm Parallel Reprocessing Chain will be based on a cloud infrastructure. One of the major benefits of the selected cloud computing solution is that it provides the opportunity to scale the computing resources whenever necessary (Target: full mission reprocessing in 6 weeks) Reprocessed products will be ingested off-line in the nominal APDF after being produced and validated on the Swarm Parallel Reprocessing Chain.

Integration of new products During 2018, new data products generated outside externally will be integrated into the PDGS ingestion, archiving, and dissemination dataflow. Historical & current data production from CASSIOPE/e-POP (Swarm-e). New data products coming from the 1st & 2nd DISC Call for Ideas

Swarm Web Information & News Technical Information and Operational News: Swarm Mission WebSite: https://earth.esa.int/swarm Swarm Mission on Twitter: @esa_swarm For Data Access: FTP dissemination Server: ftp://swarm-diss.eo.esa.int Data Access Instructions: https://earth.esa.int/web/guest/swarm/data-access For Data Quality: Data Quality for L1B and L2 Cat-2 products: https://earth.esa.int/web/guest/swarm/data-access/quality-of-swarm-l1b-l2cat2-products Swarm Level-1b Data Availability: https://earth.esa.int/documents/10174/1514862/Swarm_L1b_data_availability

Conclusions PDGS is in good shape. Major architecture upgrade completed without downtime. Ready for the extension up to 2021 and beyond. User’s Registration and Data Dissemination to partners/users/friends is steadily growing Please, switch to personal ftp accounts, otherwise you will not have Swarm data for Christmas Eve! L1b & L2 Cat-2 Full Mission Reprocessing Campaign Preparations: New APDF Parallel Reprocessing Chain New data products will be integrated into the PDGS ingestion, archiving, and dissemination dataflow. Stay tuned at http://earth.esa.int/swarm and @esa_swarm for Swarm Ops news

Many thanks for your attention! Questions?

FTP Advanced Folder 1. Orbit_and_Attitude_Data 3. Plasma_Data a) Star Tracker Thermally Corrected Dataset 2. Magnetic_Data a) ASM - VFM Residuals Dataset b) ASM Vector Data. b.1 Set of ASM Vector data b.2 Manoeuvre 4x90 c) ASM Burst Mode Data. d) L1B_MAGxLR_0502 3. Plasma_Data a) Provisional Plasma Dataset a.1 - Langmuir Probes Data a.2 - Thermal Ion Imagers Data b) TII Level 0 16 Hz image moments c) Faceplate derived electron density at 16 Hz d) Langmuir Probes extended dataset at 2 Hz e) TII cross-track ion flow dataset at 2 Hz f) Langmuir Probe Sweep Mode at 1/128 Hz g) TII cross-track ion flow dataset at 16 Hz