1. Swarm PDGS Status & Outlook
Antonio de la Fuente & Swarm PDGS team
Swarm 7th Data Quality Workshop
24th – 27th October 2017, Delft

2. Summary
Swarm PDGS Status
Swarm PDGS Outlook
Conclusions

3. Swarm PDGS Status Swarm PDGS Architecture
Swarm PDGS Architecture Evolution (Jan 2017)
Swarm PDGS Configuration
Processing Baseline Status
Swarm Data Access & Dissemination Statistics
Swarm Dissemination FTP Structure Change (Jul 2017)
Personal credentials for Mission Users (Aug 2017)

4. Swarm PDGS Architecture
The Swarm PDGS functions are distributed among different locations:
the Archiving, Processing and Dissemination Facility (APDF) is hosted in Newport and operated from Guildford (UK) by Airbus D&S. In the APDF, satellite science telemetry coming from FOS and auxiliary data are ingested, L1b & L2-Cat2 systematically processed, archived, and disseminated.
the Level 2 Processing System (L2PS), also known as SCARF, is a cluster of institutions where scientific Level 2 Cat-1 products are generated, and sent back to the APDF for archiving and distribution.
the Support to Operations, E2E performance monitoring, and calibration and data quality assurance subsystems (CVQ) are located in ESRIN
All PDGS functions are running without major problems since Launch

5. PDGS Architecture Evolution completed
The architecture of the Swarm PDGS is always evolving to respond to new operational and science requirements, and to improve the robustness, maintainability, and efficiency. During the transfer to operations (TTO) the following major evolution projects were identified:
APDF Archive back-end reengineering (APDF – CSA deployment)
APDF Reference Platform Virtualization.
Evolution of the Horus E2E mission monitoring tool.
Implementation of the Full CDF Level-1b and Level-2 Cat-2 Operational Chain
The Swarm PDGS has been completely reengineered, and all these changes were deployed in Operations without any disruption in January 2017.
The Swarm PDGS is now prepared for the Mission Extension (2021 and beyond), and we have a firm foundation for the new activities to come (Mission Reprocessing.

6. PDGS Operational Configuration
Component
Rev
Installation Date
PAC
SARA
SW: CFG: 1.1 (SW-CP-AR-350)
04th May th Aug. 2015
SODA
1.2.0
12th Jul. 2017
CVT
1.0.6
19th Jan. 2017
APDF
APDF SW APDF CFG 3.0.0
10th Aug. 2017
L0GF
V1.9
05th April. 2017
L1B
V 3.18 p2
L2 CAT2
V 1.16 p2
10th Jul. 2017
DQC
V3.05
CCDB
Version 0015 (SW-A) Version 0015 (SW-B) Version 0016 (SW-C)
18th Oct th Jun th Oct. 2017
EFI Post Calibration Files
Version 3
8th April 2014
Component
Rev
Installation Date
MPPF-DVT
V3.0.2p3
19th Jan. 2017
MPPF-ICM
Horus
2.3.5
10th Aug. 2017
CCDBTool
2.3
Almost all PDGS components have been updated since last Swarm DQW (in red).
Major Evolution of the Swarm PDGS deployed in January 2017.

7. APDF Processing Baseline Status
The table beside summarizes the L1 and L2 Cat-2 Current baseline status.
The Product Baseline is a counter associated to a specific product type and satellite. It is increased when changes in a processor or calibrations file lead to significant improvements in the data quality.
The Product Baseline is identified by the first two of the four digits placed at the end of the file name, while the last two represent an incremental File Counter which is increased in case of a regeneration of the product.
SW_OPER_MAGC_LR_1B_ T000000_ T235959_0408.CDF.ZIP
Since Sep. 2015, all L1 and L2 Cat-2 processors are running nominally.
No Processing Baseline Changes since last IOPR

8. Swarm Data Access
Swarm data products are routinely made available to the Mission Teams and Registered Users immediately after production, through the Swarm ftp Dissemination Server (ftp://swarm-diss.eo.esa.int) hosted on the EO-GT ELVIS virtual infrastructure in Amsterdam with high availability, large bandwidth, and very good connectivity.
Validated Level 1b and Level 2 products are openly, and freely available in accordance to the ESA EO data policy, after completion of the Fast Registration Procedure. More info at
In total, 720 Swarm routine users and 191 Mission users/Partners from more than 63 countries have been registered until Oct new routine users since September Steady growth.

9. Swarm Data Dissemination Statistics
Since the beginning of the mission about 110 TB of data, and more that 6.7 millions of files have been distributed to Mission teams and registered users via the Swarm ftp dissemination server. Currently, around half a million of products are downloaded on a quarterly basis by users around the world. About 37 TB of data and 2.7 millions of files were downloaded during last year. Steady growth

10. New FTP Layout: Routine Users
Following a Swarm community recommendation, several changes were introduced in the Swarm Dissemination FTP (swarm-diss.eo.esa.int) directory layout in early July All users were informed in advance both via and with news publication on the Swarm Web portal.
Routine Users
Under the “Level1b” and the “Level2daily” folders, the Swarm data is now presented through the set up of two main corresponding folders:
The “Entire_Mission_Data” folder which contains the full data coverage of the entire mission, regardless of any consideration of interoperability among the same product type. May include mixed Product Baselines. Handle with caution!
The “Latest_Baselines” folder which contains interoperable products generated in a consistent way with the application of all significant data quality improvements, but not necessarily covering the entire mission.
The Orbit Counter file is now classified as a Level 1B product instead of an Auxiliary File. The “Auxiliary” and “Previous” Folders were removed

11. New FTP Layout: Mission Users
Under the “Level0”, “Level1a”, “Level1b” and the “Level2daily” folders, the Swarm data is now presented through the set up of three main corresponding folders:
The “Entire_Mission_Data” folder which contains the full data coverage of the entire mission, regardless of any consideration of interoperability among the same product type. May include mixed Product Baselines. Handle with caution!
The “Latest_Baselines” folder which contains interoperable products generated in a consistent way with the application of all significant data quality improvements, but not necessarily covering the entire mission.
The “Older_Baselines” folder which contains data that are not interoperable with the latest existing products. Such products are identified with an older Product Baseline number.
The Orbit Counter file is now classified as a Level 1B product instead Aux

12. Personal credentials for Mission Users
On August 9th all Mission users have received personal credentials to access the Swarm Dissemination server, to replace the current group mission accounts (i.e. ‘calval2’, ‘cnes’, ‘csa’, ‘prime’, ‘project’, etc.) which need to be dismissed due to the ESA security policy. A reminder asking to complete the registration process and to start using the personal credentials provided has been sent to all Mission users on October 24th. The group accounts will be maintained active up until December 1st 2017 to ensure a smooth transition. Please, switch to your personal ftp account!

13. Swarm PDGS Outlook L1b & L2 Cat-2 Full Mission Reprocessing Campaign
Swarm APDF Parallel Reprocessing Chain
Integration of new data products
Swarm Web Information & News

14. L1b & L2 Cat-2 Full Mission Reprocessing
In Nov. 2017, new versions of the L1b v3.19 (ORBATT, ACCELE, MAGNET, PLASMA) & L2-Cat2 v1.17 operational processors (FAC, FAC_Comb, EEF, IBI, TEC) will be released. These new versions will include the latest improvements in the algorithms and numerous bug fixes (L1b: 25 bug fixes, 13 Evolution changes; L2-Cat2: 5 bug fixes, 7 Evolution changes) which need to integrated and validated carrefully in the APDF Reference Platform. The deployment of the new L1b/L2-Cat2 processors in Operations is planned in early Q It is expected to start the Full Mission L1b/L2-Cat2 reprocessing in Q1 2018, being completed early Q The Swarm Full Mission reprocessing will provide full consistent product datasets with complete coverage.

15. APDF Parallel Reprocessing Chain
An Swarm APDF Parallel Processing chain is being implemented to carry out the coming Full Mission L1b/L2-Cat2 Reprocessing. This will ensure that the nominal processing activity is not impacted by the full reprocessing activities and that the dedicated chain can be designed in order to satisfy faster throughput requirements.
The Swarm Parallel Reprocessing Chain will be based on a cloud infrastructure. One of the major benefits of the selected cloud computing solution is that it provides the opportunity to scale the computing resources whenever necessary (Target: full mission reprocessing in 6 weeks)
Reprocessed products will be ingested off-line in the nominal APDF after being produced and validated on the Swarm Parallel Reprocessing Chain.

16. Integration of new products
During 2018, new data products generated outside externally will be integrated into the PDGS ingestion, archiving, and dissemination dataflow.
Historical & current data production from CASSIOPE/e-POP (Swarm-e).
New data products coming from the 1st & 2nd DISC Call for Ideas

17. Swarm Web Information & News
Technical Information and Operational News:
Swarm Mission WebSite:
Swarm Mission on
For Data Access:
FTP dissemination Server:
ftp://swarm-diss.eo.esa.int
Data Access Instructions:
For Data Quality: Data Quality for L1B and L2 Cat-2 products: Swarm Level-1b Data Availability:

18. Conclusions
PDGS is in good shape. Major architecture upgrade completed without downtime. Ready for the extension up to 2021 and beyond.
User’s Registration and Data Dissemination to partners/users/friends is steadily growing
Please, switch to personal ftp accounts, otherwise you will not have Swarm data for Christmas Eve!
L1b & L2 Cat-2 Full Mission Reprocessing Campaign Preparations: New APDF Parallel Reprocessing Chain
New data products will be integrated into the PDGS ingestion, archiving, and dissemination dataflow.
Stay tuned at for Swarm Ops news

19. Many thanks for your attention! Questions?

20. FTP Advanced Folder 1. Orbit_and_Attitude_Data 3. Plasma_Data
a) Star Tracker Thermally Corrected Dataset
2. Magnetic_Data
a) ASM - VFM Residuals Dataset
b) ASM Vector Data.
b.1 Set of ASM Vector data
b.2 Manoeuvre 4x90
c) ASM Burst Mode Data.
d) L1B_MAGxLR_0502
3. Plasma_Data
a) Provisional Plasma Dataset
a.1 - Langmuir Probes Data
a.2 - Thermal Ion Imagers Data
b) TII Level 0 16 Hz image moments
c) Faceplate derived electron density at 16 Hz
d) Langmuir Probes extended dataset at 2 Hz
e) TII cross-track ion flow dataset at 2 Hz
f) Langmuir Probe Sweep Mode at 1/128 Hz
g) TII cross-track ion flow dataset at 16 Hz