Born EFW SOC Science Data Center (SDC) SOC-SDC Software Design Matt Born 85RBSP-EFW SOC-CDR 26 Jan 2010

Born SDC-RET: You Are Here 86RBSP-EFW SOC-CDR 26 Jan 2010

SDC-RET Data Retrieval Modules Requirements: – Retrieve data from external sources – Deposit the data into an archive – Log the transfer of data – Operate within fixed latency bounds Design principles: – Single “pull” interface per module – Single “push” interface per module, always to ARC-INT Implementation: – Python first draft – C optimizations where required – Most data transfers over SFTP per interface documents 87 RBSP-EFW SOC-CDR 26 Jan 2010

SDC Data Retrieval Modules The recipe: 88 RBSP-EFW SOC-CDR 26 Jan 2010

SDC-RET Data Retrieval Modules Module (CSCI)RetrievesFromUsing SDC-RET-MDPEFW SOC L0RBSP MOCSFTP SDC-RET-MAG MOC EMFISIS MAG RAW, MOC EMFISIS MAG CAL RBSP EMFISIS SOCSFTP SDC-RET-ECTECT SOC L2RBSP ECT SOCSFTP SDC-RET-SCLKMOC SCLKRBSP MOCSFTP SDC-RET-STATEMOC STATERBSP MOCSFTP SDC-RET-ANCMOC ANCRBSP MOCSFTP SDC-RET-GEO[TBD] 89 RBSP-EFW SOC-CDR 26 Jan 2010

Born SDC-ARC: You Are Here 90RBSP-EFW SOC-CDR 26 Jan 2010

SDC-ARC Data Storage Modules Requirements: – Archive data from internal and external sources – Log the source and time of deposited data – Maintain previous versions of deposited data Design principles: – No direct producer-to-consumer data transfer – Separate archives for public data versus private data Implementation: – Python first draft – C optimizations where required – At least HTTP and SSH access to archives 91 RBSP-EFW SOC-CDR 26 Jan 2010

Born SDC-PDP: You Are Here 92RBSP-EFW SOC-CDR 26 Jan 2010

SDC-PDP Data Processing Modules Requirements: – Synthesize new data products from existing ones – Log the sources and methods used for data creation – Deposit the data into an archive – Operate within fixed latency bounds Design principles: – No direct producer-to-consumer data transfer – Fully automatic processes with no user intervention Implementation: – Python first draft – C optimizations where required – Scripted execution of the NRT module for plot generation 93 RBSP-EFW SOC-CDR 26 Jan 2010

SDC-PDP Data Processing Modules The recipe: 94 RBSP-EFW SOC-CDR 26 Jan 2010

SDC-PDP Data Processing Modules CSCIRequired Data ProductsGenerated Data Products SDC-PDP-L0->L1EFW SOC L0EFW SOC L1 SDC-PDP-L1->L2 EFW SOC L1, EMFISIS SOC L0, MOC EMFISIS CAL, MOC EMFISIS RAW, EFW SOC MAINT, EFW SOC CAL, EFW SOC STATE EFW SOC L2 SDC-PDP-SOHEFW SOC L0EFW SOC MAINT SDC-PDP-STATEMOC STATEEFW SOC STATE SDC-PDP-CAL EFW SOC L1, EFW SOC L2, EFW SOC STATE, EMFISIS SOC L0, ECT SOC L0 EFW SOC CAL SDC-PDP-QLEFW SOC L2, MOC EMFISIS CAL EFW SOC QL 95 RBSP-EFW SOC-CDR 26 Jan 2010

SOC-PDP Data Products ProductContentsProcessingVolume L0Raw telemetry-130 MB/day/sat L1 Time-tagged raw waveform and spectral data (expressed in spinning-spacecraft coordinate system) Reformat from proprietary binary files to CDF 520 MB/day/sat L2 Calibrated waveform and spectral data (expressed in despun spacecraft coordinate system and other geophysical coordinate systems as required) Calibrate data and recalculate coordinates in despun spacecraft coordinate system, GSE, GSM, SM, etc. 2.1 GB/day/sat L3 Calibrated waveform and spectral data with VxB subtraction for DC electric field estimates SC Attitude Correction (EFW Science Team Input). 3.1 GB/day/sat L4Global electric field pattern. Statistical patterns of spectral properties. Data selection and sorting by relevant geophysical indices (Kp, Dst, etc.). Negligible (datasets supporting published works) 96 RBSP-EFW SOC-CDR 26 Jan 2010

SOC-PDP Monitoring of Critical Telemetry 97 RBSP-EFW SOC-CDR 26 Jan 2010

SOC-SDC Trending of Critical Telemetry 98 RBSP-EFW SOC-CDR 26 Jan 2010

Born SDC-MET UTC: You Are Here 99RBSP-EFW SOC-CDR 26 Jan 2010

SDC-MET UTC Time Conversion Utility Requirements: – Convert between times expressed in MET or UTC – Validate conversions against canonical MOC conversions Design principles: – Conversion speed is essential Implementation: – Python first draft – C optimizations where required – At least HTTP access to service 100RBSP-EFW SOC-CDR 26 Jan 2010

Born SDC-MET UTC Time Conversion Utility 101RBSP-EFW SOC-CDR 26 Jan 2010

Born SDC-NRT: You Are Here 102RBSP-EFW SOC-CDR 26 Jan 2010

SDC-NRT Near Real Time Analysis Tool Requirements: – Display waveforms and spectra – Display header-level information Design principles: – Leverage past project code base and IDL expertise – Provide an extensible and usable tool for future re-use Implementation: – IDL with Python extensions – Integration of legacy code – Scriptable operation 103RBSP-EFW SOC-CDR 26 Jan 2010

Born SDC-NRT Near Real Time Analysis Tool Instrument I&T Usage 104RBSP-EFW SOC-CDR 26 Jan 2010

Born SDC-NRT Near Real Time Analysis Tool Spacecraft I&T Usage 105RBSP-EFW SOC-CDR 26 Jan 2010

Born SDC-NRT Near Real Time Analysis Tool Flight Usage 106RBSP-EFW SOC-CDR 26 Jan 2010

Born SDC-BSEL: You Are Here 107RBSP-EFW SOC-CDR 26 Jan 2010

SDC-BSEL Burst Selection Utility Requirements: – Allow selection of burst data by timestamp – Log the selection of data – Automatically select data if user declines to do so Implementation: – Python first draft – C optimizations where required – Interface with CTG module for command generation 108RBSP-EFW SOC-CDR 26 Jan 2010

Born SDC-BSEL: Burst Selection Utility 109RBSP-EFW SOC-CDR 26 Jan 2010

Born SDC-BSEL: Burst Selection Utility 110RBSP-EFW SOC-CDR 26 Jan 2010

Born SDC-BSEL: Burst Selection Utility 111RBSP-EFW SOC-CDR 26 Jan 2010

Born SDC-DVAL: You Are Here 112RBSP-EFW SOC-CDR 26 Jan 2010

SDC-DVAL Data Validation and Release Requirements: – Display data eligible for release along with data necessary to decide upon release – Publish data deemed releasable – Operate within fixed latency bounds Design principles: – Reuse of existing modules for plot display – Strictly controlled route between private and public archive Implementation: – Python first draft – C optimizations where required 113RBSP-EFW SOC-CDR 26 Jan 2010

Born SDC-DVAL Data Validation and Release 114RBSP-EFW SOC-CDR 26 Jan 2010

Born SDC-SDT: You Are Here 115RBSP-EFW SOC-CDR 26 Jan 2010

SDC-SDT Science Data Tools Requirements: – Display and analyze waveforms and spectra – Access data from the public archive, ARC-PUB We have determined that there exist numerous tools which will be applicable for scientific analysis of the instrument data. Software PackageSupported byNotes SDC-NRT moduleRBSP EFW SOCPer requirements THEMIS Data Analysis Suite THEMIS SOC (core) and RBSP EFW SOC (extended) Per requirements SDT “Science Data Tool” Cluster EFW Data Analysis EFW Science Team L1 and up, requires CDF format AutoplotOutside VendorL1 and up, requires CDF format 116RBSP-EFW SOC-CDR 26 Jan 2010