1. Page 1GS CDR May 2005 JSOC SDP Agenda Significant Level 4 requirements SDP Architecture Decomposition –Datacapture System –Datacapture System Components –Pipeline Processing System –Pipeline Processing System Components –Software Configuration Items –Hardware Configuration Items –Data Archive –Data Distribution –Performance Analysis –Heritage SDP Network Design & Security Test approach Implementation plan Deliverables hardware/software/documentation Development environment/CM Process Sustaining support pre-launch and post-launch Identify organization(s) providing support Identify types of support: vendor licenses, h/w, etc. Trade Studies and Prototyping Efforts since PDR Risks & Mitigations Procurement status

2. Page 2GS CDR May 2005 DMR – JSOC SDP Requirements 8000.2.4 Science Data Processing, Archiving and Distribution Each SOC shall provide the necessary facility, software, hardware and staff to receive, process, archive and distribute the science data generated by its instruments. My Documents\DMR_SOC_Req.ppt

3. Page 3GS CDR May 2005 JSOC (Stanford Science Data Processing) Configuration IDC Pipeline Processor [etc.] Analysis Cluster jim\My Documents\hardware_config3.vsd Pipeine Processing Data Base Server LAN Tape Robot Server DDS Switch heartbeat Active Passive Fibre Channel Switch Tape Robot Disks Switch ethernet Disk arrays File Server Datacapture System SDP

4. Page 4GS CDR May 2005 JSOC Data Capture System Components Database Server Tape Server DSDS Data Storage & Distribution System Tape Drive Primary Storage Disks lev0 “module” Record Cache GUI Data Capture Interface Operator Processing History Log FTP Disks DDS Utility Libraries

5. Page 5GS CDR May 2005 JDAT network

6. Page 6GS CDR May 2005 JSOC Pipeline Processing System Components Database Server SUMS Storage Unit Management System DRMS Data Record Management System SUMS Tape Farm SUMS Disks Pipeline Program, “module” Record Manage ment Keyword Access Data Access DRMS Library Link Manage ment Utility Libraries JSOC Science Libraries Record Cache PUI Pipeline User Interface Pipeline processing plan Processing script, “mapfile” List of pipeline modules with needed datasets for input, output Pipeline Operato r Processing History Log

7. Page 7GS CDR May 2005 S/W Configuration Items SUMS DRMS PUI Event Mgr Science Libs Pipeline Util Libs Export Support Lev0 Lev1 Standard Products

8. Page 8GS CDR May 2005 H/W Configuration

9. Page 9GS CDR May 2005 H/W Configuration (cont)

10. Page 10GS CDR May 2005 Telemetry Data Archive Telemetry data is archived twice The Data Capture System archives tlm files for offsite storage Archive tapes are shipped to the offsite location and verified for reading A feedback mechanism will be established to ack the SDP that a tape is verified or that another copy needs to be sent The Data Capture System copies tlm files to the Pipeline Processing System The Pipeline Processing System archives tlm data for local storage and acks the SDP when it is successful Only when the SDP has received positive acks on both archive copies does it inform the Data Capture System to include the tlm file in the.arc file to the DDS, who is now free to remove the file from its tracking logic The lev0 data on the Pipeline Processing System is also archived

11. Page 11GS CDR May 2005 JSOC Data Export System DRMS Package Format Custom Keywords Utilities Selecte d Data Records API Drilldown Overview New/Avail Statistics Keywords Range Search Browse SOHO GONG WCS-x FITS VOTable plain (FITSz) (CDF) (JPEG) Researcher A General Public Grid Adaptor Grid VSO Adaptor VSO CoSEC CoSEC Adaptor Researcher B Pre- process Filter Mask Track Auto Update Redistribute Notify Filename Schema 20090401_1958.fits AR12040_171.vot magCR2801:lon064lat20S.txt Compress gzip jpeg low-resolution Script Access Space Weather

12. Page 12GS CDR May 2005 Performance Analysis AIA/HMI combined data volume2 PB/yr = 60 MB/s –read + write x 2 –quick look + final x 2 –one reprocessing x 2 –25% duty cycle x 4 2 GB/s (disk) 0.5 GB/s (tape) NFS over gigabit ethernet (50-100MB/s/channel) 1 – 4 GB/s –4 – 8 channels per server, 5 servers (today) SAIT-1 native transfer rate (25-30 MB/s/drive) 0.5 – 0.6 GB/s –10 SAIT-1 drives per library, 2 libraries (today)

13. Page 13GS CDR May 2005 Heritage MDI ground system design and implementation MDI production processing for 9+ years MDI sustaining engineering for h/w and s/w upgrades MDI “lessons learned” folded into JSOC SDP design

14. Page 14GS CDR May 2005 Stanford/Lockheed Connections Stanford DDS MOC LMSAL 1 Gb Private line JSOC Disk array NASA AMES “White” Net

15. Page 15GS CDR May 2005 Test Approach Telemetry data is simulated with MDI data packaged in SDO VCDUs Various telemetry pathologies can be created to validate telemetry processing The DDS is simulated by packaging the VCDUs into files and injecting them into the SOC front end using the DDS/SOC protocol The lev0 processing is validated by comparing input and output images Timing analysis and throughput studies will be performed to evaluate h/w and s/w tradeoffs Instrument sunlight tests will be used to validate the lev1 processing Each s/w library and subsystem has a regression test suite that is run to verify each new CM release Full end to end testing is accomplished via the mission I&T plan

16. Page 16GS CDR May 2005 Integrated Schedule JSOC Science Data Processing (SDP) / DDS I&T Start Dates Delivery of Flight EGSE SDPJune 2005 Prototype SDP System ReadyDec 2005 JSOC Network ReadyDec 2006 DDS-JSOC Testing Dec 2006 GSRT#2- Science Data Processing Test (Ka-band)Jan 2007 HMI Connectivity, Dataflow, Retransmissions TestFeb 2007 AIA Connectivity, Dataflow, Retransmissions TestFeb 2007 GSRT#3-Mission Operations& RF Communications TestMar 2007 GSRT#4-Fully Integrate Ground System Mar 2007 Ground System FreezeJan 2008 GSRT#4-Launch Readiness TestFeb 2008

17. Page 17GS CDR May 2005 Implementation Plan TBD – Get from Carl

18. Page 18GS CDR May 2005 Deliverables EGSE hardware EGSE software TBD – Get from Carl (from contract)

19. Page 19GS CDR May 2005 Configuration Management & Control Capture System –Managed by JSOC CCB –Controlled in CVS SUMS, DRMS, PUI, etc. Infrastructure –Managed by JSOC CCB after launch –Controlled in CVS PUI Processing Tables –Managed by HMI and/or AIA Instrument Scientist –Controlled in CVS Level 0,1 Pipeline Modules –Managed by HMI and/or AIA Instrument Scientist –Controlled in CVS Science Analysis Pipeline Modules –Managed by program author –Controlled in CVS

20. Page 20GS CDR May 2005 CM with CVS

21. Page 21GS CDR May 2005 Prototype/Trade Study Sequence “MDI Heritage” software for EGSE Science W/S (currently running) Study to separate keywords from image data in a keyword Database MDI database optimization study MDI internal data set representation study Storage management trade study (commercial HSM vs. MDI heritage) Archive media study to replace MDI Ampex robotic units Storage Area Network (SAN) / Network Attached Storage (NAS) trade study Location of offsite media storage

22. Page 22GS CDR May 2005 Risks and Mitigations Physical location of JSOC - Tracking through Stanford Management - Rent space off-campus Cost risk in assumed rate of computer technology development - Advances already made - Wide margin of costing allowed Data Base Performance - Prototyping underway - Design alternatives - Higher performance equipment

23. Page 23GS CDR May 2005 Development & Test Plans and Procurement Schedule HMI and AIA Data EGSE installed –Prototype for I/F testing with GSMar, 2005 onward –Version 2 to support flight inst.June 2005 JSOC Capture System –Purchase computersSummer 2006 –Support DDS testingFall 2006 –Final system installedSpring 2007 JSOC SDP Infrastructure, SUMS, DRMS, PUI –Prototype testing of core systemJune 2005 –Fully functionalDec, 2005 Purchase computers for JSOC Jan, 2007 Infrastructure Operational April, 2007 Data Product Modules Jan, 2008 Test in I&T and with DDS,MOC as called for in SDO Ground System schedule During Phase-E Add media and disk farm capacity in staged plan, half-year or yearly increments First two years of mission continue Co-I pipeline testing support

24. Page 24GS CDR May 2005

25. Page 25GS CDR May 2005 Test Approach Unit test Sub System Integration Regression Tests Performance Test System Integration I &T with DDS/MOC On-line test results and history

26. Page 26GS CDR May 2005 Procurement Status –2004 - 2005: First 2 Years Procure development system with most likely components (e.g. tape type, cluster vs SMP, SAN vs NAS, etc) Modify pipeline and catalog infrastructure and implement on prototype system. Modify analysis module API for greater simplicity and compliance with pipeline. Develop calibration software modules. Complete level 0 processing code to support HMI/AIA instrument testing. –2006 - 2007: Two years prior to launch Complete Level-1 analysis development, verify with HMI/AIA test data. Populate prototype system with MDI/TRACE data to verify performance. Procure, install, verify computer hardware. Implement higher-level pipeline processing modules with Co-I support –During Phase-E Add media and disk farm capacity in staged plan, half-year or yearly increments First two years of mission continue Co-I pipeline testing support