1. Page 1LWS Teams Day JSOC Overview HMI-AIA Joint Science Operations Center Science Data Processing a.k.a. JSOC-SDP Overview

2. Page 2LWS Teams Day JSOC Overview HMI & AIA JSOC Concept Science Team Forecast Centers EPO Public Catalog Primary Archive HMI & AIA Operations House- keeping Database MOC SDOGS DDS Redundant Data Capture System 10-Day Archive Offsite Archiv e LMSA L Offline Archiv e HMI JSOC Pipeline Processing System Data Export & Web Service Stanford LMSAL High-Level Data Import AIA Analysis System Local Archive Quicklook Viewing GSFC White Sands World

3. Page 3LWS Teams Day JSOC Overview JSOC Dataflow Rates

4. Page 4LWS Teams Day JSOC Overview JSOC-SDP Major Components

5. Page 5LWS Teams Day JSOC Overview JSOC DRMS/SUMS Basic Concepts Each “image” is stored as a record in a data “series”. There will be many series: e.g. hmi_ground.lev0 is ground test data The image metadata is stored in a relational database – our Data Record Management System (DRMS) The image data is stored in SUMS (Storage Unit Management System) which itself has database tables to manage its millions of files. SUMS owns the disk and tape resources. Users interact with DRMS via a programming lauguage, e.g. c, FORTRAN, IDL. The “name” of a dataset is actually a query in a simplified DRMS naming language that also allows general SQL clauses. Efficient use of the system relies on direct use of DRMS Data may be exported from DRMS as FITS or other (TBD) protocols for remote users. Several Remote DRMS (RDRMS?) sites will be established which will “subscribe” to series of their choice. They will maintain RSUMS containing their local series and cached JSOC series. The JSOC may act as an RDRMS to access products made at remote sites.

6. Page 6LWS Teams Day JSOC Overview Pipeline Program, “module” Record Manage ment Keyword Access Data Access DRMS Library Link Manage ment Utility Libraries JSOC Science Libraries Record Cache JSOC Pipeline Processing System Components Database Server SUMS Storage Unit Management System DRMS Data Record Management System SUMS Tape Farm SUMS Disks PUI Pipeline User Interface Pipeline processing plan Processing script, “mapfile” List of pipeline modules with needed datasets for input, output Pipeline Operato r Processing History Log

7. Page 7LWS Teams Day JSOC Overview Simple example – find and look at an image P% P% show_keys "ds=hmi_ground.lev0[][2007.09.09_23:50/1m]" key=FSN,T_OBS FSN T_OBS 0566684 2007.09.09_23:50:01_UTC 0566685 2007.09.09_23:50:06_UTC 0566686 2007.09.09_23:50:11_UTC 0566687 2007.09.09_23:50:16_UTC 0566688 2007.09.09_23:50:21_UTC 0566689 2007.09.09_23:50:26_UTC 0566690 2007.09.09_23:50:31_UTC 0566691 2007.09.09_23:50:36_UTC 0566692 2007.09.09_23:50:41_UTC 0566693 2007.09.09_23:50:46_UTC 0566694 2007.09.09_23:50:51_UTC 0566695 2007.09.09_23:50:56_UTC P% P% ds9 `show_keys "ds=hmi_ground.lev0[566686]" seg=file -p -q` P% Example of simple utility “module” called “show_”keys” First find images in the minute starting 9 Sept at 23:50 then look at one with ds9

8. Page 8LWS Teams Day JSOC Overview JSOC Export JSOC will support VSO access JSOC will also have a direct web access There will be remote DRMS/SUMS systems at key Co-I institutions ALL HMI and AIA data will be available for export at level-0 through standard products (level-1 for both and level-2 for HMI) It would be unwise to expect to export all of the data. It is simply not a reasonable thing to expect and would be a waste or resources. Our goal is to make all useful data easily accessible. This means “we” must develop browse and search tools to help generate efficient data export requests.

9. Page 9LWS Teams Day JSOC Overview DRMS/SUMS Configuration DRMS and SUMS use the open-source PostgreSQL database engine. DRMS will run on a pair of dedicated servers likely 4-quad-core processors with up to 10TB fast disk. SUMS will consist of file servers with attached tape systems. SUMS will manage 200TB cache disk with 150TB/year of permanent archive for level-1 and higher level products. SUMS archive will use LTO-4 tapes in a robotic system with at least 10 drives and 2000 tapes near-line. The pipeline processing system will have about 50 processor cores dedicated to level-0 to level-1 processing and about 450 cores for higher level processing in the pipeline. HMI standard products will need about half of these.

10. Page 10LWS Teams Day JSOC Overview Extra Info

11. Page 11LWS Teams Day JSOC Overview Data Record Management Service (DRMS) Data Record Management Service (DRMS) Pipeline client-server architecture JSOC Disks Analysis code C/Fortran/IDL/Matlab JSOC Library Record Cache (Keywords+Links+Data paths) OpenRecords CloseRecords GetKeyword, SetKeyword GetLink, SetLink OpenDataSegment CloseDataSegment Pipeline client process JSOC Disks Data Record Management Service (DRMS) Storage Unit Management Service (SUMS) Tape Archive Service Series Tables Record Catalogs Storage Unit Tables Record Catalogs Record Tables PostgreSQL Database Server AllocUnit GetUnit PutUnit SQL queries Storage unit transfer Data Segment I/O Storage unit transfer Generic file I/O DRMS socket protocol

12. Page 12LWS Teams Day JSOC Overview Pipeline batch processing A pipeline batch is encapsulated in a single database transaction, “DRMS session”: –If no module fails all data records are commited and become visible to other clients of the JSOC catalog at the end of the session –If failure occurs all data records are deleted and the database rolled back –It is possible to commit data produced up to intermediate checkpoints during sessions DRMS Service = Session Master Input data records Output data records DRMS API Register session DRMS API Module 1 DRMS API Module 2.2 … DRMS API Module N DRMS API Commit Data & Deregister Pipeline batch = atomic transaction Record & Series Database SUMS DRMS API Module 2.1

13. Page 13LWS Teams Day JSOC Overview HMI module status and MDI heritage Doppler Velocity Heliographic Doppler velocity maps Tracked Tiles Of Dopplergrams Stokes I,V Continuum Brightness Tracked full-disk 1-hour averaged Continuum maps Brightness feature maps Solar limb parameters Stokes I,Q,U,V Full-disk 10-min Averaged maps Tracked Tiles Line-of-sight Magnetograms Vector Magnetograms Fast algorithm Vector Magnetograms Inversion algorithm Egression and Ingression maps Time-distance Cross-covariance function Ring diagrams Wave phase shift maps Wave travel times Local wave frequency shifts Spherical Harmonic Time series Mode frequencies And splitting Brightness Images Line-of-Sight Magnetic Field Maps Coronal magnetic Field Extrapolations Coronal and Solar wind models Far-side activity index Deep-focus v and c s maps (0-200Mm) High-resolution v and c s maps (0-30Mm) Carrington synoptic v and c s maps (0-30Mm) Full-disk velocity, sound speed, Maps (0-30Mm) Internal sound speed Internal rotation Vector Magnetic Field Maps MDI pipeline modules exist Standalone production codes in use at Stanford Research codes in use by team Codes to be developed at HAO Codes being developed in the community Codes to be developed at Stanford Primary observables Intermediate and high level data products

14. Page 14LWS Teams Day JSOC Overview AIA Level-2

15. Page 15LWS Teams Day JSOC Overview JSOC Data Volumes from Proposal

16. Page 16LWS Teams Day JSOC Overview Sample of DRMS/SUMS Resource Assessment

17. Page 17LWS Teams Day JSOC Overview JSOC Level-0 processing flow

18. Page 18LWS Teams Day JSOC Overview JSOC-SDP Status Capture system complete, waiting testing with DDS at White Sands DRMS and SUMS virtually done Level-0 work progressing, to be done by instrument deliveries Level-1 work to start after delivery Getting serious about work on basic pipeline modules Review of status and computer hardware plans in early November

19. Page 19LWS Teams Day JSOC Overview Stanford JSOC effort plan