Page 1LWS Teams Day JSOC Overview HMI-AIA Joint Science Operations Center Science Data Processing a.k.a. JSOC-SDP Overview

Page 2LWS Teams Day JSOC Overview HMI & AIA JSOC Concept Science Team Forecast Centers EPO Public Catalog Primary Archive HMI & AIA Operations House- keeping Database MOC SDOGS DDS Redundant Data Capture System 10-Day Archive Offsite Archiv e LMSA L Offline Archiv e HMI JSOC Pipeline Processing System Data Export & Web Service Stanford LMSAL High-Level Data Import AIA Analysis System Local Archive Quicklook Viewing GSFC White Sands World

Page 3LWS Teams Day JSOC Overview JSOC Dataflow Rates

Page 4LWS Teams Day JSOC Overview JSOC-SDP Major Components

Page 5LWS Teams Day JSOC Overview JSOC DRMS/SUMS Basic Concepts Each “image” is stored as a record in a data “series”. There will be many series: e.g. hmi_ground.lev0 is ground test data The image metadata is stored in a relational database – our Data Record Management System (DRMS) The image data is stored in SUMS (Storage Unit Management System) which itself has database tables to manage its millions of files. SUMS owns the disk and tape resources. Users interact with DRMS via a programming lauguage, e.g. c, FORTRAN, IDL. The “name” of a dataset is actually a query in a simplified DRMS naming language that also allows general SQL clauses. Efficient use of the system relies on direct use of DRMS Data may be exported from DRMS as FITS or other (TBD) protocols for remote users. Several Remote DRMS (RDRMS?) sites will be established which will “subscribe” to series of their choice. They will maintain RSUMS containing their local series and cached JSOC series. The JSOC may act as an RDRMS to access products made at remote sites.

Page 6LWS Teams Day JSOC Overview Pipeline Program, “module” Record Manage ment Keyword Access Data Access DRMS Library Link Manage ment Utility Libraries JSOC Science Libraries Record Cache JSOC Pipeline Processing System Components Database Server SUMS Storage Unit Management System DRMS Data Record Management System SUMS Tape Farm SUMS Disks PUI Pipeline User Interface Pipeline processing plan Processing script, “mapfile” List of pipeline modules with needed datasets for input, output Pipeline Operato r Processing History Log

Page 7LWS Teams Day JSOC Overview Simple example – find and look at an image P% P% show_keys "ds=hmi_ground.lev0[][ _23:50/1m]" key=FSN,T_OBS FSN T_OBS _23:50:01_UTC _23:50:06_UTC _23:50:11_UTC _23:50:16_UTC _23:50:21_UTC _23:50:26_UTC _23:50:31_UTC _23:50:36_UTC _23:50:41_UTC _23:50:46_UTC _23:50:51_UTC _23:50:56_UTC P% P% ds9 `show_keys "ds=hmi_ground.lev0[566686]" seg=file -p -q` P% Example of simple utility “module” called “show_”keys” First find images in the minute starting 9 Sept at 23:50 then look at one with ds9

Page 8LWS Teams Day JSOC Overview JSOC Export JSOC will support VSO access JSOC will also have a direct web access There will be remote DRMS/SUMS systems at key Co-I institutions ALL HMI and AIA data will be available for export at level-0 through standard products (level-1 for both and level-2 for HMI) It would be unwise to expect to export all of the data. It is simply not a reasonable thing to expect and would be a waste or resources. Our goal is to make all useful data easily accessible. This means “we” must develop browse and search tools to help generate efficient data export requests.

Page 9LWS Teams Day JSOC Overview DRMS/SUMS Configuration DRMS and SUMS use the open-source PostgreSQL database engine. DRMS will run on a pair of dedicated servers likely 4-quad-core processors with up to 10TB fast disk. SUMS will consist of file servers with attached tape systems. SUMS will manage 200TB cache disk with 150TB/year of permanent archive for level-1 and higher level products. SUMS archive will use LTO-4 tapes in a robotic system with at least 10 drives and 2000 tapes near-line. The pipeline processing system will have about 50 processor cores dedicated to level-0 to level-1 processing and about 450 cores for higher level processing in the pipeline. HMI standard products will need about half of these.

Page 10LWS Teams Day JSOC Overview Extra Info

Page 11LWS Teams Day JSOC Overview Data Record Management Service (DRMS) Data Record Management Service (DRMS) Pipeline client-server architecture JSOC Disks Analysis code C/Fortran/IDL/Matlab JSOC Library Record Cache (Keywords+Links+Data paths) OpenRecords CloseRecords GetKeyword, SetKeyword GetLink, SetLink OpenDataSegment CloseDataSegment Pipeline client process JSOC Disks Data Record Management Service (DRMS) Storage Unit Management Service (SUMS) Tape Archive Service Series Tables Record Catalogs Storage Unit Tables Record Catalogs Record Tables PostgreSQL Database Server AllocUnit GetUnit PutUnit SQL queries Storage unit transfer Data Segment I/O Storage unit transfer Generic file I/O DRMS socket protocol

Page 12LWS Teams Day JSOC Overview Pipeline batch processing A pipeline batch is encapsulated in a single database transaction, “DRMS session”: –If no module fails all data records are commited and become visible to other clients of the JSOC catalog at the end of the session –If failure occurs all data records are deleted and the database rolled back –It is possible to commit data produced up to intermediate checkpoints during sessions DRMS Service = Session Master Input data records Output data records DRMS API Register session DRMS API Module 1 DRMS API Module 2.2 … DRMS API Module N DRMS API Commit Data & Deregister Pipeline batch = atomic transaction Record & Series Database SUMS DRMS API Module 2.1

Page 13LWS Teams Day JSOC Overview HMI module status and MDI heritage Doppler Velocity Heliographic Doppler velocity maps Tracked Tiles Of Dopplergrams Stokes I,V Continuum Brightness Tracked full-disk 1-hour averaged Continuum maps Brightness feature maps Solar limb parameters Stokes I,Q,U,V Full-disk 10-min Averaged maps Tracked Tiles Line-of-sight Magnetograms Vector Magnetograms Fast algorithm Vector Magnetograms Inversion algorithm Egression and Ingression maps Time-distance Cross-covariance function Ring diagrams Wave phase shift maps Wave travel times Local wave frequency shifts Spherical Harmonic Time series Mode frequencies And splitting Brightness Images Line-of-Sight Magnetic Field Maps Coronal magnetic Field Extrapolations Coronal and Solar wind models Far-side activity index Deep-focus v and c s maps (0-200Mm) High-resolution v and c s maps (0-30Mm) Carrington synoptic v and c s maps (0-30Mm) Full-disk velocity, sound speed, Maps (0-30Mm) Internal sound speed Internal rotation Vector Magnetic Field Maps MDI pipeline modules exist Standalone production codes in use at Stanford Research codes in use by team Codes to be developed at HAO Codes being developed in the community Codes to be developed at Stanford Primary observables Intermediate and high level data products

Page 14LWS Teams Day JSOC Overview AIA Level-2

Page 15LWS Teams Day JSOC Overview JSOC Data Volumes from Proposal

Page 16LWS Teams Day JSOC Overview Sample of DRMS/SUMS Resource Assessment

Page 17LWS Teams Day JSOC Overview JSOC Level-0 processing flow

Page 18LWS Teams Day JSOC Overview JSOC-SDP Status Capture system complete, waiting testing with DDS at White Sands DRMS and SUMS virtually done Level-0 work progressing, to be done by instrument deliveries Level-1 work to start after delivery Getting serious about work on basic pipeline modules Review of status and computer hardware plans in early November

Page 19LWS Teams Day JSOC Overview Stanford JSOC effort plan