The SCEC CSEP TESTING Center Operations Review Philip J. Maechling Southern California Earthquake Center 3 May 2017 Fabio Silva, John Yu, and collaborators I’d like to start by describing more details about two phases in our title. One is software ecosystem, one is earthquake system science research.

CSEP Operational Systems – Computers and Storage CSEP Computer Hardware: Multiple Computers and Storage Systems: Operational image server a (full data archive): Certification image server b (partial data archive): Debug image server c (full data archive): Debug2 image server d (partial data archive) : Development image server d (partial data archive): Publication image server e (partial data archive): New Zealand image server f (backup data archive for NZ Testing Center) : Seismological Society of America Meeting, Denver, April 18 2017

CSEP Operational Systems –Storage CSEP Data Storage: Operational Data, Log, and Results Scientific codes and binaries Earthquake catalogs Forecast results Evaluations results Runtime log files Special studies results SVN Code repository ~10TB Data (Start Date: 09-01-2007 End Date: 03-30-2017) Data backup offline Seismological Society of America Meeting, Denver, April 18 2017

CSEP Data Storage Usage – Dec 2016: Forecasts, Catalogs, Evaluations: 2.5 /home/csep/operations/SCEC-natural-laboratory 0.576 /home/csep/operations/testing-regions/Global 0.0001 /home/csep/operations/testing-regions/GlobalPDE 0.112 /home/csep/operations/testing-regions/SWPacific 0.091 /home/csep/operations/testing-regions/NWPacific 0.003 /home/csep/operations/testing-regions/OceanicTransformFaults 2.7 /home/csep/operations/testing-regions-2/Global 1.7 /home/csep/operations/testing-regions-2/one-day-model-archives ------- 7.68T Forecast Archive Storage Total Runtime Logs: 0.442 /home/csep/operations/dispatcher/runs/csep 0.062 /home/csep/operations/dispatcher/runs/Global/csep 0.105 /home/csep/operations/dispatcher/runs/NWPacific/csep 0.013 /home/csep/operations/dispatcher/runs/OceanicTransformFaults/csep 0.106 /home/csep/operations/dispatcher/runs/SWPacific/csep 1.5 /home/csep/operations/testing-regions2/dispatcher Saved Catalogs: 0.0033 /home/csep/operations/catalogs/ANSS 0.000075 /home/csep/operations/catalogs/CMT Source / Buildlogs / Binaries v16.10 0.020 /usr/local/csep-16.10.0 0.043 /home/csep/buildCSEP 0.029 northridge:/work/source/svn/csep ------ 2.32T + (7.68T) = 10.00T Seismological Society of America Meeting, Denver, April 18 2017

CSEP Testing Center Software Framework CSEP Earthquake Forecast Testing Center Framework (distributed to external testing centers): Earthquake Catalog retrieval and filtering codes Python implementations of forecast and evaluation methods Object model for forecast and evaluations Dispatcher software framework schedules forecast and evaluation updates V16_10_0 was the 40th CSEP software release since Sept 1 2007 CSEP Testing Center Software Used At: California Testing Center – USC New Zealand Testing Center - GNS European Testing Center - ETH China Testing Center - Japan Testing Center - Seismological Society of America Meeting, Denver, April 18 2017

SCEC California Region CSEP Testing Center SCEC CSEP Testing Center California Region and Global Regions: SCECNaturalLab (California) Global GlobalPDE SWPacific NWPacific OceanicTransformFaults Seismological Society of America Meeting, Denver, April 18 2017

CSEP Operational Testing Center Updates: CSEP Operational software modified through public software releases. The last CSEP release was Oct 2016. There are processing errors with some of the CSEP V16_10_0 forecasts. Currently, SCEC CSEP runs 19 dispatcher scripts and 4 are failing consistently. That means we need to fix CSEP V16.10 and make a new CSEP release. Seismological Society of America Meeting, Denver, April 18 2017

CSEP Processing Status (1) one-day forecasts (dispatcher_ANSS1985_forecasts.tcsh) (2) five-year models (dispatcher_ANSS1985.tcsh) (3) one-day models 1985 (dispatcher_ANSS1985_one_day.tcsh) - Fail (4) 3-month and 5-year models 1932 (dispatcher_ANSS1932.tcsh) (5) 1-day 3-month alarm models 1932 (dispatcher_ANSS1932_notFiltered.tcsh) (6) 30-minute models 1985 (batch_ANSS1985_30min.tcsh) - Fail (7) 30-minute models 1985 T/W tests (batch_ANSS1985_30min_TWTests.tcsh) (8) one-day models synthetic catalogs 1985 Md2.95 (dispatcher_ANSS1985_M2_95.tcsh) - Fail (9) one-day models not filtered 1932 using Md2 (dispatcher_ANSS1932_notFiltered_Md2_one_day.tcsh)-Fail (10) California models (not filtered) 1932 using Md2 (dispatcher_ANSS1932_notFiltered_Md2.tcsh) (11) NW Pacific 1-year (dispatcher_one_year.tcsh) (12) NW Pacific 1-day (dispatcher_daily.tcsh) (13) SW Pacific 1-year (dispatcher_one_year.tcsh) (14) SW Pacific 1-day (dispatcher_daily.tcsh) (15) Global 1-year (dispatcher_one_year.tcsh) (16) Global 0.1-degree 1-year (dispatcher_one_year_V12.1.tcsh) (17) Global 0.1Degree 1-year (dispatcher_one_year_V14.1.tcsh) (18) Global 0.1Degree 1-day (dispatcher_one_year_0.1degree_V14.4.tcsh) (19) Oceanic Transform Fault 1-day (dispatcher_daily.tcsh) Seismological Society of America Meeting, Denver, April 18 2017

CSEP System Development Planning: We need to update the CSEP operational system to fix the problems in V16.10. To make changes to the operational systems, we need to make a new CSEP software release. Seismological Society of America Meeting, Denver, April 18 2017

CSEP System Development Planning: CSEP operational system runs all forecasts and evaluation on single server: SCEC CSEP Testing Center must complete daily forecast processing within 24 hours. Reprocessing must occur on single server when not in use. We have incomplete forecast archives for 4 of 19 forecast groups. We can reproduce them, but will take months to catch up. Seismological Society of America Meeting, Denver, April 18 2017

CSEP System Development Planning: Expand CSEP operational testing center processing to support multi-processing capabilities: Next CSEP release will use multi-processing to speed up operational system. All forecasts will be run in identical virtual software environment. We will run multiple serial virtual image-based processing where possible. We will stage consistent results in cumulative data archives. Seismological Society of America Meeting, Denver, April 18 2017

CSEP System Development Planning: Curate the forecasts and evaluations completed before new release starts (end of Phase 1): Review expected results of all Phase 1 forecasts Evaluate completeness of Phase 1 evaluation results Reprocess forecast and evaluations as appropriate Seismological Society of America Meeting, Denver, April 18 2017

CSEP Multi-stage Development Plan CSEP Testing Center Development Phases: Phase 1: Single Server Phase (Sept 2007 – Sept 2017) Phase 2: Multi-Server Phase (Sept 2017 – Sept 2018) Phase 3: Multi-Server Workflow Phase (Sept 2018 – Sept 2022 Seismological Society of America Meeting, Denver, April 18 2017

CSEP Operations Review Hiring situation: Operations person, that will operate and improve the CSEP testing center. Part-time CSEP CSEP Scientific programmer that implements new forecast methods and evaluation techniques, and serves as technical representative at meetings. Full-time CSEP Seismological Society of America Meeting, Denver, April 18 2017

Prepare for next Release of CSEP Software: Currently operation version of CSEP v16.10 will complete the Phase 1 processing. We can assume all Phase 1 processing was done in a consistent way with a consistent codebase: Preserve source code repo: Preserve Complete Preliminary Data Set for CSEP Phase 1 Create inventory of expected and current results Curate Preliminary Data Set by Reprocessing Next CSEP Release will start the CSEP Multi-server Phase: Will maintain the goals of transparency, controlled environment, reproducibility. Give up single server controlled environment. But server images are good way to ensure multiple computers are running the same software. The challenge is the data needed. Possible shared input data, and definite need to recombine results in single archive at end. Release new version that runs independent forecasts on identical computing environments. Multi-processor system will do separate computing but transfer results to single testing center prospective archives. Will introduce any fixes we have for current CSEP operational system. Future workflow system will: Define csep data retrieval, forecasts, and evaluations as directed acyclic graphs Increase automatic error retry minimize data access and migration Improve scheduling Improve data transfer Support prospective and retrospective testing Seismological Society of America Meeting, Denver, April 18 2017

CSEP Testing Center Inventory – v16.10 Seismological Society of America Meeting, Denver, April 18 2017

CSEP Testing Center Inventory – v16.10 Seismological Society of America Meeting, Denver, April 18 2017