1. The SCEC CSEP TESTING Center Operations Review
Philip J. Maechling
Southern California Earthquake Center
3 May 2017
Fabio Silva, John Yu, and collaborators
I’d like to start by describing more details about two phases in our title. One is software ecosystem, one is earthquake system science research.

2. CSEP Operational Systems – Computers and Storage
CSEP Computer Hardware: Multiple Computers and Storage Systems:
Operational image server a (full data archive):
Certification image server b (partial data archive):
Debug image server c (full data archive):
Debug2 image server d (partial data archive) :
Development image server d (partial data archive):
Publication image server e (partial data archive):
New Zealand image server f (backup data archive for NZ Testing Center) :
Seismological Society of America Meeting, Denver, April

3. CSEP Operational Systems –Storage
CSEP Data Storage: Operational Data, Log, and Results
Scientific codes and binaries
Earthquake catalogs
Forecast results
Evaluations results
Runtime log files
Special studies results
SVN Code repository
~10TB Data (Start Date: End Date: )
Data backup offline
Seismological Society of America Meeting, Denver, April

4. CSEP Data Storage Usage – Dec 2016:
Forecasts, Catalogs, Evaluations:
2.5 /home/csep/operations/SCEC-natural-laboratory
/home/csep/operations/testing-regions/Global
/home/csep/operations/testing-regions/GlobalPDE
/home/csep/operations/testing-regions/SWPacific
/home/csep/operations/testing-regions/NWPacific
/home/csep/operations/testing-regions/OceanicTransformFaults
2.7 /home/csep/operations/testing-regions-2/Global
1.7 /home/csep/operations/testing-regions-2/one-day-model-archives
7.68T Forecast Archive Storage Total
Runtime Logs:
/home/csep/operations/dispatcher/runs/csep
/home/csep/operations/dispatcher/runs/Global/csep
/home/csep/operations/dispatcher/runs/NWPacific/csep
/home/csep/operations/dispatcher/runs/OceanicTransformFaults/csep
/home/csep/operations/dispatcher/runs/SWPacific/csep
1.5 /home/csep/operations/testing-regions2/dispatcher
Saved Catalogs:
/home/csep/operations/catalogs/ANSS
/home/csep/operations/catalogs/CMT
Source / Buildlogs / Binaries v16.10
/usr/local/csep
/home/csep/buildCSEP
northridge:/work/source/svn/csep
------
2.32T + (7.68T) = 10.00T
Seismological Society of America Meeting, Denver, April

5. CSEP Testing Center Software Framework
CSEP Earthquake Forecast Testing Center Framework (distributed to external testing centers):
Earthquake Catalog retrieval and filtering codes
Python implementations of forecast and evaluation methods
Object model for forecast and evaluations
Dispatcher software framework schedules forecast and evaluation updates
V16_10_0 was the 40th CSEP software release since Sept
CSEP Testing Center Software Used At:
California Testing Center – USC
New Zealand Testing Center - GNS
European Testing Center - ETH
China Testing Center -
Japan Testing Center -
Seismological Society of America Meeting, Denver, April

6. SCEC California Region CSEP Testing Center
SCEC CSEP Testing Center California Region and Global Regions:
SCECNaturalLab (California)
Global
GlobalPDE
SWPacific
NWPacific
OceanicTransformFaults
Seismological Society of America Meeting, Denver, April

7. CSEP Operational Testing Center Updates:
CSEP Operational software modified through public software releases. The last CSEP release was Oct 2016.
There are processing errors with some of the CSEP V16_10_0 forecasts. Currently, SCEC CSEP runs 19 dispatcher scripts and 4 are failing consistently. That means we need to fix CSEP V16.10 and make a new CSEP release.
Seismological Society of America Meeting, Denver, April

8. CSEP Processing Status
(1) one-day forecasts (dispatcher_ANSS1985_forecasts.tcsh)
(2) five-year models (dispatcher_ANSS1985.tcsh)
(3) one-day models 1985 (dispatcher_ANSS1985_one_day.tcsh) - Fail
(4) 3-month and 5-year models 1932 (dispatcher_ANSS1932.tcsh)
(5) 1-day 3-month alarm models 1932 (dispatcher_ANSS1932_notFiltered.tcsh)
(6) 30-minute models 1985 (batch_ANSS1985_30min.tcsh) - Fail
(7) 30-minute models 1985 T/W tests (batch_ANSS1985_30min_TWTests.tcsh)
(8) one-day models synthetic catalogs 1985 Md2.95 (dispatcher_ANSS1985_M2_95.tcsh) - Fail
(9) one-day models not filtered 1932 using Md2 (dispatcher_ANSS1932_notFiltered_Md2_one_day.tcsh)-Fail
(10) California models (not filtered) 1932 using Md2 (dispatcher_ANSS1932_notFiltered_Md2.tcsh)
(11) NW Pacific 1-year (dispatcher_one_year.tcsh)
(12) NW Pacific 1-day (dispatcher_daily.tcsh)
(13) SW Pacific 1-year (dispatcher_one_year.tcsh)
(14) SW Pacific 1-day (dispatcher_daily.tcsh)
(15) Global 1-year (dispatcher_one_year.tcsh)
(16) Global 0.1-degree 1-year (dispatcher_one_year_V12.1.tcsh)
(17) Global 0.1Degree 1-year (dispatcher_one_year_V14.1.tcsh)
(18) Global 0.1Degree 1-day (dispatcher_one_year_0.1degree_V14.4.tcsh)
(19) Oceanic Transform Fault 1-day (dispatcher_daily.tcsh)
Seismological Society of America Meeting, Denver, April

9. CSEP System Development Planning:
We need to update the CSEP operational system to fix the problems in V To make changes to the operational systems, we need to make a new CSEP software release.
Seismological Society of America Meeting, Denver, April

10. CSEP System Development Planning:
CSEP operational system runs all forecasts and evaluation on single server:
SCEC CSEP Testing Center must complete daily forecast processing within 24 hours.
Reprocessing must occur on single server when not in use.
We have incomplete forecast archives for 4 of 19 forecast groups. We can reproduce them, but will take months to catch up.
Seismological Society of America Meeting, Denver, April

11. CSEP System Development Planning:
Expand CSEP operational testing center processing to support multi-processing capabilities:
Next CSEP release will use multi-processing to speed up operational system.
All forecasts will be run in identical virtual software environment.
We will run multiple serial virtual image-based processing where possible.
We will stage consistent results in cumulative data archives.
Seismological Society of America Meeting, Denver, April

12. CSEP System Development Planning:
Curate the forecasts and evaluations completed before new release starts (end of Phase 1):
Review expected results of all Phase 1 forecasts
Evaluate completeness of Phase 1 evaluation results
Reprocess forecast and evaluations as appropriate
Seismological Society of America Meeting, Denver, April

13. CSEP Multi-stage Development Plan
CSEP Testing Center Development Phases:
Phase 1: Single Server Phase (Sept 2007 – Sept 2017)
Phase 2: Multi-Server Phase (Sept 2017 – Sept 2018)
Phase 3: Multi-Server Workflow Phase (Sept 2018 – Sept 2022
Seismological Society of America Meeting, Denver, April

14. CSEP Operations Review
Hiring situation:
Operations person, that will operate and improve the CSEP testing center. Part-time CSEP
CSEP Scientific programmer that implements new forecast methods and evaluation techniques, and serves as technical representative at meetings. Full-time CSEP
Seismological Society of America Meeting, Denver, April

15. Prepare for next Release of CSEP Software:
Currently operation version of CSEP v16.10 will complete the Phase 1 processing. We can assume all Phase 1 processing was done in a consistent way with a consistent codebase:
Preserve source code repo:
Preserve Complete Preliminary Data Set for CSEP Phase 1
Create inventory of expected and current results
Curate Preliminary Data Set by Reprocessing
Next CSEP Release will start the CSEP Multi-server Phase:
Will maintain the goals of transparency, controlled environment, reproducibility.
Give up single server controlled environment. But server images are good way to ensure multiple computers are running the same software. The challenge is the data needed. Possible shared input data, and definite need to recombine results in single archive at end.
Release new version that runs independent forecasts on identical computing environments.
Multi-processor system will do separate computing but transfer results to single testing center prospective archives.
Will introduce any fixes we have for current CSEP operational system.
Future workflow system will:
Define csep data retrieval, forecasts, and evaluations as directed acyclic graphs
Increase automatic error retry
minimize data access and migration
Improve scheduling
Improve data transfer
Support prospective and retrospective testing
Seismological Society of America Meeting, Denver, April

16. CSEP Testing Center Inventory – v16.10
Seismological Society of America Meeting, Denver, April

17. CSEP Testing Center Inventory – v16.10
Seismological Society of America Meeting, Denver, April