On-Time Product Delivery COPC - HPCC Best Practices March 2011 Allan Darling Deputy Director NCEP Central Operations Where America’s Climate, Weather, Ocean and Space Weather Services Begin

COPC HPCC Best Practices March On-Time Product Delivery

NCEP Mission NCEP delivers science-based environmental predictions to the Nation and the global community. We collaborate with partners and customers to produce reliable, timely, and accurate analyses, guidance, forecasts and warnings for the protection of life and property and the enhancement of the national economy. NCEP Goals and Strategies Information Systems –Enhance the real-time, on-time, all the time access, display and delivery of NCEP products and services. COPC HPCC Best Practices March 20113

On-Time Product Delivery The principle performance metric for NCEP Operational Supercomputing, measured since 1999 Underlying Philosophy Product delivery is the last event in the whole modeling process. To deliver on time, the entire chain of events must work as intended. One Measurement of Operational Success COPC HPCC Best Practices March 20114

Incentives for Capability COPC HPCC Best Practices March 20115

6 On-Time Product Delivery Dual SystemCM & Ops Practice Refinement

Enabling the Capability COPC HPCC Best Practices March System Architecture System Architecture High Availability

Technical Practice Measurement Products are “on time” if they are released within 15 minutes of their assigned target delivery time Target delivery times are based on 30-day average availability times of products Target delivery times are adjusted as needed –Model changes –System changes New products added as part of the model implementation process Timeliness measured for ~720,000 products today COPC HPCC Best Practices March 20118

Technical Practice Measurement Some products are excluded from measurement –Inconsistent delivery times (e.g. on-demand dispersion models) –Not delivered through operational dissemination services Measurement performed daily at 1200Z –Entire previous day –First half of current day COPC HPCC Best Practices March 20119

Operations Practice Daily Meeting to review: –Operations log –Status of open issues –On time delivery metrics –Calendar of planned events Weekly Meeting with HPC vendors to review: –Facility and system status –System utilization –Vendor open issues COPC HPCC Best Practices March

COPC HPCC Best Practices March

COPC HPCC Best Practices March On-Time Product Delivery Dual SystemCM & Ops Practice Refinement

CM Incentive Backup supercomputer implemented, with associated IT infrastructure and requirements –Network between systems –System configuration synchronization –Coordinated model implementations –Failover capability COPC HPCC Best Practices March Expectation – Better Performance Reality – Greater Complexity

Configuration Management Ensure system integrity Weekly meeting to review executed and proposed changes Before change occurs… –Validate and test –Schedule appropriately –Review and approve –Communicate with customers After change occurs… –Identify and communicate outcomes COPC HPCC Best Practices March

Configuration Management Covers all NCO IT practice, not just supercomputers Includes NWS and other partners Full-time staff (primary and backup) Weekly tempo with daily tie-in to operations COPC HPCC Best Practices March

COPC HPCC Best Practices March On-Time Product Delivery Dual SystemCM & Ops Practice Refinement

CM Evolution w/ On-time Feedback COPC HPCC Best Practices March First CM Attempt CM Process Focus CM Refinement

Change Metrics Last 12 months – 15 changes withdrawn out of 1004

Ancillary Benefits Daily review –Identifies performance problems before customers are affected –Reveals silent failures Weekly & Monthly Reviews –Identify system management gaps –Identify model instability COPC HPCC Best Practices March

COPC HPCC Best Practices March On-Time Product Delivery Yearly Average 2006: 99.42% 2007: 99.70% 2008: 99.82% 2009: 99.85% 2010: 99.83% Yearly Average 2006: 99.42% 2007: 99.70% 2008: 99.82% 2009: 99.85% 2010: 99.83%

Questions / Discussion