This research was made possible by a grant from BP/GOMRI to the Deep-C Consortium and a grant from Bureau of Ocean Energy, Management, Regulation, and Enforcement (BOEM)

Model Skill Assessment – Quantification of (Dis)Similarity Quantitative assessment: Model-model or model-data comparison Tracking divergence of numerical solution in time Assessment of model sensitivity Model adjustment Model1Model2Model3Model4 Ranking (Skills) Model5 Parrish and Derber, 1992 Model Adjustment: Variational Approach Best Worst Model1 Model2Model3 Model4Model5 Control

Some Metrics: Statistical Approach Root Mean Square Deviation (Weighted) Mean Displacement

Skill Assessment: A Topological Approach Hausdorff Distance (HD) Modified Hausdorff Distance (MHD)

Testing Skill Metrics Attributes that are considered significant to the shape of the sea ice fields: Scale Translation Rotation A skill metric should also be resistant to noise To quantify similarity in shape, a metric needs to be able to penalize differences in scale, translation, and rotation

Scale Test Rotation Test Translation Test: MHD Translation Test: HD Translation Test: RMSD

Application of Validation Metrics to the Sea Ice Fields from the 1/12  HYCOM-CICE Model Domain and Grid Resolution (km)  ARCc0.08: Coupled HYbrid Coordinate Ocean Model and Los Alamos Sea Ice Model (CICE 4.0)  32 vertical ocean levels  Atlantic and Pacific Boundaries at ~39°N Closed (no-ice) in CICE Nested into 1/12° Global HYCOM

Definition of the Shape Boundary The 80% concentration contour that is: The longest Includes the North Pole Closed (including the coast line) The 80% Contour Lines The blue line: Selected contour that defines the shape boundary

Differences in the Sea Ice Concentration between the Model Experiments and the Control Run (NCPER) (CCMP) (ASR) (Greenl)

Time Series of the Skill Metric Scores 051 (Gr) 030 (CCMP) 040 (ASR) 020 (NCEPR) 1 Jan11 Jan21 Jan31 Jan10 Feb20 Feb1 Jan11 Jan21 Jan31 Jan10 Feb20 Feb1 Jan11 Jan21 Jan31 Jan10 Feb20 Feb 1 Jan11 Jan21 Jan31 Jan10 Feb20 Feb 1 Jan11 Jan21 Jan31 Jan10 Feb20 Feb Hausdorff Distance (Shape) Modified Hausdorff Distance (Shape) Modified Hausdorff Distance (Shape + Concentration) Mean Displacement (Shape) Weighted Mean Displacement (Shape + Concentration) Anticipated Ranking Best Worst Best Worst ? ? Jan 12 Feb 19

80% Concentration Contours in the Model Experiments 051 (Gr) 030 (CCMP) 040 (ASR) 020 (NCEPR) 011 (Contr.) February 18, 2006February 19, (Gr) 030 (CCMP) 040 (ASR) 020 (NCEPR) 011 (Contr.)

Concentration Whole Field Sea Ice Concentration in the Control Run February 19, 2006 Concentration Inside the Shape Boundary February 19, 2006

Summary  Several metrics have been considered for a quantitative skill assessment of sea ice models (Mean Displacement, Weighted Mean Displacement, Hausdorff Distance, Modified Hausdorff Distance).  All metrics demonstrate the ability to identify differences in the shape of the sea ice fields and sea ice concentration among the model experiments.  The HD and Mean Displacement metrics are noisy due to their higher sensitivity to outliers compared to other metrics. This may cause problems in accurate ranking of the analyzed fields.  The HD and MHD methods ranked all experiments close to expectation. The MHD metrics provide the most accurate and robust ranking.  All metrics correctly identified two cases of distinct differences in the sea ice fields among the models, approximated by the 80% concentration contours.  Selecting an accurate and robust definition of the sea ice shape boundary is critical for the skill assessment and seems to be a challenging problem.

Testing MHD: Facial Recognition Test “Control Run”

MHD Ranking of the Facial Images Rank 1Rank 2Rank 3Rank 4Rank 5Rank 6Rank 7 Closest resemblance

Model Adjustment: Variational Approach Selecting a set of parameters (p) for a model by solving the variational optimization problem Form a cost (convex) function J based on validation metrics Model state vector (function of parameters) True model state (approximated by observations) Parrish and Derber, 1992 Schematic representation of the variational cost-function minimization Minimize J by finding the optimal analysis (X m = X a ) with the corresponding set of parameters Model Runs Validation Metrics Cost Function (J) Minimization Parameter vector (p) Cost-function minimization algorithm

MHD vs Translation Testing Modified Hausdorff Distance (MHD) MHD vs Scale MHD vs Rotation 180  90  270  360  00 Attributes that are considered significant to the shape of the sea ice fields: Scale Translation Rotation A skill metric should also be resistant to noise HD & MHD vs Noise MHD HD