GEO-CAPE Team Meeting May 21-22, 2013 Status and Discussion of Science Value and Science Traceability Matrix Doreen Neil With original thinking from many others involved in GEO-CAPE formulation

The SWG defines mission requirements, evaluates implementation options Science Traceability Matrix (STM) Science Value Matrix (SVM) Weiss et al., IEEAC 2004 GEO-CAPE also has an Applications Traceability Matrix (ATM) and corresponding Applications Value Matrix (AVM) STM leads: Doreen Neil. Daniel Jacob SVM leads: Doreen Neil, David Edwards ATM/AVM leads: Jessica Neu, Rob Pinder D. Jacob

GEO-CAPE Atmospheres Science Traceability Matrix 3 Science Questions were developed over 6 months by SWG Working Group. Atmospheres Science Traceability Matrix was: 1.“ratified” at St Petersburg Community Meeting. 2.STM was “re-affirmed” (with minor changes agreed by SWG) at Boulder Community meeting. 3.Published (with minor revisions agreed by SWG) in 2012.

GEO-CAPE Atmospheric Science STM as published in BAMS 2012

AOD=Aerosol optical depth, AAOD=Aerosol absorption optical depth, AI=Aerosol index. The mixing ratio [mole fraction], ppb, of a target gas is number of moles of that gas/mole of air, invariant with temperature and pressure. The number density is the number of molecules of the target gas/unit volume of air; the total column concentrations in the table above are the integral of the number density from the surface to space. 1 Baseline: Measured quantities deliver the full science requirements for GEO-CAPE. 2 Typical column amount. Units are molecules cm -2 for gases and unitless for aerosols, unless specified. Typical AOD and AAOD values are provided for mid- visible wavelengths over North America. 3 Retrieval aerosol height from different techniques, e.g. O2-O2 band at 477 nm, O2-A band at 760 nm, O2-B band at 680 nm. * = background value. Pollution is higher, and in starred constituents, the precision is applied to polluted cases. GEO-CAPE Atmospheric Science STM as published in BAMS 2012 (continued) STM Working Group

GEO-CAPE Science Value 6 A relative valuation approach to allow common assessment of GEO-CAPE full-mission and partial-mission implementation options

GEO-CAPE Science Value Metrics 7 Science Impact (the potential to meet STM requirements) = S Science Expectation = S * P * R Science Value = Science Expectation / C CriteriaValuation S Science Impact: Completeness of accomplishing GEO-CAPE science measurements as defined in the STMs (include uniqueness here). SWGs define appropriately so different concepts may be evaluated. Full STM value = 100% P Programmatic Impact: Is compatible with NASA strategic plans and programmatic constraints (i.e. complementarity and timing/schedule compatibility with other US and international missions) >1 for synergies with other missions R Provides low technical and programmatic risk to NASA (e.g. the greatest likelihood technically for mission success; mission can be successfully implemented as designed within defined constraints). Could include instrument/algorithm risk here) < 1 for higher risks C Full life cycle cost estimate Scale to same- year dollars

Expert opinion gathered through workbooks to assess Contribution of each product toward answering each question 8 Expert 2

9 19% 11% 8% 7% 6% 4% 3% 20% CriteriaValuation S Science Impact: Completeness of accomplishing GEO-CAPE science measurements as defined in the STM (include uniqueness here). Total STM value = 100% Average contribution to all Questions (TEMPO product) Science Questions: Emissions Processes Climate Assessment, forecast Intercontinental Impact Events 21% Average contribution to all Questions (not provided by TEMPO)

10 TEMPO contributes significantly toward an international constellation of air quality measurements: TEMPO has high Programmatic Impact. The CEOS Atmospheric Composition Constellation White Paper defines essential measurements beyond what TEMPO provides. Providing the missing CEOS measurements would have high Programmatic Impact. The definition above suggests that GEO-CAPE products not provided by TEMPO have high programmatic impact if they are concurrent with TEMPO. Programmatic Impact is not a discriminator among adequately mature GEOCAPE instrument concepts that provide the missing measurements. Several EV-class proposals address one or more of the missing measurements. We have set P=1 for this evaluation. CriteriaValuation P Programmatic Impact: Is compatible with NASA strategic plans and programmatic constraints (i.e. complementarity and timing/schedule compatibility with other US and international missions) >1 for synergies with other missions

11 Peer reviewed concepts receive technical, management, and cost risk assessments Concepts with acceptable TRL (TRL>6) have been peer reviewed, including TEMPO. TRL is not a discriminator among these concepts. TMC risk is not a discriminator among selectable proposals (as reviewed, they are low risk). GEO-CAPE SWG developed a Product Confidence Concept. High Product Confidence (5) indicates that the required product can be delivered with low risk. We use the (normalized) Product Confidence to assess product risk. 1  Theoretical sensitivity study 2  Theoretical study with error budget; expectation of meeting GEOCAPE req.* 3  Data products expected in 2-4 years would meet GEOCAPE req.*  Initial algorithm tested with LEO satellite data  Initial retrievals & calculated errors being evaluated with independent data sets and/or modeling results 4  Data products expected in 1-2 years would meet GEOCAPE req.*  Algorithm being used from LEO  Data products using this algorithm are being used for science, (e.g., compared to models, used in process studies, emission inventories)  Validation against independent data sets to identify bias and consistency of calculated vs. actual errors 5  Mature operational data product from LEO meets GEOCAPE req.*  Data products are routinely used for science, (e.g., assimilated for weather/air quality forecasting)  Systematic and precision errors well understood and characterized with reported error covariance  Routine ongoing validation of data products CriteriaValuation R Provides low technical and programmatic risk to NASA (e.g. the greatest likelihood technically for mission success; mission can be successfully implemented as designed within defined constraints). Could include instrument/algorithm risk here) < 1 for higher risks

GEO-CAPE STM Baseline "Product Confidence“ Index V5 12 SpeciesO3O3 CONO 2 HCHO STM Baseline Requirement With 2 pieces of information in the troposphere in daylight with sensitivity to the lowest 2 km; 0-2 km: 10 ppbv; 2km- tropopause: 15 ppbv With 2 pieces of information in the troposphere in daylight with sensitivity to the lowest 2 km; 0-2 km: 20 ppbv; 2km-trop: 20 ppbv COLUMN; 1×10 15, SZA<70 COLUMN ; 1×10 16 ; 3/day, SZA<50 Baseline Spectral Range(s) UV + VIS2 UV + TIR UV + VIS2 + TIR MWIR + SWIRVIS1UV Product Confidence Index Average* “Product Confidence" scores are based on maturity of heritage algorithms and products demonstrated from space SpeciesSO 2 AerosolCH 4 NH 3 CHOCHO STM Baseline Requirement COL; 1×10 16 ; 3/day, SZA<50 AOD 0.05; Hourly; SZA<70 AAOD 0.02; Hourly; SZA<70 AOCH 1 km; Hourly; SZA<70 AI 0.1; Hourly; SZA<70 COL; 20 ppbv; 2/day COL; 0-2 km: 2ppbv; 2/day COL; 4×10 14 ; 2/day, SZA<50 Baseline Spectral Range(s) UVUV + VISSWIRTIRUV Product Confidence Index Average* Product Confidence Index represents an algorithm “risk” (R) in the value framework.

GEO-CAPE Science Expectation 13 A relative valuation approach to allow common assessment of GEO-CAPE full-mission and partial-mission implementation options Science Impact (the potential to meet STM requirements) = S Science Expectation = S * P * R Science Value = Science Expectation / C CriteriaValuation S Science Impact: Completeness of accomplishing GEO-CAPE science measurements as defined in the STMs (include uniqueness here). SWGs define appropriately so different concepts may be evaluated. Full STM value = 100% P Programmatic Impact: Is compatible with NASA strategic plans and programmatic constraints (i.e. complementarity and timing/schedule compatibility with other US and international missions) >1 for synergies with other missions R Provides low technical and programmatic risk to NASA (e.g. the greatest likelihood technically for mission success; mission can be successfully implemented as designed within defined constraints). Could include instrument/algorithm risk here) < 1 for higher risks CFull life cycle cost estimate Scale to same- year dollars

Science Expectation = S i * P(=1) * R i 14 Results are consistent with Science Impact alone. Some high value measurements are not made by TEMPO. TEMPO Product Product not produced by TEMPO

GEO-CAPE Science Value 15 A relative valuation approach to allow common assessment of GEO-CAPE full-mission and partial-mission implementation options Science Impact (the potential to meet STM requirements) = S Science Expectation = S * P * R Science Value = Science Expectation / Cost CriteriaValuation S Science Impact: Completeness of accomplishing GEO-CAPE science measurements as defined in the STMs (include uniqueness here). SWGs define appropriately so different concepts may be evaluated. Full STM value = 100% P Programmatic Impact: Is compatible with NASA strategic plans and programmatic constraints (i.e. complementarity and timing/schedule compatibility with other US and international missions) >1 for synergies with other missions R Provides low technical and programmatic risk to NASA (e.g. the greatest likelihood technically for mission success; mission can be successfully implemented as designed within defined constraints). Could include instrument/algorithm risk here) < 1 for higher risks CFull life cycle cost estimate Scale to same- year dollars

16 GEO-CAPE Science Value Metrics GEO-CAPE has developed an approach to assess the “value” of different products based on their 1.Contributions toward answering the GEO-CAPE Science Questions 2.Programmatic value in context of what has already been funded 3.Peer-reviewed technical, management, and cost risk of any specific instrument concept 4.Product maturity. 5.Lifecycle cost.

SVM Discussion Topics 17 1.Application of the Science Value metrics to the descope mission described in the published STM 2.Value of the cloud camera