Uncertainty Subgroup ARB Expert Workgroup June 17,

Michael O’Hare, chair Wes Ingram, ARB staff Paul Hodson Stephen Kaffka Keith Kline Michelle Manion Richard Nelson Mark Stowers With assistance from Richard Plevin, UCB 2

Uncertainty and ILUC How has uncertainty about the ‘real value’ of ILUC resulting from substitution of biofuel B for (fossil or other) fuel F been described? Given a characterization (PDF function, range, table of estimate values, etc.) ‘how big’ is the uncertainty in a given estimate? Across all estimates? How can it be reduced? How should LCFS implementation accommodate it? Cost of error Objective function 3

Implementation Estimate physical GWI for MJ/MJ substitution of fuels GHG emissions Analyze optimal operational GWI and choose (implement) Fuel system responds with blends and driving behavior; nature “responds” with real GWI Social cost (  T etc.) 4

Three distinct variables for a single fuel g* ij = estimate of “physical” GWI of fuel i from model j  i = real “physical” GWI: if fuel i is substituted for fuel k on a MJ/MJ basis, additional GHG release is (  i -  k ).  i is a random variable. g i = operational GWI of fuel i used in LCFS implementation 5

Sources of uncertainty Errors in assumed conceptual frameworks: relationships between biofuel policy and land use change Errors and variety in model structure Errors in model parameters Intrinsic variation in real-world variables Are new model results converging on a narrow range of values? 6

Uncertainty in existing models Modelers’ uncertainty reporting Variation among model results Random error vs. bias Can a probability distribution be constructed? 7

Estimates of ILUC emissions from corn ethanol

StudyTarget Year Shock size (10 9 L) ILUC Estimate (g CO 2 MJ -1 ) ILUC Range (g CO 2 MJ -1 ) Searchinger et al – 200 a Hertel et al b – 90 c Dumortier et al /201930n/a21 – 118 d USEPA – 104 e USEPA – 76 e USEPA – 45 e Al-Riffai et al f – 53 g CARB b – 44 Tyner et al variable-14 – 23 Plevin et al. (95% interval) – 140 Plevin et al. (full range) – 340

a. Calculated from reported sensitivity results. b. Analysis was performed using the GTAP-6 database, based on 2001 data, but the results were adjusted post facto to account for the 10% greater average corn yield in c. Range is based on a combination of high and low values for various uncertain economic model parameters. d. Range is based on evaluating alternative model assumptions. e. Range is 95% CI around mean considering only the uncertainty in satellite data analysis and carbon accounting. f. Analysis was performed using the GTAP-7 database, based on 2004 data, using the model to project out to g. Effect of additional 10 6 GJ after meeting 5.6% mandate. Higher value is for greater trade liberalization. Notes about prior table 10

Economic model uncertainty dominates Source: Plevin, O’Hare et al

EPA Uncertainty Analysis Soybean biodiesel N.B. Does not include uncertainties related to economic modeling or production period. 12

Large, but nebulous uncertainty “The lesson for policymakers is that results from economic models depend heavily on assumptions, and because we are trying to predict long-run human behavior, there can be legitimate differences in these assumptions.” – Dumortier et al “[T]his modeling project has demonstrated how the current limits to data availability create significant uncertainty regarding outcomes predicted by these policy simulations.” – Al-Riffai et al. (IFPRI) 2010 [O]ur experience with modeling, data, and parameter estimation and assumptions leads us to conclude that one cannot escape the conclusion that modeling land use change is quite uncertain. Of course, all economic modeling is uncertain, but it is important to point out that we are dealing with a relatively wide range of estimation differences. – Tyner et al

Uncertainty characterization is inconsistent and incomplete Searchinger et al.: Halving and doubling average emissions per hectare, other local SA Dumortier et al.: different FAPRI versions; crop yields; GREET vs BESS; w/wo forest conversion Hertel et al.: Systematic Sensitivity Analysis on few parameters, with notable limitations. USEPA: Uncertainty analysis of remote sensing and emission factors. CARB: OAT sensitivity analysis 14

Reducing uncertainty Better models More models Fewer models Better data for models Selection of “best” models Most of this work is being done in other subgroups Uncertainty will not be reduced to zero 15

Policy and implementation challenge The LCFS requires publishing a GWI with infinite precision for each fuel, but our knowledge of the ‘real’ GWI is imperfect What is the best GWI to use within that framework? 16

Recall: three distinct variables for a single fuel g* ij = estimate of “physical” GWI of fuel i from model j  i = real “physical” GWI: if fuel i is substituted for fuel k on a MJ/MJ basis, additional GHG release is (  i -  k ).  i is a random variable. g i = operational GWI of fuel i used in LCFS implementation 17

Optimal value for GWI Elements of choice Action: Implement a set of values g i for fuel GWI’s Choose objective Factors: Distribution of  i (real values): data is {g* ij } System response R({  i }, {g i }) Cost of ‘errors’  i – g i, R ≠ Ρ (real response) The most likely value of  i is not necessarily the optimal value of g i (safety factor principle). 18

Optimal g i ’s Classical rule: Max {g} Expectation of value 19 where f and h are distributions of  and R

20

What is the goal of the LCFS? Climate – Minimize difference  i – g i ? – Minimize  T at a date (end of LCFS?) – Minimize additional GHG in atmosphere at a date (end of LCFS?) – Minimize total forcing up to a date? – Minimize probability that a condition will fail (LCFS increases one of the above)? Other – Minimize social cost (including cost of  T, food effects, socioeconomic/welfare effects…?) – As of when (discounted how?) 21

Why might the cost of error be asymmetric? Could it be better/worse for  i – g i to be +10 than -10? (Using too much/too little biofuel) ILUC is irreversible even if biofeedstock production stops. Encourage/obstruct advanced biofuels industry. Food effects from overuse of biofuel 22

Why might the cost of error be non-linear? Catastrophic events (Greenland ice cap, Gulf stream, peat feedback, etc.) Socio-economic effects Other costs may be non-linear even if symmetric: – Encourage/obstruct advanced biofuels industry. – Food effects from overuse of biofuel 23

Recognition of uncertainty entails attention to cost of error Asymmetric distribution of  i means E(  i ) is not necessarily the optimal g i. This means the cost of error (functional form and asymmetry) also affects choice of g i 24

25

Uncertainty-explicit policy Few examples to learn from in fuel policy More theory, few examples in climate policy Many examples in health and safety regulation – Safety factor concept – Risk aversity for large costs 26

PolicyMetric Metric Uncertainty Proxy Uncertainty Treatment of Uncert. Acid rain program Tons SO 2 and NO X Low Conservative assumptions PM 2.5 standards Measured PM 2.5 conc. LowMedium 98 th percentile value used Kyoto protocol Direct CO 2 eLowMedium Error propagation recommended RFS2 Life cycle CO 2 e High Partial uncertainty analysis LCFS Life cycle CO 2 e High None

How much investment in reducing uncertainty? --LUC is a complex social and biophysical phenomenon with significant variation across space and time. --Validating the role of any one factor/driver of LUC is difficult, at best, and very information-intensive. --So, even with substantial additional investments in GTAP, some uncertainties surrounding current iLUC estimates are irreducible. 28

“...in many situations, limitations of data, scientific understanding, and the predictive capacity of models will make (uncertainty) estimates unavailable, with the result that they must be supplemented with other sources of information." --Morgan et al. (2010), "Best Practices for Characterizing, Communicating, and Incorporating Scientific Uncertainty in Climate Decisionmaking." "Expert elicitation," i.e., a formal assessment by individual experts, based on the full range of scientific evidence, of their best judgement of a subjective probability distribution for the value in question. --Expert elicitations have been used to address many questions of climate science (e.g., role of aerosols). --While not a substitute for further research, expert elicitation can allow for a formal expression of diversity of opinion not fully reflected in the literature 29

Extending policy scope The greatest uncertainty and largest GHG effects are associated with a few landscape types/areas A separate policy restricting use of such landscapes for economic purposes may reduce uncertainty and the risk of policy failure more than improved analysis and modeling. 30

Expert elicitation --Experts could review full suite of information defining the influence of biofuels on iLUC, not only from GTAP, but including other models (FASOM/FAPRI), databases, and studies from other disciplines --In lieu of quantitative values for iLUC, experts could recommend adaptive decision rules/guidance for categorizing feedstocks and fuel pathways according to relative iLUC risk, based on an assessment of all available information 31

Recommendations Model estimates should present complete, systematic uncertainty characterization Expert elicitation may help infer distribution of  i from {g* ij } CARB should establish objectives Cost of error should be examined and estimated CARB should implement g i with explicit attention to cost of error and distribution of  i CARB should explore mechanisms to incorporate estimates from multiple models in determining g i. 32