1. 1 of 36 The EPA 7-Step DQO Process Step 6 - Specify Error Tolerances (60 minutes) (15 minute Morning Break) Presenter: Sebastian Tindall DQO Training Course Day 3 Module 17

2. 2 of 36 Terminal Course Objective To be able to define, for a specific project: 1. The variability for each COPC 2. the decision errors, 3. the consequences of the errors, 4. the null hypothesis, 5. the lower bound of the gray region

3. 3 of 36 Step Objective: n To specify the decision makers’ tolerable limits on decision errors, which are used for limiting uncertainty in the data –Since analytical data can only provide an estimate the true condition of a site, decisions that are based on such data could potentially be in error Step 6: Specify Error Tolerances Step 4: Specify Boundaries Step 2: Identify Decisions Step 3: Identify Inputs Step 1: State the Problem Step 5: Define Decision Rules Step 6 : Specify Error Tolerances Step 7 : Optimize Sample Design

4. 4 of 36 Decision Error Tolerances Bounds of the Gray Region Assign probability limits on either side of the gray region Information INActions Information OUT From Previous Step To Next Step Decision Rules Step 5 Determine the variability of the environmental variables Step 6- Specify Error Tolerances Choose the null hypothesis Identify the decision errors Specify the boundaries of the gray region

5. 5 of 36 Decision Error Tolerances n The goal of the planning team is to develop a data collection design that reduces the chance of making a decision error to a tolerable level n Step 6 provides a mechanism for allowing the decision maker to define tolerable limits on the probability of making a decision error

6. 6 of 36 Two Reasons Why Decision Makers Make Decision Errors n Sampling error occurs because the sampling design is unable to capture and control the complete extent of heterogeneity that exists in the true state of the environment n Measurement error occurs because analytical methods and instruments are not absolutely perfect

7. 7 of 36 Decision Error Tolerances Bounds of the Gray Region Assign probability limits on either side of the gray region Information INActions Information OUT From Previous Step To Next Step Decision Rules Step 5 Determine the variability of the environmental variables Choose the null hypothesis Identify the decision errors Specify the boundaries of the gray region In order to calculate the number of samples needed (in DQO Step 7), an estimate of the population standard deviation is needed for each environmental variable. Compile a list of the “driver” COPCs Use existing data (must pass Step 3 data assessments) Establish the range based on historical information – Existing data – Process knowledge – Professional judgment Estimate of the population standard deviation – Reference source – Method of calculating Step 6- Specify Error Tolerances

8. 8 of 36 Decision Error Tolerances Bounds of the Gray Region Assign probability limits on either side of the gray region Information INActions Information OUT From Previous Step To Next Step Decision Rules Step 5 Determine the variability of the environmental variables Choose the null hypothesis Identify the decision errors Specify the boundaries of the gray region In order to calculate the number of samples needed (in DQO Step 7), an estimate of the population standard deviation is needed for each environmental variable. Compile a list of the “driver” COPCs Use existing data (must pass Step 3 data assessments) Establish the range based on historical information – Existing data – Process knowledge – Professional judgment Estimate of the population standard deviation – Reference source – Method of calculating Estimate the standard deviation by using the Deming approach of dividing the range by 2 or 3, depending on the frequency distribution. Step 6- Specify Error Tolerances

9. 9 of 36 Estimated Standard Deviations

10. 10 of 36 Estimated Standard Deviations CS

11. 11 of 36 Decision Error Tolerances Bounds of the Gray Region Assign probability limits on either side of the gray region Information INActions Information OUT From Previous Step To Next Step Decision Rules Step 5 Determine the variability of the environmental variables Choose the null hypothesis Identify the decision errors Specify the boundaries of the gray region Define both types of decision error: Determine which one occurs above and which one occurs below the action level. Two Types of Decision Error: Cleaning up a clean site Walking away from a dirty site Step 6- Specify Error Tolerances

12. 12 of 36 Decision Error Tolerances Bounds of the Gray Region Assign probability limits on either side of the gray region Information INActions Information OUT From Previous Step To Next Step Decision Rules Step 5 Determine the variability of the environmental variables Choose the null hypothesis Identify the decision errors Specify the boundaries of the gray region For each Alternative Action: Create a list of possible decision error(s) that may occur if an action is incorrectly taken Discuss the consequences of making each decision error Rate the severity of the consequences of a decision error (i.e., low, moderate, severe) at a point: –Far below the action level –Below but near the action level –Above but near the action level –Far above the action level Indicate which decision error has the most severe consequence near the action level Step 6- Specify Error Tolerances

13. 13 of 36 Decision Error Consequences CS DR # AA # Possible Decision Error Consequences of Decision Error Severity of Consequences of Decision Error When True Parameter Is … (Rated as Low/Moderate/Severe) Decision Error That Has More Severe Consequences Near the Action Level Far Below the Action Level Below But Near the Action Level Above But Near the Action Level Far Above the Action Level 1a 1 1b 1 1: Conduct remedial action Remediating an uncontaminated site Expense and schedule impacts of remediating an uncontaminated site; worker safety SevereModerateNone Not remediating a contaminated site 2: Take no further action Failing to remediate a contaminated site Leaving a site in place the poses a threat to human health and safety None ModerateSevere 2a 2 2b 2 1: Conduct remedial action Remediating an uncontaminated site Expense and schedule impacts of remediating an uncontaminated site; worker safety SevereModerateNone Not remediating a contaminated site 2: Take no further action Failing to remediate a contaminated site Leaving a site in place the poses a threat to human health and safety None ModerateSevere 1 Applies to the depth of 0-6”. 2 Applies to the depth of 6”-10’.

14. 14 of 36 Decision Error Tolerances Bounds of the Gray Region Assign probability limits on either side of the gray region Information INActions Information OUT From Previous Step To Next Step Decision Rules Step 5 Determine the variability of the environmental variables Choose the null hypothesis Identify the decision errors Specify the boundaries of the gray region Provide rationale for rating the severity of consequences as low or severe Step 6- Specify Error Tolerances

15. 15 of 36 Rationale for Error Consequence Ratings CS

16. 16 of 36 Decision Error Tolerances Bounds of the Gray Region Assign probability limits on either side of the gray region Information INActions Information OUT From Previous Step To Next Step Decision Rules Step 5 Determine the variability of the environmental variables Choose the null hypothesis Identify the decision errors Specify the boundaries of the gray region Define the null hypothesis (baseline condition) and the alternative hypothesis: The decision error that has the most adverse potential consequences should be defined as the null hypothesis. The null hypothesis should state the OPPOSITE of what the project hopes to demonstrate. Site is assumed to be contaminated until shown to be clean Site is assumed to be clean until shown to be contaminated Step 6- Specify Error Tolerances

17. 17 of 36 Null Hypothesis Contaminated: H O :  > Action Level Uncontaminated: H A :  < Action Level CS

18. 18 of 36 Decision Error Tolerances Bounds of the Gray Region Assign probability limits on either side of the gray region Information INActions Information OUT From Previous Step To Next Step Decision Rules Step 5 Determine the variability of the environmental variables Choose the null hypothesis Identify the decision errors Specify the boundaries of the gray region The gray region is a range of possible parameter values within which the consequences of a decision error are relatively minor. Step 6- Specify Error Tolerances

19. 19 of 36 Decision Error Tolerances Bounds of the Gray Region Assign probability limits on either side of the gray region Information INActions Information OUT From Previous Step To Next Step Decision Rules Step 5 Determine the variability of the environmental variables Choose the null hypothesis Identify the decision errors Specify the boundaries of the gray region The gray region is bounded on one side by the action level, and on the other side by the parameter value where the consequences of decision error begins to be significant. This point is labeled LBGR, which stands for lower bound of the gray region. Step 6- Specify Error Tolerances

20. 20 of 36 Decision Error Tolerances Bounds of the Gray Region Assign probability limits on either side of the gray region Information INActions Information OUT From Previous Step To Next Step Decision Rules Step 5 Determine the possible range of the parameter of interest Choose the null hypothesis. Identify the decision errors. Specify the boundaries of the gray region Determine the variability of the environmental variables Choose the null hypothesis Identify the decision errors It is necessary to specify the gray region because variability in the population and unavoidable imprecision in the measurement system combine to produce variability in the data such that a decision may be “too close to call” when the true parameter value is very near the action level. Step 6- Specify Error Tolerances

21. 21 of 36 Width of the Gray Region (  ) : UBGR - LBGR or AL - LBGR n  = Analytical + Sampling Error –Estimated based on past data and general knowledge n  = 1/2 of the AL –For each COPC, calculate and set LBGR n  = 10 - 30% of the AL –For each COPC, calculate and set LBGR n  = PDF method –Use PDF for worst COPC to set LBGR

22. 22 of 36 Decision Error Tolerances Bounds of the Gray Region Assign probability limits on either side of the gray region Information INActions Information OUT From Previous Step To Next Step Decision Rules Step 5 Determine the variability of the environmental variables Choose the null hypothesis Identify the decision errors Specify the boundaries of the gray region Present the rationale of how the LBGR was calculated or determined. Step 6- Specify Error Tolerances

23. 23 of 36 Lower Bound of the Gray Region n Because the null hypothesis is that the site is contaminated, the upper bound of the gray region is set equal to the action level n The LBGR should be set at a value where the consequences of the decision error begin to be significant

24. 24 of 36 How to Set the LBGR n LBGR set by the Analytical + Sampling Error n LBGR set to 1/2 Action Level n LBGR set to ~ 50 to 90% of AL (Decision- Maker “whim”) n LBGR set by the Probability Distribution Function (PDF) method UBGR - GR = LBGR AL -  = LBGR

25. 25 of 36 n The LBGR is often based on unavoidable variability in the concentration data –The GR may be estimated based on the precision that the analytical methods allow plus an estimate as to the sampling variance –LBGR = AL – GR (Analytical + Sampling Error) n 100 ppm – (10 ppm + 31 ppm) = 59 ppm n MARSSIM suggests the LBGR be set as: –LBGR = AL – GR (1/2 AL) n 100 ppm – 50 ppm = 50 ppm How to Set the LBGR (cont.)

26. 26 of 36 n The LBGR is often set at some other value –This is based on the decision makers’ choice and is not scientifically based –LBGR = AL – GR (20% of AL); n 100 ppm – 20 ppm = 80 ppm How to Set the LBGR (cont.)

27. 27 of 36 n Use the Frequency Distribution method –The LBGR may be estimated based the Probability Distribution Function (PDF) –Place the Action Level on the mean of the PDF –Ask: “Does a substantial amount of contaminant concentration values exceed the Action Level?” –If yes, begin moving the PDF backwards along the x-axis towards zero concentration –Pause and ask again –When the answer is no, you have set the LBGR (e.g., where the mean of the PDF lies on the x-axis is now the LBGR) Use probability theory to show/prove this How to Set the LBGR (cont.)

28. 28 of 36 Methods for Evaluating the Attainment of Cleanup Standards - Volume 1: Soils and Solid Media EPA, February 1989 PB89-234959 How to set the LBGR  1 is a hypothetical “true mean concentration where the site should be declared clean with a high probability”. (  1 = LBGR)

29. 29 of 36 Show Probability Density Function Distribution Demonstration

30. 30 of 36 Setting the GR for Lead CS

31. 31 of 36 Decision Error Tolerances Bounds of the Gray Region Assign probability limits on either side of the gray region Information INActions Information OUT From Previous Step To Next Step Decision Rules Step 5 Determine the variability of the environmental variables Choose the null hypothesis Identify the decision errors Specify the boundaries of the gray region Assign probability values that reflect the decision maker’s tolerable limits for making an incorrect decision. At the action level (Alpha error) At the other bound of the gray region (Beta error) At a point far below the action level At a point far above the action level Step 6- Specify Error Tolerances

32. 32 of 36 Site is dirtySite is clean 100 True State of Site Alternative Action Walk away from siteClean up site 75 Probability of deciding that the site is dirty 0.0 0.5 1.0 Lower Bound of Gray Region Four Decision-Maker Error Tolerance Locations True mean COPC Concentration Action Level Null Hypothesis: Site is dirty.

33. 33 of 36 Step 6 Summary n Determine the variability for each COPC n Define the two types of error –Incorrectly walking away from a dirty site, or –Incorrectly cleaning a clean site n Evaluate severity of the incorrect decisions both below, above, and near the action level n Select the null hypothesis

34. 34 of 36 Step 6 Summary n Establish a LBGR based on one of the four methods shown previously n Provide the basis for selecting the LBGR n Remember the closer the LBGR is to the action level, the more samples are needed n Assign probability limits on either side of the gray region –Specify the error rates (Alpha and Beta) decision makers are willing to accept and provide rational for the rates

35. 35 of 36 Decision Error Tolerances Bounds of the Gray Region Assign probability limits on either side of the gray region Information INActions Information OUT From Previous Step To Next Step Decision Rules Step 5 Determine the variability of the environmental variables Choose the null hypothesis Identify the decision errors Specify the boundaries of the gray region Step 6- Specify Error Tolerances

36. 36 of 36 End of Module 17 Thank you Questions? We will now take a 15 minute break. Please be back in 15 minutes.