1 of 50 The EPA 7-Step DQO Process Step 7 - Optimize Sample Design 60 minutes Presenter: Sebastian Tindall DQO Training Course Day 3 Module 16

2 of 50 Step Objective: Identify the most resource effective data collection and analysis design that satisfies the DQOs specified in the preceding 6 steps Step 7: Optimize Sample Design Step 4: Specify Boundaries Step 2: Identify Decisions Step 3: Identify Inputs Step 1: State the Problem Step 5: Define Decision Rules Step 6 : Specify Error Tolerances Step 7 : Optimize Sample Design

3 of 50 Terminal Objective To be able to use the output from the previous DQO Process steps to select sampling and analysis designs and understand design alternatives presented to you for a specific project

4 of 50 Information INActions Information OUT From Previous Step To Next Step Select the optimal sample size that satisfies the DQOs for each data collection design option For each design option, select needed mathematical expressions Check if number of samples exceeds project resource constraints Decision Error Tolerances Gray Region Optimal Sample Design Go back to Steps 1- 6 and revisit decisions. Yes No Review DQO outputs from Steps 1-6 to be sure they are internally consistent Step 7- Optimize Sample Design Develop alternative sample designs

5 of 50 Information INActions Information OUT From Previous Step To Next Step Select the optimal sample size that satisfies the DQOs for each data collection design option For each design option, select needed mathematical expressions Check if number of samples exceeds project resource constraints Decision Error Tolerances Gray Region Review DQO outputs from Steps 1-6 to be sure they are internally consistent Step 7- Optimize Sample Design Develop alternative sample designs The outputs should provide information on the context of, requirements for, and constraints on data collection design. Optimal Sample Design Go back to Steps 1- 6 and revisit decisions. Yes No

6 of 50 Optimal Sample Design Go back to Steps 1- 6 and revisit decisions. Yes No Information INActions Information OUT From Previous Step To Next Step Select the optimal sample size that satisfies the DQOs for each data collection design option For each design option, select needed mathematical expressions Check if number of samples exceeds project resource constraints Decision Error Tolerances Gray Region Review DQO outputs from Steps 1-6 to be sure they are internally consistent Step 7- Optimize Sample Design Develop alternative sample designs Based on the DQO outputs from Steps 1-6, for each decision rule develop one or more sample designs to be considered and evaluated in Step 7.

7 of 50 Optimal Sample Design Go back to Steps 1- 6 and revisit decisions. Yes No Information INActions Information OUT From Previous Step To Next Step Select the optimal sample size that satisfies the DQOs for each data collection design option For each design option, select needed mathematical expressions Check if number of samples exceeds project resource constraints Decision Error Tolerances Gray Region Review DQO outputs from Steps 1-6 to be sure they are internally consistent Step 7- Optimize Sample Design Develop alternative sample designs For each option, pay close attention to the Step 4 outputs defining the population to be represented with the data: Sample collection method Sample mass size Sample particle size Etc. Remember: Sampling Uncertainty is decreased when sampling density is increased.

8 of 50 Sampling Probability-samples (P-S), for which sampling errors can be calculated, and, for which the biases of selection and estimation are virtually eliminated or contained within known limits. Judgment-samples (J-S), for which the biases and sampling errors can not be calculated from the sample but instead must be settled by judgment. Deming, W.E., 1950, Some Theory of Sampling, Dover Publications, New York

9 of 50 Sampling Non-Probabilistic selection: When certain constituent elements of the lot to be evaluated have a zero probability of being taken into the sample. Sampling for Analytical Purpose, Pierre Gy, J. Wiley & Sons, 1998; pg 28

10 of 50 Types of P-S Designs n Simple Random n Systematic Grid with random start n Geometric Probability or “Hot Spot” Sampling n Stratified Random –Stratified Simple Random –Stratified Systematic Grid with random start Statistical Methods for Environmental Pollution Monitoring, Richard O. Gilbert, 1987

11 of 50 Simple Random n Definition- choice of sampling location or time is random n Assumptions –Every portion of the population has equal chance of being sampled –Population is “relatively homogeneous” n Limitations: –may not cover area

12 of 50 Simple Random n To generate a simple random design: –Either grid the site - set up equalateral triangles or equal side rectangles and number each grid, use a random number generator to pick the grids from which to collect samples –Randomly select x, y, z coordinates, go to the random coordinates and collect samples

13 of 50 Example - Simple Random Using Coordinates

14 of 50 Systematic Grid, Random Start n Definition-taking measurements at locations or times according to spatial or temporal pattern (e.g., equidistant intervals along a line or grid pattern) n Assumptions –Good for estimating means, totals and patterns of contamination –Good when population is not “relatively homogeneous”

15 of 50 Systematic Grid, Random Start (cont.) n Limitations –Biased results can occur if assumed pattern of contamination does not match the actual pattern of contamination –Inaccurate if have serial correlation n NPDES outfall –Periodic recurring release; time dependent n Groundwater: –seasonal recurrence; water-level dependence

16 of 50 Remember: Start at random location Move in a pre-selected pattern across the site, making measurements at each point Systematic Grid, Random Start (cont.)

17 of 50 Geometric Probability or Hot- Spot Sampling n Uses squares, triangles, or rectangles to determine whether hot spots exist n Finds hot spot, but may not estimate the mean with adequate confidence

18 of 50 n Number of samples is calculated based on probability of finding hot area or geometric probability n Assumptions –Target hot spot has circular or elliptical shape –Samples are taken on square, rectangular or triangular grid –Definition of what concentration/activity defines hot spot is unambiguous Geometric Probability or Hot- Spot Sampling (cont.)

19 of 50 n Limitations –Not appropriate for hot spots that are not elliptical –Not appropriate if cannot define what is hot or the likely size of hot spot Geometric Probability or Hot- Spot Sampling (cont.)

20 of 50 Example Grid for Hot-Spot Sampling

21 of 50 Action Limit = 50.0 ppm Average = 33.8 ppm

22 of 50 n In order to use this approach the decision- makers MUST –Define the size of the hot spot they wish to find –Provide rationale for specifying that size. –Define what constitutes HOT (e.g., what concentration is HOT) –Define the effect of that HOT spot on achieving the release criteria Geometric Probability or Hot- Spot Sampling (cont.)

23 of 50 Stratified Random n Definition-divide population into strata that are “relatively homogeneous” and collect samples in each strata randomly n Attributes –Provides excellent coverage of area –Need process knowledge to create strata –Yields more precise estimate of mean –Typically more efficient then simple random n Limitations –Need process knowledge –Assumes population is “relatively homogenous”

24 of 50 Example - Stratified Simple Random Strata 1 Strata 2

25 of 50 Information INActions Information OUT From Previous Step To Next Step Select the optimal sample size that satisfies the DQOs for each data collection design option For each design option, select needed mathematical expressions Check if number of samples exceeds project resource constraints Decision Error Tolerances Gray Region Review DQO outputs from Steps 1-6 to be sure they are internally consistent Step 7- Optimize Sample Design Develop alternative sample designs 1. Statistical Method/Sample Size Formula 2. Cost Function Optimal Sample Design Go back to Steps 1- 6 and revisit decisions. Yes No

26 of 50 Optimal Sample Design Go back to Steps 1- 6 and revisit decisions. Yes No Information INActions Information OUT From Previous Step To Next Step Select the optimal sample size that satisfies the DQOs for each data collection design option For each design option, select needed mathematical expressions Check if number of samples exceeds project resource constraints Decision Error Tolerances Gray Region Review DQO outputs from Steps 1-6 to be sure they are internally consistent Step 7- Optimize Sample Design Develop alternative sample designs 1. Statistical Method/Sample Size Formula Define suggested method(s) for testing the statistical hypothesis and define sample size formula(e) that corresponds to the method(s).

27 of 50 Optimal Sample Design Go back to Steps 1- 6 and revisit decisions. Yes No Information INActions Information OUT From Previous Step To Next Step Select the optimal sample size that satisfies the DQOs for each data collection design option For each design option, select needed mathematical expressions Check if number of samples exceeds project resource constraints Decision Error Tolerances Gray Region Review DQO outputs from Steps 1-6 to be sure they are internally consistent Step 7- Optimize Sample Design Develop alternative sample designs Perform a preliminary DQA: Generate frequency distribution histogram(s) for each population Select one or more statistical methods that will address the PSQs List the assumptions for choosing these statistical methods List the appropriate formula for calculating the number of samples, n

28 of 50 3 Approaches for Calculating n n “Symmetrical” distribution approach n Skewed distribution approach n FAM/DWP approach –Badly skewed or for all distributions use computer simulation approach e.g., Monte Carlo Show Monte Carlo simulation (Coming Soon!)

29 of 50 How Many Samples do I Need? Begin With the Decision in Mind Optimal Sampling Design Alternative Sample Designs , , ,  Correct Equation for n (Statistical Method) Population Frequency Distribution Contaminant Concentrations in the Spatial Distribution of the Population The end Data field onsite methods traditional laboratory

30 of 50 Logic to Assess Distribution and Calculate Number of Samples

31 of 50 Logic to Assess Distribution and Calculate Number of Samples

32 of 50 Optimal Sample Design Go back to Steps 1- 6 and revisit decisions. Yes No Information INActions Information OUT From Previous Step To Next Step Select the optimal sample size that satisfies the DQOs for each data collection design option For each design option, select needed mathematical expressions Check if number of samples exceeds project resource constraints Decision Error Tolerances Gray Region Review DQO outputs from Steps 1-6 to be sure they are internally consistent Step 7- Optimize Sample Design Develop alternative sample designs 2. Cost Function For each selected sample size, develop a cost function that relates the number of samples to the total cost of sampling and analysis.

33 of 50 Optimal Sample Design Go back to Steps 1- 6 and revisit decisions. Yes No Information INActions Information OUT From Previous Step To Next Step Select the optimal sample size that satisfies the DQOs for each data collection design option For each design option, select needed mathematical expressions Check if number of samples exceeds project resource constraints Decision Error Tolerances Gray Region Review DQO outputs from Steps 1-6 to be sure they are internally consistent Step 7- Optimize Sample Design Develop alternative sample designs In order to develop the cost function, the aggregate unit cost per sample must be determined. This is the cost of collecting one sample and conducting all the required analyses for a given decision rule.

34 of 50 AUSCA$ = USC$ +  USA$ i Where (here): USC$ = Unit Sample Collection Cost USA$ = Unit Sample Analysis Cost AUSCA$ = Aggregate Unit Sample Collection and Analysis Cost j = Number of analytical methods planned Aggregate Unit Sampling and Analysis Cost i=1 j

35 of 50 Information INActions Information OUT From Previous Step To Next Step Select the optimal sample size that satisfies the DQOs for each data collection design option For each design option, select needed mathematical expressions Check if number of samples exceeds project resource constraints Decision Error Tolerances Gray Region Review DQO outputs from Steps 1-6 to be sure they are internally consistent Step 7- Optimize Sample Design Develop alternative sample designs Using the formulae appropriate to these methods, calculate the number of samples required, varying ,  for a given . Repeat the same process using new  s. Review all of calculated sample sizes and along with their corresponding levels of , , and . Optimal Sample Design Go back to Steps 1- 6 and revisit decisions. Yes No

36 of 50 Optimal Sample Design Go back to Steps 1- 6 and revisit decisions. Yes No Information INActions Information OUT From Previous Step To Next Step Select the optimal sample size that satisfies the DQOs for each data collection design option For each design option, select needed mathematical expressions Check if number of samples exceeds project resource constraints Decision Error Tolerances Gray Region Review DQO outputs from Steps 1-6 to be sure they are internally consistent Step 7- Optimize Sample Design Develop alternative sample designs Merge the selected sample size outputs with the Aggregate Unit Sample Collection and Analysis cost output. This results in a table that shows the product of each selected sample size and the AUSCA$. This table is used to present the project managers and decision makers with a range of analytical costs and the resulting uncertainties. From the table, select the optimal sample size that meets the project budget and uncertainty requirements.

37 of 50 Information INActions Information OUT From Previous Step To Next Step Select the optimal sample size that satisfies the DQOs for each data collection design option For each design option, select needed mathematical expressions Check if number of samples exceeds project resource constraints Decision Error Tolerances Gray Region Review DQO outputs from Steps 1-6 to be sure they are internally consistent Step 7- Optimize Sample Design Develop alternative sample designs Observe the uncertainty a given cost (and number of samples) will “buy”. Select those sample sizes that have acceptable levels of , , and  (uncertainty) associated with them. If this will exceed the project sampling and analysis budget: -Increase the budget -Relax the error tolerances (accept greater uncertainty) Optimal Sample Design Go back to Steps 1- 6 and revisit decisions. Yes No

38 of 50 SHOW SCA EXCEL File

39 of 50 Optimal Sample Design Go back to Steps 1- 6 and revisit decisions. Yes No Information INActions Information OUT From Previous Step To Next Step Select the optimal sample size that satisfies the DQOs for each data collection design option For each design option, select needed mathematical expressions Check if number of samples exceeds project resource constraints Decision Error Tolerances Gray Region Review DQO outputs from Steps 1-6 to be sure they are internally consistent Step 7- Optimize Sample Design Develop alternative sample designs If no sample design meets the error tolerances within the budget: relax one or more of the constraints or request more funding, etc.

40 of 50 Steps 1- 6 Step 7 Optimal Design Iterative Process

41 of 50 Information INActions Information OUT From Previous Step To Next Step Select the optimal sample size that satisfies the DQOs for each data collection design option For each design option, select needed mathematical expressions Check if number of samples exceeds project resource constraints Decision Error Tolerances Gray Region Review DQO outputs from Steps 1-6 to be sure they are internally consistent Step 7- Optimize Sample Design Develop alternative sample designs Justification for a judgmental sampling design Timeframe Qualitative consequences of an inadequate sampling design (low, moderate, severe) Re-sampling access after decision has been made (accessible or inaccessible) Optimal Sample Design Go back to Steps 1- 6 and revisit decisions. Yes No

42 of 50 Sampling Probability-samples, for which sampling errors can be calculated, and, for which the biases of selection and estimation are virtually eliminated or contained within known limits. Judgment-samples, for which the biases and sampling errors can not be calculated from the sample but instead must be settled by judgment. Deming, W.E., 1950, Some Theory of Sampling, Dover Publications, New York

43 of 50 Sampling Non-Probabilistic selection: When certain constituent elements of the lot to be evaluated have a zero probability of being taken into the sample. Sampling for Analytical Purpose, Pierre Gy, J. Wiley & Sons, 1998; pg 28

44 of 50 WARNING!! If a judgmental design is selected in lieu of a statistical design the following disclaimer must be stated in the DQO Summary Report: “Results from a judgmental sampling design can only be used to make decisions about the locations from which the samples were taken and cannot be generalized or extrapolated to any other facility or population, and error analysis cannot be performed on the resulting data. Thus, using judgmental designs prohibits any assessment of uncertainty in the decisions.”

45 of 50 Information INActions Information OUT From Previous Step To Next Step Select the optimal sample size that satisfies the DQOs for each data collection design option For each design option, select needed mathematical expressions Check if number of samples exceeds project resource constraints Decision Error Tolerances Gray Region Review DQO outputs from Steps 1-6 to be sure they are internally consistent Step 7- Optimize Sample Design Develop alternative sample designs The output is the most resource-effective design for the study that is expected to achieve the DQOs. Optimal Sample Design Go back to Steps 1- 6 and revisit decisions. Yes No

46 of 50 Data Quality Assessment Guidance for Data Quality Assessment, EPA QA/G9, 2000 Step 1: Review DQOs and Sampling Design n Step 2: Conduct Preliminary Data Review n Step 3: Select the Statistical Test n Step 4: Verify the Assumptions of the Test n Step 5: Draw Conclusions From the Data

47 of 50 need Statistical Support: To succeed in a systematic planning process for environmental decision making, you need Statistical Support: qualified statisticians environmental data collection designsstatistical data quality assessments One or more qualified statisticians, experienced in environmental data collection designs and statistical data quality assessments of such designs. Summary

48 of 50 Summary (cont.) Going through the 7-Step DQO Process will ensure a defensible and cost effective sampling program n In order for the 7-Step DQO Process to be effective: –Senior management MUST provide support –Inputs must be based on comprehensive scoping and maximum participation/contributions by decision makers –Sample design must be based on the severity of the consequences of decision error –Uncertainty must be identified and quantified

49 of 50 Information INActions Information OUT From Previous Step To Next Step Select the optimal sample size that satisfies the DQOs for each data collection design option For each design option, select needed mathematical expressions Check if number of samples exceeds project resource constraints Decision Error Tolerances Gray Region Review DQO outputs from Steps 1-6 to be sure they are internally consistent Step 7- Optimize Sample Design Develop alternative sample designs Optimal Sample Design Go back to Steps 1- 6 and revisit decisions. Yes No

50 of 50 End of Module 16 Thank you Questions?