1. INTRODUCTION TO THE DATA QUALITY OBJECTIVES PROCESS
DISTRIBUTE and REVIEW the Course Agenda.
EXPLAIN: This module is 30 minutes in length.
DISCUSS the following:
Course overview
Course objectives
Questions from participants
EXPLAIN: Scientific terminology used throughout the course will be explained as it appears.

2. Course Objectives
At the conclusion of this course, participants will understand:
The Agency's Quality System and the elements of the DQO Process
How the DQO process applies to EPA programs
How to interpret the consequences of potential decision errors.
REVIEW the course objectives.

3. Systematic Planning
Agency policy requires the use of a systematic planning process to develop performance criteria
DQO Process defines performance and acceptance criteria for decision making
EPA recommends the DQO Process
EXPLAIN: The Agency policy is defined in EPA Order A2 (May 2000).
EXPLAIN: The DQO process is preferred when choosing between opposing conditions or clear alternatives
EXPLAIN: Here are some examples of such situations:
To satisfy a regulation or a permit
To determine if contamination exists.

4. What is the DQO Process? The DQO Process is a systematic
planning process for generating
environmental data that will be sufficient
for their intended use.
DISCUSS the student's knowledge of the DQO process, particularly the concepts of:
Systematic Planning
Sufficient data for use.
EXPLAIN the general purpose of the DQO process: To help to ensure that the generation of environmental data is completed as efficiently as possible.

5. What are DQOs? DQOs are quantitative and qualitative criteria that:
Clarify study objectives
Define appropriate types of data to collect
Specify the tolerable levels of potential decision errors
EXPLAIN the history of the term "Data Quality Objectives" at EPA.
The 7-step process currently used evolved from the earlier DQO Process of the mid 1980's.
Originally, the term DQO was used to describe data quality indicators such as the PARCC parameters:
Precision
Accuracy
Representativeness
Comparability
Completeness.
The first version of the DQO Process, developed in the mid-1980s, involved 3 stages, and steps within stages.
The current 7-step process was developed in the early 1990s and refined into what is now described in the current guidance. DISCUSS the students' knowledge of other terminology used for DQO.

6. DQO Process Planning Tool for Managing Decision Errors Improves:
Planning Effectiveness
Design Efficiency
Defensibility of results/decisions
Generates appropriate data
Type
Quality
Quantity
EXPLAIN: The purpose of the DQO process is to break things down into a logical process to obtain the right kind of data. It also facilitates management buy-in and support of the process.
EXPLAIN: DQOs have been around for years but in different forms.
For a scientist, it is called the Scientific Method.
For a statistician, it is called the Test of Hypotheses.
EXPLAIN that this process provides defensibility. The use of a documented decision-making process increases the likelihood that decisions will withstand legal challenges.

7. DQO Process Designed to answer: What do you need? Why do you need it?
How will you use it?
What is your tolerance for errors?
EXPLAIN: The DQO process allows the user to answer these specific questions. It provides a logical flow for problem solving.
EXPLAIN: In the past, the term DQO has been confused with data quality indicators (precision, accuracy, detection limits). However, it is important to remember that, while these issues are important to the decision-making process, the purpose of the DQO process is to clearly link sampling and analysis efforts to an action and a decision.
EXPLAIN: The "new" thing about the DQO process is the concept of trying to estimate or control the chance of making a decision error.

8. DQO Process: Underlying Principles
1. All collected data have error.
2. Nobody can afford absolute certainty.
3. The DQO Process defines tolerable error rates.
4. Absent DQOs, decisions are uninformed.
5. Uninformed decisions tend to be conservative and expensive.
EXPLAIN: Estimation tools are imperfect by definition.
DISCUSS how decisions are made and the factors that go into making them:
Cost of absolute certainty versus its benefit
Tolerable decision error rates (risk of making a wrong decision)
Unstated factors such as political concerns or fear of decision making itself.

9. DQOs Strike a Balance DISCUSS the impact of various decisions:
The need to collect more data
Decisions based on inaccurate, incomplete, or missing data.
EXPLAIN the necessity of striking a balance between time and resources and levels of certainty and uncertainty.
EXPLAIN that everyone wants fast, cheap, high quality data and results and compromise must be reached for the practical solution of problems.

10. DQOs in the Context of the Project Life Cycle
DISCUSS the project life cycle and the guidance that exists for each stage in the life cycle.
NOTE: Describe any program-specific guidance if applicable.

11. The DQO Process State the Problem. Identify the Decision.
Identify the Inputs to the Decision.
Define the Boundaries of the Study.
Develop a Decision Rule.
Specify Tolerable Limits on Decision Errors.
Optimize the Design.
EXPLAIN that some of these steps are more difficult than others, but they are all required for the DQO process.
EXPLAIN that Steps 1-5 focus on planning and are applicable in all projects. Steps 6 and 7 are focused on data collection through a documented set of activities.

12. Repeated Application of the DQO Process
EXPLAIN: The DQO process is iterative.
EXPLAIN: After participants become familiar with the structure of the process, it moves quickly -- depending on the problem.
EXPLAIN: Only do as much as is necessary for the problem. It is possible to end the DQO process early.

13. Data Quality Objectives: Outputs from Each Step of the Process
EXPLAIN: Documentation of each step assists in the overall record-keeping of the project. Documentation leads to defensibility of decisions, aids in peer review, and facilitates writing the Quality Assurance Project Plan.

14. The DQO Process Promotes Communication
EXPLAIN: These elements need to be covered effectively during communication between field and management personnel (parameter, risk, media, variance, and sample).
EXPLAIN: Communication gaps exist when using scientific terminology.
DISCUSS methods of bridging the communication gaps.
Avoid using jargon.
Be brief and to the point.
Don't assume that the receiver has all of the information.
Check in periodically to make sure the receiver is hearing and understanding your communication.
DISCUSS an example: "sample“
Chemist sees the physical characteristics.
Statistician sees the theoretical design.

15. A Quality Planning Model
EXPLAIN: The prescribed model provides data to support regulatory decisions.
EXPLAIN: Communication is essential between various factions participating in a decision:
Decision maker
Data collector
Project managers
Other stakeholders.

16. The DQO Process Encourages Efficient Planning
Clearly stated objectives
A framework for organizing complex issues
Limits on decision errors specified
Efficient resource expenditure
EXPLAIN: The DQO process provides a structural, logical, approach to planning.
EXPLAIN: The more complicated the study, the more a definite framework is needed.
EXPLAIN: The DQO process is designed to produce simple answers to complex problems.
EXPLAIN: More simple problems should use a different process to find the necessary answers.
NOTE: Before continuing to the next module, answer any questions participants might have.
NOTE: Explain that you will next discuss data quality objectives.

17. DATA QUALITY OBJECTIVES
NOTE: The approximate time for presenting the module is 60 minutes.
NOTE: The most important thing to keep in mind is pacing. It is essential that the last several slides ( ) are not rushed as they contain the most difficult concepts for people to understand.
NOTE: While this module works well as an introduction, the presentation's alternation between theory and application can create a disjointed impression if it is not followed up with another coherent example. The module that is intended to follow this one (242) takes a problem in the RCRA program directly from start to finish without interruption.
EXPLAIN: This module integrates theory and application at each step of the DQO Process. The example (creosote contamination at a wood-preserving site) is designed to illustrate a practical application of DQO principles at each step.

18. Seven Steps of the DQO Process
1. State the problem to be resolved.
2. Identify the decision to be made.
3. Identify the inputs to the decision.
4. Define the boundaries of the study.
5. Develop a decision rule.
6. Specify the tolerable limits on decision errors.
7. Optimize the design for obtaining the data.
EXPLAIN that the DQO Process starts broadly and proceeds systematically toward more specific and focused activities needed to design a sampling plan.
EXPLAIN that this is an iterative process: Each step can be revisited and changed in light of developments.
EXPLAIN: Each step does not require the same intensity of effort. The effort will vary with the complexity of the project under development.
EXPLAIN: The DQO process ends with the approval of an optimal design.

19. Stating the Problem Who should participate on the planning team?
Risk Assessor
Scientist/Engineer
Statistician/Data Analyst
Data User/Decision Maker
Lab and Field Personnel
QA Specialist
What is the problem?
What resources are available?
What time is available?
What important social/political issues have an impact on the decision?
EXPLAIN that what is really required is someone who is not afraid to consider statistical solutions to problems, and who may have only encountered statistics as an undergraduate.
EMPHASIZE the importance of having the Decision Maker involved early in the process.
EXPLAIN: Social and political issues need to be known up front because they set limits on the domain of science in a public policy decision-making setting.

20. Wood Preserving Site: Background
U.S. State - led investigation of possible soil contamination problem
Creosoting of timbers
Soil contaminated with creosote
Contains Polyaromatic Hydrocarbons (PAHs)
Early Sampling Results:
Soil PAH concentration in low activity area 0-80 mg/kg
Soil PAH concentration in high activity area mg/kg
Off site: Not detected
Future land use will be residential
INTRODUCE the example/case study.
EXPLAIN: This example (creosote contaminates at a wood-preserving site) is broken down step-by-step to parallel the seven steps of the DQO Process. The actual site is in rural South Carolina.

21. Wood Preserving Site: Background
The Team:
Decision Maker
Chemist
Field Sampling Technician
QA Specialist
Risk Assessor/Toxicologist
Environmental Scientist with Statistical Training
EXPLAIN: A "statistician" was not part of the team; they had someone who had some statistics in college.
EXPLAIN: The team includes the decision maker and the specialist.
ASK: What other individuals could be included on the team?

22. Wood Preserving Site: Problem Statement
The Problem: Obvious creosote contamination in
the soil may pose a danger to
human health or the environment.
Information is necessary to
determine the extent of danger.
Resources: Measurement Budget = $100,000
Time Limit: Remediate in 1 year
Socio-political: Future land use is residential
EXPLAIN: The "measurement budget" is intended to cover all field investigation costs, from sampling through chemical analysis and data analysis.
EXPLAIN: The first step is to know what you have, who's going to work on the problem, and how much money you have to work with.

23. Identifying the Decision
Identify the principal study question.
Clarify the main issue to be resolved.
Specify the alternative actions that would result from each resolution.
Associate a course of action with each possible answer.
Define the decision statement that must be resolved to address the problem.
Combine the principal study question and the alternative actions into a specific decision statement.
EXPLAIN: The first two activities are intended to help the team analyze their problem and develop a well-formulated decision statement.
EXPLAIN: Define the principal study question and determine the actions that can be taken.
EXPLAIN: Sometimes getting the team to agree on the main issue can be difficult.

24. Wood Preserving Site: Identifying the Decision
Study Question:
Does creosote contamination in the soil pose an unacceptable danger to human health or the environment?
Alternative Actions:
Remediate the soil
Do not remediate the soil (no action)
Decision Statement:
Determine whether the creosote contamination in soil poses a danger that requires remediation.
EXPLAIN: The action taken depends on the answer to the study question, which will be decided using environmental data.
EXPLAIN: The team should not get too specific here; this decision statement will be honed into a decision rule later in the DQO Process.

25. Identifying Inputs for the Decision
Focus on what information is needed for the decision.
Identify the variables/characteristics to be measured.
Identify the information needed to establish the action level.
EXPLAIN: The numerical value of the action level need not be established here (in fact, depending on circumstances, it may be premature to do that now). What is required here is knowing the basis for the action level -- whether it will be based on regulatory standards, project-specific risk assessment, technology-based standards, a negotiated process among stakeholders, or some other approach. This clarifies the source of information for the action level.
EXPLAIN that Federal regulations play a significant part in the decision process.

26. Wood Preserving Site: Inputs Needed for Decision
Variable of Interest: PAHs Some PAHs are carcinogens that
are dangerous to human health.
Action Level: Set by a toxicologist using
relevant site-specific exposure
assessment at 50 ppm.
EXPLAIN: The action level will be based on site-specific risk assessment; the team has a preliminary number, so they were able to state it here.

27. Defining the Boundaries
Define the spatial boundary for the decision
Define the geographical area within which decisions apply
Define the media of concern
Divide each medium into homogeneous strata
Define the temporal boundary of the decision
Determine the time frame to which the study results apply
Determine when to study
Define a scale of decision making
Identify practical constraints on data collection
EXPLAIN: Up to now, the DQO Process has been fairly general (problem definition, clarifying the decision and inputs). The boundaries step is where the DQO Process begins to focus on more technical details that will be important to the sampling design.
ENCOURAGE the participants to discuss potential practical problems.

28. Wood Preserving Site: Spatial Boundaries
Define the geographical area within which decisions apply:
The property boundary (No PAHs detected off site)
Specify the characteristics that define the population of interest:
PAHs in surface soil to 15 cm depth
Divide each medium into homogeneous strata:
The site has been divided into two areas:
1) Area of high activity where the concentration is
expected to be high
2) Area of low activity where the concentration is expected to be low
EXPLAIN: Note that "surface soil" is being defined more precisely.
EXPLAIN: Stratifying the site will help improve the power to see conclusive results or trends in the contamination patterns, given the "noise" or variability in the data. The phrase "...where the concentration is expected to be high" (and "...low") is intended to imply that the contaminant variability will be relatively high (low).
ASK: What other kinds of spatial parameters could be considered?

29. Wood Preserving Site: Temporal Boundaries
Determine the time frame to which the study results apply:
The results will represent future conditions at the site. (Future lifetime exposure for residents)
Determine when data should be collected:
Sampling begins in 3 months. Remediation completed within 1 year. Sampling results will not vary depending on weather conditions
EXPLAIN: It's important to understand the relationship between the sampling time frame and the temporal aspects of the decision. For example, decisions about acute exposures imply sampling time frames that might differ from those for decisions about chronic exposures.
ASK: What other kinds of time parameters could be considered?

30. Wood Preserving Site: Defining the Boundaries
Scale of Decision Making:
Decisions will be made for each residential lot-sized area (based on future land use)
Practical Constraints:
Existing structures and debris may limit sampling locations
EXPLAIN: Scale of decision making is the area (or for some problems, the time frame) over which the data will be summarized to draw conclusions. We will see later that the scale of decision making is the scale at which decision errors are controlled.
EXPLAIN: Practical constraints can be an important issue for many problems. A "show stopper" constraint could cause the planning team to iterate back to previous DQO steps to find a way around the constraint.
STRESS to participants the need to be aware of structures in the areas where they may want to conduct testing.
URGE participants to consider remediation costs for support crews, i.e., back hoe crews, etc.
EXPLAIN the difference between the scale of decision making (a residential lot) versus the strata (high activity/low activity).

31. Develop a Decision Rule
Develop an "if/then" statement that incorporates:
The population parameter of interest
(e.g., mean, maximum, percentile)
The scale of decision making
(e.g., residential lot size)
The action-triggering value
The alternative actions
EXPLAIN: Some of the elements are taken from previous steps; the new elements to be added are the parameter of interest, and possibly the actual numerical value for the action level.
REF First bullet: EXPLAIN: "Population parameter" means the same as "overall value known only to God/Mother Nature." It is not the same as the information we obtain in a sample, which is only an estimate of the population parameter. Try as an example the overall average contaminants of a hazardous site. We don't know what it is but we will try to estimate it by sampling. To the list of examples, add "3rd highest exceedence" or something to that effect.
REF Third bullet: EXPLAIN: "Action-triggering value" is the action level. The DQO guidance includes an activity to verify that measurement methods exist with detection levels low enough to allow one to reliably distinguish whether the data are above or below the action level. If not, then iteration back to previous DQO steps may be needed.

32. Wood Preserving Site: Decision Rule
Use average (mean) PAH concentrations to identify lots that pose a health threat.
If the true mean PAH concentration within a residential lot is greater than 50 mg/kg, then the soil will be remediated.
If not, then the soil will be left in situ.
EXPLAIN: Note that this is the "theoretical" decision rule stated in terms of the "true mean." This is what the decision maker would really like to know in order to make the correct decision. Data uncertainty and decision errors enter the picture later, not here explicitly.
EMPHASIZE true mean, not estimated mean at this point.

33. Specify Limits on Decision Errors
Determine the possible range of the parameter of interest
Determine baseline condition (null hypothesis)
Determine consequences of each decision error. Consequences may include:
Health risks
Ecological risks
Political risks
Social risks
Resource risks
NOTE: Here is a suggested presentation for this slide's content.
Explain that they do this almost every day of their lives.
At a free standing easel with paper in large numbers, write "55."
The audience will recognize it as the speed limit.
Ask if a police officer would pull you over if you went 75.
(The audience says yes.)
Ask the same question for "56.“
(The audience says no.)
Hone in on 65 and make the argument that the baseline condition for the cop is that you obey the speed limit. He will keep that idea until faced with overwhelming evidence you are speeding (i.e., over 65). The area is then the gray area.

34. Specifying Limits on Decision Error
Specify the gray region - a range of possible parameter values where the consequences of decision errors are relatively minor (too close to call)
Bounded on one side by the action level
Bounded on the other side by the parameter value where the consequences of making a decision error begins to be significant
Set quantitative limits on false rejection and false acceptance errors by considering the consequences of these potential decision errors.
NOTE: This step is usually the hardest for the audience to grasp. One hurdle is the underlying statistical theory, but another consideration is the issue of how value judgments enter into a scientific/technical enterprise.
EXPLAIN: Setting limits on decision errors is a judgment call about how much uncertainty is tolerable to the decision maker. There are many technical aspects to estimating the consequences of one choice versus another, but deciding on the level of uncertainty that is tolerable involves value judgments or policy decisions.
EXPLAIN: It is not easy to decide on the impact of selecting a value for false positive and false negative. It is customary to set them both very stringently (i.e., 1% for each) and relax these slowly until a compromise can be reached.
HINT that later you'll show students a handy program (DEFT) that will enable them to ask the question, "What number of samples would I need if I change the false-negative from this to that?"

35. Statistical Error Types
Rejecting the baseline condition when it is true is a False Rejection error, F(r).
Decision: Not hazardous when it actually is hazardous
Accepting the baseline condition when it is false is a False Acceptance error, F(a).
Decision: Hazardous when it actually is not hazardous
EXPLAIN: These statements are definitions and are therefore always true.
DISCUSS the consequences of these errors.

36. Decision Errors: Synonyms and Plain English
If the baseline assumption is that the program or site is in compliance, then:
False Rejection Error F(r), Type I Error, False Positive
Deciding program or site not in compliance when it is
An overreaction to a situation
Wasted resources, unnecessary expenditure
False Acceptance Error F(a), Type II Error, False Negative
Deciding program or site is in compliance when it is not
A missed opportunity for correction
Allowing a hazard to public health or the ecosystem
EXPLAIN: You would always like to make the correct decision (e.g., determine if a program is in compliance when it really is, and detect a program not in compliance when it violated a regulation).
EXPLAIN: You will make the decision based on what the sample tells you.
NOTE: Introduce the concept that the sample may, by sheer random luck, be predominated by "hot" items, or may have an unexpected number of non-detects.
NOTE: Link the idea of inadvertently making a wrong decision with F(r) and F(a).
NOTE: At the end of the overhead, go back to the assumption (in italics).
ASK: "What if the baseline assumption had been that the program or site is out of compliance?"
NOTE: Allow the audience to reverse the false rejection and false acceptance.
ASK: What is a Type 3 error? (Answer: when you get Type 1 and Type 2 confused).
EXPLAIN: Consider a pregnancy test with the assumption of NOT pregnant. Would the results of a False Rejection and False Acceptance impact them the same way? (Answer: No.)

37. False Rejection and False Acceptance
Baseline Condition: True mean level equal or below standard
Alternative: True mean level above standard
EXPLAIN: The only way in practice to reduce false positive and false negative simultaneously is to take huge numbers of samples. We can't afford the time or resources to do this in environmental decision making.
RE-EMPHASIZE:
"Rejecting the Null when Null is true = false rejection“
"Accepting the Null when Null is false = false acceptance."

38. Decision Errors: Synonyms and Plain English
If the baseline assumption is that the program or site is NOT in compliance, then:
False Rejection Error F(r), Type I Error, False Positive
Deciding program or site is in compliance when it is not
A missed opportunity for correction
Allowing a hazard to public health or the ecosystem
False Acceptance Error F(a), Type II Error, False Negative
Deciding program or site not in compliance when it is
An overreaction to a situation
Wasted resources, unnecessary expenditure
EXPLAIN: This has the opposite baseline condition (assumption) than Slide #20. In this case, the decision errors are flipped. What was called a false rejection is now a false acceptance.
EMPHASIZE the importance of the baseline condition to labeling these errors.

39. The Probability of Making False Rejection Decision Errors
If the true mean is much greater than the action level, few low readings will occur. So, there is a small chance of reaching a wrong conclusion.
EXPLAIN: This shows that if the true average was well above the Action Level (top figure on this slide) then most of the sample data would be around 100 ppm and very little would be near the Action Level. The chance the sample data would mislead you by being very low (i.e., well under the Action Level) is very small.
EXPLAIN: On the other hand (bottom figure on this slide) if the true average and Action Level are very close, then it is more likely that you would get values below the Action Level and so the chances of the data misleading you are much higher.
EXPLAIN: If the action level is very close to the mean (standard or base line), readings will be more difficult to ascertain.
If the true mean is close to the action level, many low readings will occur. Erroneous conclusions are much more likely.

40. Specify Limits on Decision Error (Construct a "What If" Table)
Assign probability values to points above and below the action level that reflect the tolerable probabilities for decision errors.
NOTE: Do not put up all the table at once. It is too intimidating. Block off all except the first line of data and walk the audience through the first line of the slide.
Refer to first line. ASK: "What if the sample mean came back over the Action Level?‘
SAY: You would make the decision to clean up the creosote contaminated site.
ASK: But suppose the true mean (unknown to you) was very low and unbeknownst to you all the "hot" values were in your sample? Hence the high value.
SAY: Based on the erroneous information from the sample, you made a decision. Unfortunately it was the wrong one (but you don't know it).
SAY: Making such a decision (to clean up a site that is essentially already clean) means that there will be a large waste of resources, time, and money trying to correct a problem that is not there.
SAY: The DQO team decided that the risk of such an unfortunate event happening should be quite low (i.e., a risk of only 10%).
EXPLAIN: This chance is only approximate and that it can be adjusted up and down, depending on the implication of the output from the DQO Process.
NOTE: Then block out everything except for the last line and lead them through the argument again. Repeat on another row and then reveal the whole thing.

41. Decision Performance Goal Diagram
EXPLAIN that the table can now be put into graphical form that can be presented to statisticians.
EXPLAIN that there is software (DEFT) that can help determine an estimate of the sample size and develop the decision performance goal diagram. DEFT will be discussed later in the course.

42. Optimize the Design
Develop general data collection design alternatives
Simple random sampling
Simple random sampling with compositing
Stratified random sampling
For each design, develop cost formula, select a proposed method of data analysis, develop method for estimating sample size to correspond to method for data analysis
Select the most resource-effective design
consider cost, human resources, other constraints
consider performance of design
EXPLAIN: This step uses outputs from the previous 6 steps.
REF BULLET 1: EXPLAIN: These are just a few common examples.
REF BULLET 2: EXPLAIN: Notice that we think ahead to how the data will be analyzed when coming up with a basis for determining how many samples to take.
REF BULLET 3: DEFINE: Performance of design is the statistical power curve, which relates directly to tolerable decision error rates from DQO Step 6.

43. Decision Performance Goal Diagram with Performance Curve
EXPLAIN: The statistician takes the sampling design formula and uses it to produce a performance curve (solid curved line) that will satisfy the limits specified by the DQO team.
EXPLAIN how the dotted curve has smaller false negatives than the previous one (very desirable) and this means less risk of a wrong decision. (The dotted line cuts the false acceptance error route by half but it may result in a complex design possible with increased costs.)
EXPLAIN: This diagram allows participants to create a sample design with room to improve.
EXPLAIN: The DEFT software will generate these diagrams for you.

44. DQO Process Output Qualitative and Quantitative Framework for a study
Feeds directly into the Quality Assurance Project Plan which is mandatory for EPA environmental data collection activities
NOTE: This is a wrap-up slide for emphasizing the dual qualitative and quantitative aspects of the DQO Process.
EXPLAIN that the DQO Process really does help with project planning because it helps get the sampling and analysis team organized and contributing to the planning process. In addition, the DQO outputs are used directly in the Quality Assurance Project Plan (QAPP).
NOTE: Before continuing to the next module, answer any questions participants might have.
NOTE: Explain that you will next discuss an example of data quality objectives in action.

45. DATA QUALITY OBJECTIVES: Cadmium Contaminated Fly Ash Example
EXPLAIN: This module is 60 minutes in length.
EXPLAIN: This is an opportunity to actually work through the DQO process using a case study.

46. Case Study Introduction
Case study - Cadmium contaminated fly ash waste
Output from a DQO case study
Shows how the steps of the DQO process aid in developing a sampling design
Illustrates decisions that could be made within the Resource Conservation and Recovery Act (RCRA) Program
Not intended to represent the policies of the RCRA Program
EXPLAIN: This is based on RCRA activities. There may be some simplifying assumptions but it follows the basic principles of the RCRA program.

47. Cadmium Contaminated Fly Ash Waste: Background Information
Municipal incinerator
Fly ash dumped in municipal landfill
Company calls ash "Non-hazardous"
EXPLAIN: This is the background of the problem. The old waste stream is not hazardous and the new waste stream is thought to be very similar.
DEFINE: Fly ash is a solid particulate matter that goes up the stack and is removed by electrostatic grids. The ash is then collected in containers over a period of time.

48. Background Information
New waste stream:
Contains cadmium
Toxic effects: inhalation and ingestion exposure
Short term and chronic effects
The new ash will be tested using Toxicity Characteristic Leaching Procedure (TCLP).
Waste will be classified as hazardous if the cadmium concentration in the TCLP > 1mg/liter.
EXPLAIN: Cadmium is a metal and is not good for human health.
EXPLAIN: 1mg/liter is the Toxicity Characteristic Leaching Procedure (TCLP) established by Risk Assessors.
EXPLAIN: If the waste is hazardous, then it will have to be disposed of differently than if it is non-hazardous.

49. Background Information
Pilot study - to determine the variability of the cadmium concentration in ash
Results:
Relatively constant variability within containers
Relatively high variability between containers
EXPLAIN the results.
EXPLAIN that this is a simplifying assumption. Often there is no pilot study. In that case, the sampling design would have to both determine if there was a problem while determining how to dispose of each container.

50. The DQO Process: State the Problem
Members of Planning Team
Plant Manager Chemist
Plant Engineer Manager Quality Assurance
Statistician/Data Analyst
The Problem
To determine which loads of ash should be sent to a RCRA facility and which can be dumped in the municipal landfill
Available resources
The difference in cost between municipal and RCRA disposal is $6750.
Project constraints
Cost (Budget approximately $3,000 for sampling)
EXPLAIN: A load is either hazardous or it is not.
ASK the participants if they would send all the ash to a RCRA facility and to explain their answer.
DISCUSS the costs of sending all the ash to a RCRA facility, especially repeated loads of ash.
EXPLAIN the nature of the planning team and their possible roles in the investigation.

51. The DQO Process: Identify the Decision
Define the alternative actions.
The waste fly ash could be disposed of in a RCRA landfill.
The waste fly ash could be disposed of in a municipal landfill.
Form alternatives into a decision statement.
Determine if the cadmium concentration in the TCLP leachate exceeds RCRA regulatory standards.
EXPLAIN: The DQO process can address multiple decisions so choose the one that is most important and develop the DQO for it. When dealing with multiple contaminants, base everything on the worst (most important) contaminant and all the others will have smaller risks of a false acceptance or false rejection.

52. The DQO Process: Identify the Inputs to the Decision
Identify key information.
Concentration of cadmium in fly ash
Fly ash samples subjected to the TCLP test and analyzed for cadmium
Identify information to establish the Action Level.
RCRA standard (1.0 mg/l using the TCLP method)
Confirm that appropriate analytical methods exist.
Cadmium is a metal that has a detection limit well below the RCRA standard.
EXPLAIN: This problem would be very hard if the regulatory standard was very close to the detection limit.
EXPLAIN: These inputs are important for making decisions and for developing sampling designs.

53. The DQO Process: Define the Boundaries of the Study
Identify the spatial boundaries.
Fly ash in containerized bins; at least 70% capacity
Identify temporal boundaries.
The ash does not present an exposure hazard and will not degrade; no sampling time constraints are necessary.
Define the scale of decision making.
A decision will be made about each container.
Identify practical considerations that may interfere with the study.
Physically obtaining samples from the containers
EXPLAIN the spatial considerations, temporal boundaries, and practical constraints relevant to the study.
EXPLAIN: Practical constraints can be a big issue. For example, how do you sample from a large pile and keep intact the requirements of a random sample?

54. The DQO Process: Develop a Decision Rule
The Parameter of Interest
The average concentration of cadmium
Specify the Action Level for the study.
The RCRA standard for cadmium (1.0 mg/l) in TCLP leachate
Develop a Decision Rule.
If the average cadmium concentration in a bin is more than 1.0 mg/l, then the ash will be disposed of in a RCRA facility.
If the average cadmium concentration in a bin is less than 1.0 mg/l, then the ash will be disposed of in a municipal landfill.
EXPLAIN: The DQO Process brings all the numerical (quantitative) data into the decision making.
EXPLAIN: "Decision rule" refers to the "true" average of a bin. It does not refer to the arithmetic mean or the geometric mean computed from samples.

55. The DQO Process: Specify Limits on Decision Errors
Determine baseline condition
Null hypothesis = "hazardous" (RCRA requirement)
mean > 1.0 mg/l
Identify decision errors
False rejection:
Decide mean < 1.0 mg/l when mean > 1.0 mg/l
False acceptance:
Decide mean > 1.0 mg/l when mean < 1.0 mg/l
Identify limits on decision errors & gray region
NOTE: On a flip chart, list the 2 decision errors.
ASK the audience to identify the consequences of these errors.
ASK: Which error is more severe?
POINT OUT that usually the baseline condition is then the true part of the more severe error and the more severe error is the false rejection.

56. The DQO Process: Tolerable Limits of Decision Error
ASK the students to interpret this team's decision performance goal.
EXPLAIN: The company wants to minimize the risk of False Rejection at the expense of False Acceptance (0.1 versus 0.2) as consequences for potential decision error is greater.
ASK: "What would happen if the Null Hypothesis was the other way around (i.e., mean is less than or equal to 1.0)?“
REF: The gray region on the other side of Action Level
ANSWER: False Acceptance and False Rejection would trade places.

57. Optimize the Design
Develop general data collection design alternatives
Simple random sampling
Simple random sampling with compositing
Sequential random sampling
For each design, develop cost formula, select a proposed method of data analysis, develop method for estimating sample size to correspond to method for data analysis
Select the most resource-effective design
consider cost, human resources, other constraints
consider performance of design
EXPLAIN: This step uses outputs from the previous 6 steps.
REF BULLET 1: EXPLAIN: These are just a few common examples.
REF BULLET 2: EXPLAIN: Notice that we think ahead to how the data will be analyzed when coming up with a basis for determining how many samples to take.
REF BULLET 3: DEFINE: Performance of design is the statistical power curve, which relates directly to tolerable decision error rates from DQO Step 6.

58. The DQO Process: Optimize the Design
Elements of the Design:
Hypothesis Test
Statistical Model
Design Description/Option
Sample Location
Sample Cost
Sample Size
Design Performance
REF BULLET 7: EXPLAIN: This is a key element even though it is listed last. It evaluates just how effective the sample design is in assisting the decision maker in reaching a conclusion. It is used to compare different designs.

59. Design Options: Simple Random Sampling
Simple Random Sample
Simplest type of probability sampling
Every point in the sampling medium has an equal chance of being selected.
Application
Small variance
Inexpensive sampling and analysis
REF BULLET 1, DASH 2: EXPLAIN:
DRAW on a flip chart a rough rectangle and randomly place some dots inside it.
ASK the students to imagine a 3-dimensional fly-ash pile.
ASK the students to imagine the "point" referred to in the bullet 1, dash 2.
ASK the students what the "point" means.

60. Design Options: Composite Sampling
Physically combining multiple samples then drawing one or more sub-samples for analysis
Application:
When an average concentration is sought and there is no need to detect peak concentrations
Large variance (allows the researchers to sample a larger number of locations)
Reduces total cost when analytical costs are higher than sample collection costs
REFER back to the flip chart and the rectangle.
DRAW some squares, making a grid over the rectangle, shading in a couple of random squares.
EXPLAIN: The shaded squares are selected at random from all the possible squares.
EXPLAIN: Composite sampling means taking a certain number of "mini-samples" from the shaded square, combining/aggregating them, and taking a single sample from the aggregate.
EXPLAIN: The single sample then represents that rectangle.

61. Design Options: Sequential Sampling
Conduct several rounds of sampling and analysis; perform statistical test between each round to make one of three decisions:
Accept null hypothesis
Reject null hypothesis
Collect more samples
Application
When sampling and analysis costs are high
When information about sampling or measurement variability is lacking
When the waste is stable over time frame of the sampling effort
REF bullet 2, dash 3: EXPLAIN: Sequential sampling is not effective in an unstable environment.
EXPLAIN: Sequential sampling is ineffective. There is a large lag time between collecting samples, sending them to the laboratory, and receiving the results. This design stands out if an on-site facility (mobile lab) or accurate screening method is available.

62. Sample/Analysis/Disposal Costs
Sample collection costs from each container- $10/sample
TCLP cost - $150/analysis
15 tons of ash per container
$500/ton RCRA landfill ($7,500 per container)
$50/ton municipal landfill ($750 per container)
EXPLAIN: These are average costs.
EXPLAIN: The sampling design should make sense financially. If the design costs $10,000, you may as well just send the waste to a RCRA landfill.

63. Decision Performance Goal Diagram with Performance Curve: Simple Random Sampling
EXPLAIN: The sample size of 37 came from the sample size formula associated with simple random sampling and decision error limits.
ASK and DISCUSS: Does the chosen method meet the goal of the performance criteria?
EXPLAIN: Relaxing the criteria/constraints to move the performance curve may make meeting the available dollar ceiling limit possible.
ASK the students if they are comfortable with the "tolerance" aspect of potential decision errors.
EXPLAIN: This information came from DEFT which students will see in detail during the afternoon session.

64. Decision Performance Goal Diagram with Performance Curve: Relaxed Decision Error Constraints
DISCUSS how the Tolerable False Acceptance Error Rate has been allowed to grow from 0.2 to 0.3, and the Tolerable False Rejection Rate from 0.05 to 0.1 and, as a result the sample size has decreased.
DISCUSS how the fewer number of samples impacts the decrease in cost.
ASK: What other ways could decrease cost?

65. Decision Performance Goal Diagram with Performance Curve: Increased Gray Region Width
EXPLAIN: This slide shows that we're well under budget.
ASK: What is the definition of the gray area?
ASK: What are the consequences of that definition?
EXPLAIN: The wide gray area means that we're making it even harder to prove the waste is non-hazardous.
EXPLAIN: If the sample mean came back at 0.6 mg/liter, it would not be regarded as "small enough" to prove that the true mean was below 1.0 mg/liter.

66. Decision Performance Goal Diagram with Performance Curve: Simple Random Sampling with Compositing
EXPLAIN: These are the original DQOs after compositing has been done.
EXPLAIN: 64 samples were taken in the field but only 16 were sent to the laboratory for analysis. This means 4 mini-samples were collected and aggregated, and then one sample was extracted from the aggregate. This was done 16 times.

67. Compare Overall Efficiency
*Simple Random Sampling $5920
Simple Random Sampling with $3200
Relaxed Decision Error Constraints
Simple Random Sampling with $2080
Increased Gray Region Width
*Simple Random Sampling with $3040
Compositing
ASK: Which design would you select and why?
EXPLAIN: There is no right or wrong design.
NOTE: Before continuing to the next module, answer any questions participants might have.
NOTE: Explain that you will next look at another example and do an exercise.
* Used original Decision Error Limits

68. Contamination of Tarheel County's Sole Drinking Water Source/System
NOTE: The total length of the module is approximately 1 hour and 30 minutes:
10-20 minutes of instruction
1 hour and minutes of group interaction.
NOTE: The goal is to introduce the problem and set up the exercise.
NOTE: To encourage student participation, make the presentation as interactive as possible:
Ask open-ended questions
Invite clarifying questions.
NOTE: See the first page of activity instructions for preparation tips. worksheet handout.
EXPLAIN: The objective for this module is to give participants a feel for group interaction, the consensus-building process, and using the DQO process.
EXPLAIN: The problem presented for the group to solve has been simplified to avoid getting distracted by technical issues.

69. Drinking Water Problem
Week 1 Quarterly monitoring of drinking water did not detect any contaminants above drinking water standards.
Week 2 Groundwater is the drinking water source for Tarheel
County. Atrazine was discovered in surface waters
(that are hydraulically connected to groundwater) at
level up to 500 ppb, which is well above the maximum contaminant level (MCL) of 3 ppb.
Week 3 Source of contamination has not been identified.
Week 4 - Citizens are concerned about threat to public health
Present and demand that State and Local official ensure that
water is safe to drink.
EXPLAIN: Tarheel County in eastern North Carolina. The terrain is flat and the soil is quite sandy.
EXPLAIN: This slide provides a chronology of the problem to date.
EXPLAIN the relevant points.
This was not a problem until just recently.
Atrazine in the surface water implies it might or might not be in the drinking water.
Atrazine usage is very common in that part of the country.
Citizens are quite worried.

70. Tarheel County Water Supply System
6 wells in wellfield
Water company operates water system
System capacity: 8.6 million gallons/day (MGD)
System demand: 3-5 MGD
System serves 25,000 residents
Minimal Treatment (chlorination only)
Centralized above-ground storage holds water from all wells
Capacity is nearly 10 gallons to ensure 4-hour residence time for chlorination
EXPLAIN: This slide provides some key facts about the water supply system that will be useful later.
EXPLAIN: The only possible treatment is chlorination.
EXPLAIN: The water from each well goes to a large centralized tank.

71. Tarheel County Water Supply System
Assignment:
Decide whether the level of atrazine in
drinking water exceeds the MCL and
requires corrective action.
DISTRIBUTE the handout and the worksheet. Review the handout using the activity instructor notes included with the handout (see #1).
EXPLAIN that the first few steps of the DQO Process have been completed and that the group will be asked to complete Steps 4, 5, and 6.
EXPLAIN that the statistician will actually complete Step 7, Optimize the Design, by selecting a suitable sampling scheme, but that will not be a small group task.
NOTE: Continue with #2-6 of the activity instructor notes.
NOTE: Before continuing to the next module, answer any questions participants might have.
NOTE: Explain that you will next discuss the DEFT software.

72. Data Quality Objectives Decision Error Feasibility Trials Software (DQO/DEFT)
NOTE: The instructor should be familiar with the DEFT software before teaching this module. The DEFT screen slides used in this module are from the module 242 (Cadmium Fly Ash).
EXPLAIN: The length of the module is 30 minutes.
EXPLAIN: We are now going to discuss some software to help you in completing the DQO process. The software runs on most IBM-compatible personal computers and is interactive, easy-to-use, and useful.

73. The Purpose of DEFT
DEFT determines the feasibility of DQOs based on sample size and cost for several sampling designs
DQOs are feasible if at least one sampling design can satisfy the DQOs (decision error limits, cost constraints, time limitations, etc.).
ASK: At this point, you have completed Steps 1-6 in the DQO Process. How do you get from here to the sampling design?

74. Uses of DEFT
Aids in iterations between steps 6 and 7 of the DQO process
That is, it provides a smooth transition between the specific DQOs and the development of a data collection design
As a learning tool, facilitates understanding and communication
EXPLAIN: DEFT was built for non-statisticians. It is generic in nature. It is not tailored for water applications or air applications.
EXPLAIN: DEFT runs on most IBM-compatible personal computers. It is interactive, easy to use, and useful.
EXPLAIN: You can use the software to go back and forth between Step 6 and Step 7 to see what is feasible, and what is not.
EXPLAIN: First you develop limits. Then you go right to the statistician, who will develop a sampling design based on your feasible inputs.
EXPLAIN: DEFT graphically shows the relationships to aid in learning.

75. What DEFT Cannot Do
DEFT should not be used to decide on a final data collection design or sample size.
It cannot account for differences between:
Media
Contaminants
Spatial boundaries
Temporal boundaries
EXPLAIN: DEFT doesn't understand the specifics of the problem, so the results should be considered an estimate. A statistician can improve on the sampling design considered by DEFT to create a more cost-effective design. For example, DEFT can estimate sample sizes for a stratified design but a statistician can help you select the proper stratification level.

76. How DEFT Works Utilizes outputs of the DQO process
Evaluates several basic collection designs
Estimates the number of samples
Estimates costs of data collection designs
EXPLAIN: First, you enter the results of DQO steps 1-6.
EXPLAIN: Then you use the software to develop several designs and cost estimates, much like what was done in the Cadmium Fly Ash example.
EXPLAIN: In the Cadmium example, DEFT was used to create the graphs.

77. What DQO Outputs are Necessary as DEFT Inputs?
Limits on decision errors
Action level
Possible range of parameter (minimum, maximum)
Cost of sample collection and analysis per sample
Location and width of gray region
Estimated standard deviation
Null hypothesis (H0)
EXPLAIN: These are inputs for simplest sampling design (excluding judgmental) - a simple random sample.
EXPLAIN: For other designs, DEFT will ask for more input.

78. Analysis of DEFT Allows user to: Determine effect or change DQOs
View Decision Performance Goal Diagram
Change sampling design
Simple Random Sampling
Composite Random Sampling
Stratified Random Sampling
Set sample size
Save DQOs, design information, and decision performance goal diagram to a file
NOTE: Walk through these items and explain why they are useful.

80. DEFT in the Project Life Cycle
EXPLAIN: DEFT is useful in the planning and assessing steps of a project's life cycle.
NOTE: Before continuing to the next module, answer any questions participants might have.
NOTE: Explain that you will next discuss assessment.

81. Beyond the DQO Process
NOTE: The appropriate time for this module is 30 minutes.

82. The Project Life Cycle
EXPLAIN: There are the 3 phases of a project's life: plan, implement, and assess.
EXPLAIN: This course has concentrated on the planning component of a project. After the planning is complete, there are other activities to be performed.
EXPLAIN: These activities, QA Project Plans and Data Quality Assessment are the focus of this module.
EXPLAIN: The numbers on the right correspond to Agency-wide guidance. This guidance will be discussed at the end of this module.

83. What Is A QA Project Plan?
Mandatory planning document
Part of mandatory Agency-wide Quality System
Description of how data will be collected, assessed, and analyzed
Project Blueprint - who, what, where, when, why
Living document that is revised to reflect significant changes
EXPLAIN: The QAPP is a written, formal document. There are defined procedures for its contents and its approval.
EXPLAIN: It is a mandatory part of the Agency's Quality System. Any data collection activity must have an approved QAPP as required by EPA Order A2 (2000) for EPA organizations and the Federal Acquisition Regulations for non-EPA organizations..
EXPLAIN: The QAPP spells out exactly how the data are to be obtained, analyzed, and validated.
EXPLAIN: The results of the DQO process are documented in the QAPP which will be discussed in more detail in the upcoming slides.

84. QA Project Plans (QAPPs)
QAPPs must be approved prior to the start of data collection
QAPPs are required when environmental data operations occur in:
Intramural projects
Contracts, work assignments, delivery orders
Grants, cooperative agreements
Interagency agreements (when negotiated)
State-EPA agreements
Responses to statutory or regulatory requirements and to consent agreements
EXPLAIN: QAPPs apply to:
Direct measurements or data generation
Environmental modeling
Compilation of data from literature or electronic media
Data supporting the design, construction, and operation of environmental technology
EXPLAIN: In other words, practically everything has to have an approved QAPP!

85. What Does A QA Project Plan Do For You?
When you are asked:
"What did you do?"
"How did you do it?"
"Why did you do it?"
"Did you do it correctly?"
The QA Project Plan has the answer.
EXPLAIN: The QAPP should document everything:
What you did
How you did it
Why you did it
How you can tell if you did it correctly.

86. Elements of a QA Project Plan
Group A. Project Management
Group B. Data Generation and Acquisition
Group C. Assessment and Oversight
Group D. Data Validation and Usability
EXPLAIN: The recommended format for a QAPP consists of 4 groups of elements. These groups cover
What did you do?
How did you do it?
Why did you do it?
Did you do it correctly?

87. Group A: Project Management Element
1. Title and Approval Sheet
2. Table of Contents
3. Distribution List
4. Project/Task Organization
5. Problem Definition/Background
6. Project/Task Description
7. Quality Objectives and Criteria
8. Special Training Requirements/Certification
9. Documentation and Records
EXPLAIN: There are nine elements in Group A, the Project Management group. These elements provide an overview of the problem.
EXPLAIN: Elements of the DQO Process are documented in items #5, 6, 7.

88. Group B: Data Generation & Acquisition Elements
Sampling Process Design (Experimental Design)
Sampling Methods Requirements
Sample Handling and Custody Requirements
Analytical Methods Requirements
Quality Control Requirements
Instrument/Equipment Testing, Inspection, and Maintenance Requirements
Instrument Calibration and Frequency
Inspection/Acceptance Requirements for Supplies and Consumables
Data Acquisition Requirements (Non-Direct Measurements
Data Management
EXPLAIN: There are ten elements in Group B, the Measurement and Data Acquisition group.
EXPLAIN: Outputs from the Data Quality Objectives Process are documented in items 1, 2, 4, and 5.

89. Elements in Group C & Group D
Group C: Assessment & Oversight Elements
1. Assessments and Response Actions
2. Reports to Management
Group D: Data Validation & Usability Elements
1. Data Review, Validation, and Verification Requirements
2. Validation and Verification Methods
3. Reconciliation with User Requirements
EXPLAIN: There are two elements in Group C, the Assessment and Oversight group. The elements include oversight and technical assessments.
EXPLAIN: There are three elements in Group D. Outputs from the DQO process are documented in #3.
EXPLAIN: Item #3 in Group D is planning for the data quality assessment, which is the next (and final) topic to be discussed.

90. Data Quality Assessment (DQA)
A process to determine if data are adequate for their intended use
scientific and statistical evaluation
determine if data are of the right type, quality, and quantity
Sample data are used to make decisions during DQA
Does data provide "sufficient evidence" to draw conclusions?
EXPLAIN: Data Quality Assessment (DQA) is both
Scientific - do the data make sense from a chemical, biological, etc. point of view, and
Statistical - may be complex or simple analysis.
EXPLAIN: Give examples. Data collected to see if drinking water contaminated -- are data adequate? Is a remediation technique working?
EXPLAIN: The purpose of DQA is to allow you to determine whether a specific decision can be made using a specified set of data. This includes the use of secondary data (data collected for other purposes.)

91. Data Quality
Data quality is meaningful only when "data quality" relates to intended use of data
Some data are of adequate quality for some purposes but not for others
Need to determine if the data are of the right type, quality, and quantity for their intended use
EXPLAIN: Before you can assess the data, you need to discuss what "data quality" is. EPA defines data quality in terms of its intended use.
PROVIDE examples of data that are adequate for one purpose but not for another. For example, Right-to-Know data.

92. Data Quality Assessment Can
Answer:
Do the data violate the conceptual site model or test assumptions?
Did I collect enough data?
What is my conclusion?
Can Not Answer:
Did I make a decision error?
(good decision -- bad outcome)
What are the "true" conditions?
Do I need different types of data?
EXPLAIN: DQA allows you to make a decision based on the data.
ASK why DQA can not tell you if you made a decision error.
EMPHASIZE that you will never know the absolute truth, because of various errors.

93. Data Quality Assessment Can
Decision maker's contribution:
Inspection of data for scientific anomalies
Responsibility for transcription errors
Assessment of effect of QA and QC deviations
Professional contextual judgment
EXPLAIN: DQA can not be performed by a statistician or project manager alone. The two must work hand in hand.
PROVIDE examples of the decision makers contributions - for example, a statistician can not discard outliers, the project manager can because the statistical tests do not integrate in the underlying science.
EXPLAIN: Project Manager or Analyst must provide the scientific basis for interpreting the results. They must decide if the statistical test makes sense.

94. DQA is a Joint Effort Statistician's contribution:
Graphical display of data and trends
Statistical analysis required by the DQO
Investigation of assumption violations
Identification of potential outliers
Providing direction for data improvement
PROVIDE examples of the statistician's contributions - for example, a statistician will create graphs and recommend statistical tests. Statistician can not interpret results for you

95. The 5 Steps of Data Quality Assessment
1. Review the DQOs and Sampling Design
2. Conduct a Preliminary Data Review
3. Select the Statistical Test
4. Verify the Assumptions of the Statistical Test
5. Draw Conclusions from the Data
EXPLAIN: In Step 1, performance criteria are created (e.g., data quality objectives) if they don't already exist and review all background documentation.
EXPLAIN: In Step 2, preliminary analysis is performed. Graphs and summary statistics are reviewed.
EXPLAIN: In Step 3, the statistical test is selected (if it was not done in DQO process) and its assumptions are identified- normality, independence, no outliers.
EXPLAIN: In Step 4, the assumptions of the test are verified - so a test for normality is performed, a test for outliers is performed, etc.
EXPLAIN: In Step 5, the statistical test is run and conclusions are drawn from the results. The conclusions are not based solely on the statistics but also on the science.

96. Guidance for Data Quality Assessment: Practical Methods for Data Analysis (G-9)
Written for non-statisticians
Supplements Agency guidance
Does not replace statistical texts
Regular supplements
Current examples
Shared information
EXPLAIN: This document is a 5-step process to enable non-statisticians to analyze results in a meaningful fashion. It provides an answer to the question: "Are the data any good?“
EXPLAIN: The document has two parts to each chapter. The first part describes the activities in that step of DQA. The second part provides tools with step-by-step directions for implementing these activities.

97. DataQUEST
A PC-based software package that performs baseline Data Quality Assessment
Provides simple tools to a wide audience
Implements statistical methods described in guidance (G-9)
Supplements guidance so description of statistical tools is not contained in the User's Guide
EXPLAIN: DataQUEST is simple, easy-to-use, runs on most IBM compatible PCs, no special requirements.
EXPLAIN: All tools contained in Guidance for Data Quality Assessment is contained in software. The software and guidance are used together.

98. Advantages
Menu-based System - no special language or commands like statistical packages
Does not treat data as discreet numbers in graphs like spreadsheets
More standards statistical graphs than spreadsheets
EXPLAIN: Easy to use - just select the number of the test you wish to perform. Designed for managers or those not familiar with statistical packages.
EXPLAIN: Not meant to replace a full statistics package or spreadsheet. These packages have years of development and advanced capabilities it would not be efficient for the government to redevelop the wheel. Instead, meant for basic, quick, analysis. For example, the software has no in-depth analysis capabilities, limited data manipulations, and no presentation graphics (i.e., pie charts).

99. QA Guidance www.epa.gov/quality
Guidance for the Data Quality Objectives Process (G-4)
Planning process that ties data collection designs to user defined decision error tolerances
Guidance for QA Project Plans (G-5)
Utilizes outputs of DQO Process for detailing data collection operations, the "blue-print" of data collection
Guidance for Data Quality Assessment (G-9)
Assessment of data to establish if they meet user-defined decision error limits
EXPLAIN: Where to get these documents, which document to use and when.
INCLUDE any other applicable guidance.
EXPLAIN: Remind the participants of the software that supplements G-4 and G-9.