1. Understanding Data Choices, Characteristics, Limitations
Or how do we make sure the data isn’t junk
June 23, 2006
States' Common Measures Project

2. Context: Four Questions To Answer In Phase II
What Groups Will We Evaluate?
What Indicators Will We Use To Describe Their Environmental Performance?
What Data Will We Use To Measure Performance On Each Indicator
How Will We Get The Data?
June 23, 2006
States' Common Measures Project

3. States' Common Measures Project
What IS Data, Anyway
What we will be using to measure the indicators of environmental performance of the groups we pick
June 23, 2006
States' Common Measures Project

4. Important Context (Let’s Not Go Crazy)
The level of data quality needed depends upon the uses for the information
Interesting Anecdote
Planning
Enforcement
Life or Death
Increasing data quality
June 23, 2006
States' Common Measures Project

5. What Kinds Of Data Could We Use?
“Environmental Quality Data” : chemical, physical or biological characteristics of:
Air, water, soil
Emissions, discharges wastes or raw materials
“Performance Data”
Are facilities taking the actions that we want them to (recordkeeping, operation and maintenance, monitoring, using right materials, managing wastes properly, engaging in P2 etc.)
INCREASING COST OF COLLECTING DATA
INCREASING COST OF QUALITY CONTROL
June 23, 2006
States' Common Measures Project

6. Where Can The Data Come From?
PRIMARY DATA
Data we collect in the field – inspections, sampling, surveys
Data we we collected previously -- file or data base review
SECONDARY DATA
Data submitted to us by someone else (e.g report review)
Data collected and analyzed by others
INCREASING CONTROL OVER DATA QUALITY
INCREASING COST OF COLLECTION
June 23, 2006
States' Common Measures Project

7. INCREASING CONTROL OVER DATA QUALITY
New vs. Existing Data
Data collected specifically for the project
Data collected previously for other purposes
INCREASING CONTROL OVER DATA QUALITY
June 23, 2006
States' Common Measures Project

8. What Are The Key Factors Influencing Data Quality?
Data Quality Indicators
Precision
Sensitivity
Representativeness
Comparability
Completeness
Bias
The Quality of Data Collection & Analysis
Verification
Validation
Integrity
June 23, 2006
States' Common Measures Project

9. States' Common Measures Project
Precision
Precision is the measure of agreement among repeated measurements of the same property under identical or substantially similar conditions
A precision DQI is a quantitative indicator of the random errors or fluctuations in the measurement process
Example, when I measure my kids I line them up against the wall and put a ruler across the top of their heads : its amazing: One time when I checked my son had grown a half an inch, and and when I check a few hours later he’d grown an another half inch! Did he really grow half an inch in an hour or was my measurement technique imprecise?
Source: EPA Introduction to Data Quality Indicators
June 23, 2006
States' Common Measures Project

10. Data Characteristics that Influence Precision
Environmental Quality Data
Measuring large incremental differences
Measuring small incremental differences
Performance Data
Measuring concrete requirements
Measuring subjective requirements
Easier to get needed precision
Harder to get needed precision
June 23, 2006
States' Common Measures Project

11. States' Common Measures Project
Sensitivity
Sensitivity is the capability of a method or instrument to discriminate between measurement responses representing different levels of the variable of interest
A sensitivity DQI describes the capability of measuring a constituent at low levels
Source: EPA Introduction to Data Quality Indicators
June 23, 2006
States' Common Measures Project

12. Data Characteristics that Influence Sensitivity
Environmental Quality Data
Measuring large amounts
Measuring small amounts
Performance Data
Measuring whether or not performance occurred
Measuring gradations in performance
Easier to get needed sensitivity
Harder to get needed sensitivity
June 23, 2006
States' Common Measures Project

13. States' Common Measures Project
Representativeness
Representativeness is the measure of the degree to which data suitably represent a characteristic of a population, parameter variations at a sampling point, a process condition, or an environmental condition
Representativeness DQIs are qualitative and quantitative statements regarding the degree to which data reflect the true characteristics of a well defined population
Source: EPA Introduction to Data Quality Indicators
June 23, 2006
States' Common Measures Project

14. Data Characteristics that Influence Representativeness
Existing/New
Collecting new data for project
Using preexisting data
Data Source
Using primary data
Using secondary data
Easier to get representative data
Harder to get representative data
June 23, 2006
States' Common Measures Project

15. States' Common Measures Project
Comparability
Comparability is a qualitative expression of the measure of confidence that two or more data sets may contribute to a common analysis
A comparability DQI is a qualitative indicator of the similarity of attributes of data sets
Source: EPA Introduction to Data Quality Indicators
June 23, 2006
States' Common Measures Project

16. Data Characteristics that Influence Comparability
Existing/New
Collecting new data for project, collected over shorter time period
Using preexisting data
Data Source
Primary data collected by fewer people in fewer agencies over a shorter time period
Using Secondary data
Easier to get comparable data
Harder to get comparable data
June 23, 2006
States' Common Measures Project

17. States' Common Measures Project
Completeness
Completeness is a measure of the amount of valid data obtained from a measurement system, expressed as a percentage of the number of valid measurements that should have been collected
The DQI for completeness is often expressed as a percentage
Source: EPA Introduction to Data Quality Indicators
June 23, 2006
States' Common Measures Project

18. States' Common Measures Project
Bias
Bias is systematic or persistent distortion of a measurement process that causes error in one direction
A bias DQI is a quantitative indicator of the magnitude of systematic error.
Source: EPA Introduction to Data Quality Indicators
June 23, 2006
States' Common Measures Project

19. States' Common Measures Project
Validation
Data validation is an analyte and sample matrix-specific process to determine the analytical quality of a specific data set
Source: EPA Introduction to Data Quality Indicators
EXPLAIN: Validation associated with the quality of individual data points.
June 23, 2006
States' Common Measures Project

20. States' Common Measures Project
Verification
Data verification refers to the procedures needed to ensure that a set of data is a faithful reflection of all the processes and procedures used to generate the data
Source: EPA Introduction to Data Quality Indicators
EXPLAIN: Verification usually associated with contractual requirements.
June 23, 2006
States' Common Measures Project

21. States' Common Measures Project
Integrity
Lack of integrity affects all aspects of data interpretation, especially data used for decision making
Source: EPA Introduction to Data Quality Indicators
EXPLAIN: Integrity often associated with adherence to Good Laboratory Practices and honesty.
June 23, 2006
States' Common Measures Project

22. So Why Did We Go Over All This?
These are the issues that affect the quality of the data we obtain in this project
IF our data quality is no good THEN
Our indicators won’t measure performance appropriately
And THEN
We won’t be able to meet our project goal of measuring the performance of our group(s) across our states.
June 23, 2006
States' Common Measures Project