Understanding Data Choices, Characteristics, Limitations Or how do we make sure the data isn’t junk June 23, 2006 States' Common Measures Project

Context: Four Questions To Answer In Phase II What Groups Will We Evaluate? What Indicators Will We Use To Describe Their Environmental Performance? What Data Will We Use To Measure Performance On Each Indicator How Will We Get The Data? June 23, 2006 States' Common Measures Project

States' Common Measures Project What IS Data, Anyway What we will be using to measure the indicators of environmental performance of the groups we pick June 23, 2006 States' Common Measures Project

Important Context (Let’s Not Go Crazy) The level of data quality needed depends upon the uses for the information Interesting Anecdote Planning Enforcement Life or Death Increasing data quality June 23, 2006 States' Common Measures Project

What Kinds Of Data Could We Use? “Environmental Quality Data” : chemical, physical or biological characteristics of: Air, water, soil Emissions, discharges wastes or raw materials “Performance Data” Are facilities taking the actions that we want them to (recordkeeping, operation and maintenance, monitoring, using right materials, managing wastes properly, engaging in P2 etc.) INCREASING COST OF COLLECTING DATA INCREASING COST OF QUALITY CONTROL June 23, 2006 States' Common Measures Project

Where Can The Data Come From? PRIMARY DATA Data we collect in the field – inspections, sampling, surveys Data we we collected previously -- file or data base review SECONDARY DATA Data submitted to us by someone else (e.g report review) Data collected and analyzed by others INCREASING CONTROL OVER DATA QUALITY INCREASING COST OF COLLECTION June 23, 2006 States' Common Measures Project

INCREASING CONTROL OVER DATA QUALITY New vs. Existing Data Data collected specifically for the project Data collected previously for other purposes INCREASING CONTROL OVER DATA QUALITY June 23, 2006 States' Common Measures Project

What Are The Key Factors Influencing Data Quality? Data Quality Indicators Precision Sensitivity Representativeness Comparability Completeness Bias The Quality of Data Collection & Analysis Verification Validation Integrity June 23, 2006 States' Common Measures Project

States' Common Measures Project Precision Precision is the measure of agreement among repeated measurements of the same property under identical or substantially similar conditions A precision DQI is a quantitative indicator of the random errors or fluctuations in the measurement process Example, when I measure my kids I line them up against the wall and put a ruler across the top of their heads : its amazing: One time when I checked my son had grown a half an inch, and and when I check a few hours later he’d grown an another half inch! Did he really grow half an inch in an hour or was my measurement technique imprecise? Source: EPA Introduction to Data Quality Indicators http://epa.gov/quality/trcourse.html June 23, 2006 States' Common Measures Project

Data Characteristics that Influence Precision Environmental Quality Data Measuring large incremental differences Measuring small incremental differences Performance Data Measuring concrete requirements Measuring subjective requirements Easier to get needed precision Harder to get needed precision June 23, 2006 States' Common Measures Project

States' Common Measures Project Sensitivity Sensitivity is the capability of a method or instrument to discriminate between measurement responses representing different levels of the variable of interest A sensitivity DQI describes the capability of measuring a constituent at low levels Source: EPA Introduction to Data Quality Indicators http://epa.gov/quality/trcourse.html June 23, 2006 States' Common Measures Project

Data Characteristics that Influence Sensitivity Environmental Quality Data Measuring large amounts Measuring small amounts Performance Data Measuring whether or not performance occurred Measuring gradations in performance Easier to get needed sensitivity Harder to get needed sensitivity June 23, 2006 States' Common Measures Project

States' Common Measures Project Representativeness Representativeness is the measure of the degree to which data suitably represent a characteristic of a population, parameter variations at a sampling point, a process condition, or an environmental condition Representativeness DQIs are qualitative and quantitative statements regarding the degree to which data reflect the true characteristics of a well defined population Source: EPA Introduction to Data Quality Indicators http://epa.gov/quality/trcourse.html June 23, 2006 States' Common Measures Project

Data Characteristics that Influence Representativeness Existing/New Collecting new data for project Using preexisting data Data Source Using primary data Using secondary data Easier to get representative data Harder to get representative data June 23, 2006 States' Common Measures Project

States' Common Measures Project Comparability Comparability is a qualitative expression of the measure of confidence that two or more data sets may contribute to a common analysis A comparability DQI is a qualitative indicator of the similarity of attributes of data sets Source: EPA Introduction to Data Quality Indicators http://epa.gov/quality/trcourse.html June 23, 2006 States' Common Measures Project

Data Characteristics that Influence Comparability Existing/New Collecting new data for project, collected over shorter time period Using preexisting data Data Source Primary data collected by fewer people in fewer agencies over a shorter time period Using Secondary data Easier to get comparable data Harder to get comparable data June 23, 2006 States' Common Measures Project

States' Common Measures Project Completeness Completeness is a measure of the amount of valid data obtained from a measurement system, expressed as a percentage of the number of valid measurements that should have been collected The DQI for completeness is often expressed as a percentage Source: EPA Introduction to Data Quality Indicators http://epa.gov/quality/trcourse.html June 23, 2006 States' Common Measures Project

States' Common Measures Project Bias Bias is systematic or persistent distortion of a measurement process that causes error in one direction A bias DQI is a quantitative indicator of the magnitude of systematic error. Source: EPA Introduction to Data Quality Indicators http://epa.gov/quality/trcourse.html June 23, 2006 States' Common Measures Project

States' Common Measures Project Validation Data validation is an analyte and sample matrix-specific process to determine the analytical quality of a specific data set Source: EPA Introduction to Data Quality Indicators http://epa.gov/quality/trcourse.html EXPLAIN: Validation associated with the quality of individual data points. June 23, 2006 States' Common Measures Project

States' Common Measures Project Verification Data verification refers to the procedures needed to ensure that a set of data is a faithful reflection of all the processes and procedures used to generate the data Source: EPA Introduction to Data Quality Indicators http://epa.gov/quality/trcourse.html EXPLAIN: Verification usually associated with contractual requirements. June 23, 2006 States' Common Measures Project

States' Common Measures Project Integrity Lack of integrity affects all aspects of data interpretation, especially data used for decision making Source: EPA Introduction to Data Quality Indicators http://epa.gov/quality/trcourse.html EXPLAIN: Integrity often associated with adherence to Good Laboratory Practices and honesty. June 23, 2006 States' Common Measures Project

So Why Did We Go Over All This? These are the issues that affect the quality of the data we obtain in this project IF our data quality is no good THEN Our indicators won’t measure performance appropriately And THEN We won’t be able to meet our project goal of measuring the performance of our group(s) across our states. June 23, 2006 States' Common Measures Project