1 Continuous Pollutant and Meteorological Data Continuous Pollutant and Meteorological Data

2 Routine Data Operations—Continuous (gas, PM, and met)

3 QC Check Data Entry We recommend you start with QC check data entry (except for meteorological data)We recommend you start with QC check data entry (except for meteorological data) Then you can review monitoring data with its relevant QC data as part of the data import processThen you can review monitoring data with its relevant QC data as part of the data import process This allows you to manage potential maintenance and other issues with samplerThis allows you to manage potential maintenance and other issues with sampler A set of data (two weeks worth, for example) has at least one QC check before and one after, if you routinely download data from the sampler each time you perform a QC checkA set of data (two weeks worth, for example) has at least one QC check before and one after, if you routinely download data from the sampler each time you perform a QC check

4 Routine is to 1 st enter QC check, then import data:

5 QC Data Entry—One Form

QC checks: 6

AQS-related terms: 7

Different types of QC checks: 8

Continuous PM Checks: 9

10 Click on Linear Regression

11 QC check results that are out of EPA limits cause flags, which are internal to your data Flag for level one span check

12 After QC entry, the actual Data: Data files to import are comma delimited text files with field (column) namesData files to import are comma delimited text files with field (column) names If data files not in this format, easy to convert (see SOP)If data files not in this format, easy to convert (see SOP) The text file name must be changed to NewImport.txt and saved on C drive (this step is the same for every imported file)The text file name must be changed to NewImport.txt and saved on C drive (this step is the same for every imported file) Keep a copy of file elsewhere with its original nameKeep a copy of file elsewhere with its original name Take notes of what you are importing, where the original file is, date, etc. in your Database LogbookTake notes of what you are importing, where the original file is, date, etc. in your Database Logbook

13 Import Raw Data Data are moved thru “pipeline” of tables Data ImportData Verification Archive

14 The Data Import Table (pipeline intake-the 1 st table) Data go temporarily in the Data Import tableData go temporarily in the Data Import table Data retain field (column) names from text fileData retain field (column) names from text file This Data Import table takes whatever is in the NewImport.txt fileThis Data Import table takes whatever is in the NewImport.txt file

15 First Step is to import the NewImport.txt file into the Data Import Table

Then, transfer to the middle table: Verification table is an empty shell with many columnsVerification table is an empty shell with many columns You tell the Toolbox which columns from the Import table go into which columns in the Verification table, by field mappingYou tell the Toolbox which columns from the Import table go into which columns in the Verification table, by field mapping 16

17 Transfer to Data Verification Table (middle step of the pipeline)

18 Example of Field Mapping Columns in the verific’n table

19 The Data Verification Table (middle step in pipeline) You will do most of work on data while it is in Data Verification tableYou will do most of work on data while it is in Data Verification table You will add information such as Site IDs, Sampler IDs, Units, etc.You will add information such as Site IDs, Sampler IDs, Units, etc. You may have to clarify sampling dates and times to be consistent with AQS specsYou may have to clarify sampling dates and times to be consistent with AQS specs You will do initial data flagging in this tableYou will do initial data flagging in this table

20 Check ranges, error codes, units, dates in this middle Verification Table

21 The Final Table: Data Archive Table Data are permanently stored in this tableData are permanently stored in this table Table includes all data you have previously archived (unlike Data Verification and Data Import that you clear once you have archived the data in your import)Table includes all data you have previously archived (unlike Data Verification and Data Import that you clear once you have archived the data in your import) Over years, table will get large but database can handle 2 gigabytes (one Toolbox we know has 3 years of data from four sites and the entire database is still only 35 megabytes)Over years, table will get large but database can handle 2 gigabytes (one Toolbox we know has 3 years of data from four sites and the entire database is still only 35 megabytes)