Review: Simulation (see handout) Teradata & SQL accounts LMS GIGO

Introduction to Data Quality & Data Cleaning Data and information is not static, it flows in a data collection and usage process Data gathering Data delivery Data storage Data integration Then what?

Data Quality Characteristics Data quality affects several attributes associated with data: Accuracy – Is it realistic or believable? Integrity – Is it structured and managed? Consistency – Is it consistently defined and maintained? Validity – Is the data valid, based on business or industry rules and standards?

What Causes Poor Data Quality? These factors can contribute to poor data quality: Business rules do not exist or there are no standards for data capture. Standards may exist but are not enforced at the point of data capture. Inconsistent data entry (incorrect spelling, use of nicknames, middle names, or aliases) occurs. Data entry mistakes (character transposition, misspellings, and so on) happen. Integration of data from systems with different data standards is present. Data quality issues are perceived as time-consuming and expensive to fix. There are many attributes which can contribute to poor data quality, including: A lack of business rules (or standards) at the point of data capture. **This is the cheapest place to deal with data quality. If it were dealt with at the point of data capture, it would not have to be included “downstream” during the data warehousing or decision support processes. Standards not being enforced at the point of data capture. This is extremely prevalent in situations where you have divisions and departments all running their own systems, and none can agree on what standards should be used/enforced when capturing data. Inconsistencies in data entry (across different users, different departments, or even across multiple operational systems) Common data entry mistakes Pulling data together from multiple systems, each with a different standard. This is an issue with pulling in historical data that may have been entered via an old legacy system. Also, with mergers and acquisitions going on these days, it has become more and more of an issue. The misperception that data quality issues are time consuming and expensive to fix.

Primary Sources of Data Quality Problems This chart represents the percentage of data quality problems that are caused by these sources. Why do these problems happen? Typos and misspellings when data is being entered by employees. **Sometimes even valid phone numbers and addresses are not necessarily correct! Changes in source systems that cause problems when the data is propagated downstream in the warehousing process. Data migration and conversion of data from multiple operational sources. The data was entered with different standards, and therefore does not necessarily merge together cleanly. Differing levels of expectations and standards by the users. What is good enough for one user may not necessarily be good enough for another one. **This is especially common in a situation where you have many different groups of users within an organization using the same data, but each with its own business issues and concerns (i.e. how do you define a customer, supplier, product, etc.). Source: The Data Warehousing Institute, Data Quality and the Bottom Line, 2002

How Is Clean Data Achieved? Clean data is the result of a combination of efforts: making sure that data entered into the system is clean GIGO cleaning up problems after the data is accepted.

Some errors are easier to fix than others… If we have data for Male and Female (‘M’,‘F’) then is it a reasonable assumption that ‘f’ is supposed to be ‘F’? How about age, is 135 valid? Can we fix it? What if we had DOB? What about subject codes: BI is 306622, what if we find 306662, can we fix it? This is a common data entry problem called transposition

Analysis and Standardization Example Who is the biggest supplier? Anderson Construction $ 2,333.50 Briggs,Inc $ 8,200.10 Brigs Inc. $12,900.79 Casper Corp. $27,191.05 Caspar Corp $ 6,000.00 Solomon Industries $43,150.00 The Casper Corp $11,500.00 The example shows that the vendor Briggs Incorporated has two incomplete expenditure summations and Casper Corporation has three incomplete expenditure summations (with one entry listed near the end of the report as The Casper Corp). This is a sample from 27,000 entries. ... ...

Standardization Scheme Briggs, Inc  Brigs Inc.  Briggs Inc. Casper Corp.  Caspar Corp  The Casper Corp  Casper Corp. Using dfPower Studio to identify the different permutations of each company name, you can build a scheme and use it to standardize the company name so that each vendor has one unique representation within the database. ... ...

Supplier Spending $ Spent 50,000 40,000 30,000 20,000 10,000 10,000 20,000 30,000 40,000 50,000 $ Spent Casper Corp. Solomon Ind. Briggs Inc. Anderson Cons. The Cost by Vendor report will be more organized, useful and accurate, because vendor expenditure data is now consolidated. A section of the final report appears to the left.

Data Matching Example ... Operational System of Records Data Warehouse Mark Carver SAS SAS Campus Drive Cary, N.C. Mark W. Craver Mark.Craver@sas.com M Craver Systems Engineer 01 Mark Carver SAS SAS Campus Drive Cary, N.C. 02 Mark W. Craver Mark.Craver@sas.com Consider the situation where you have records representing an employee stored across 3 different operational systems. The goal is to merge the records together into one master record for the employee. The problem is that the employee’s name is stored slightly differently in each of the data sources. An attempt to merge the records together based on employee name does not yield the desired results. 03 M Craver Systems Engineer SAS ... ...

Data Quality Process ... Operational System of Records Data Warehouse Mark Carver SAS SAS Campus Drive Cary, N.C. Mark W. Craver Mark.Craver@sas.com Mark Craver Systems Engineer 01 Mark Craver Systems Engineer SAS SAS Campus Drive Cary, N.C. 27513 Mark.Craver@sas.com DQ If, however, we were able to add a data cleansing process that applied fuzzy logic to the records such that they can be merged, we can attain the desired results. This is the type of functionality the algorithms in the SAS Data Quality Solution make possible. ... ...

ETL What is ETL? E T L How important is it in BI? Exercise one: data cleaning with SAS an MS Access Run SAS 9.1 and follow the demo, answer the questions as you go Exercise two: OLAP When you are finished you can try eTrainer

Exercise two: OLAP Answer the following on the back of the handout. Query 1: How many Sales Districts are there? Which district is consistently the top performer? Which quarter had the best revenue (use totals)? What’s the total revenue for 2004? Which region would you be concerned about and why? What’s the total revenue for 2003? Query 2: How many Sales Regions are there? How many Sales Districts in the USA? How many Sales Reps in NE USA? How many Sales Reps in the USA? Best and Worst Sales Reps? Query 3: Graph the results like this, what’s missing? How can you fix this? What’s happened in Q1 2005?

Query 1

Query 4 Create this report (Sales Reps by Revenue) Look for the SQL that is created and run for you, copy and save it (in a text file)