[ March 9, 2017] [ Bill Bowles, Audit Supervisor] Data Analytics for Improved Decision Making [ March 9, 2017] [ Bill Bowles, Audit Supervisor]

Making Government Work Better Workshop Objective Review two different Sampling approaches, and the benefits of each Use of Data Analytics with large data sets to make valid judgments Become familiar with several key ACL commands 9/19/2018 Making Government Work Better

Making Government Work Better The Challenge 9/19/2018 Making Government Work Better

Sampling is for all of us Best Practices in Audit Sampling Sampling is for all of us Engineers Health care providers Data processing professionals Puplic Safety professionals Time card departments Human Resource professionals Manufacturing Transportation professionals Management

The Sampling Challenge Best Practices in Audit Sampling The Sampling Challenge I know how to measure and test things, but I’m not a statistician. Tools are available to let you leverage off of the knowledge needed to make fact based decisions Use of these tools will make you a sought after individual

Best Practices in Audit Sampling Some Sampling Basics Population is large relative to the sample size Must be Random; Each member of the population must have an equal chance of being chosen Data values should be reasonably consistent Two basic types of samples: Statistical or Biased(Judgmental)

Best Practices in Audit Sampling Some Sampling Basics Statistical- You can make a scientifically valid projection of a attribute of a population, based on the results of a randomly selected sample test. Biased(Judgmental- You can make a judgement on the non randomly selected sample, but CANNOT project the results to the general population. The results may point to an inference about the population, but it cannot be used as a scientifically valid projection of the population.

Best Practices in Audit Sampling Binomial Sampling Binomial sampling provides the most information about the characteristics of a population, but requires larger sample sizes. It provides: Confidence level Projection of a finite value of the population Sampling Error Example: With a 95% confidence level, there are 75.5%, +/-5% residents who are registered to vote in this population of 68,000 residents.

Best Practices in Audit Sampling Poisson Sampling Poisson sampling provides support for an assumption about a population, such as the population is compliant: Its advantage is that it requires very small sample sizes, but only defines the maximum error rate of a population. It produces a Confidence level Maximum Error Rate of the population Example: With a 95% confidence level, there are no more than 5% of the 68,000 voters with bad addresses

Which Sampling method to choose Best Practices in Audit Sampling Which Sampling method to choose Poisson: If you believe your population is compliant, use this method to validate your assumption. Good choice for compliance audits Binomial: If you need to know the value of an attribute of a population with a sampling error. Good choice for compliance audits the large non compliance rates or effectiveness or efficiency assessments.

Which Sampling method to choose Best Practices in Audit Sampling Which Sampling method to choose This presentation will focus on Binomial Sampling

Best Practices in Audit Sampling Some Sampling Basics Avg 1 Std Deviation

Picking your sample size Best Practices in Audit Sampling Picking your sample size OR Use a tool

Picking your sample size Best Practices in Audit Sampling Picking your sample size http://www.raosoft.com/samplesize.html http://www.surveysystem.com/sscalc.htm

Best Practices in Audit Sampling Your population data When choosing a sample from the population, make sure The data is reasonably consistent. The sample is random. Every member of the population has an equal chance of selection

Reasonably consistent data Best Practices in Audit Sampling Reasonably consistent data

Reasonably consistent data Best Practices in Audit Sampling Reasonably consistent data Small Std Deviation relative to the average Data variation is small

Reasonably consistent data Best Practices in Audit Sampling Reasonably consistent data

Reasonably consistent data Best Practices in Audit Sampling Reasonably consistent data Relatively Small Std Deviation relative to the average Data variation is small

Best Practices in Audit Sampling Our Challenge You manage a healthcare office, and need to know what % of all your office visits are between $40 and $50. You have 544,459 claims, with a total value of $20.2 million and each claim can involve multiple charges. Can I use sampling to reliably answer this question?

Sample Size calculator Best Practices in Audit Sampling Sample Size calculator

Lets put the theory to work Best Practices in Audit Sampling Lets put the theory to work Sample Metrics $40 to $50 Count Percent of Count Percent of Field Amount_Paid NO 294 76.56% 71.9% 10,351.62 YES 90 23.44% 28.1% 4,045.54 Totals 384 100% 14,397.16 With a 95% confidence level, we conclude that 28.1% of the $20.2 million in claims, +/- 5% is between $40 to $50 per claim.

Lets put the theory to work Best Practices in Audit Sampling Lets put the theory to work Population Metrics $40 to $50 Count Percent of Count Percent of Field Amount_Paid NO 428,591 78.72% 74.32% 15,058,062.88 YES 115,868 21.28% 25.68% 5,202,967.30 Totals 544,459 100% 20,261,030.18 The actual % of claims between $40 and $50 is 25.7%, which falls within the projection of 28.1%, +\- 5%. Sampling error is 2.4%

Best Practices in Audit Sampling Tricky Data My data is not uniform. It varies all over the place. Can I use sampling? YES, but Stratify

Best Practices in Audit Sampling Stratification 723,746 records Range is from $40 to $75K Total revenue = $3.5 billion

Best Practices in Audit Sampling Stratification Std Deviation is twice the Average Range is $40 to $75,000

Best Practices in Audit Sampling Stratification

Best Practices in Audit Sampling Stratification CAUTION If you don’t stratify your population where you have large variations in data values, your sampling results will NOT represent the population.

Without Stratification Best Practices in Audit Sampling Without Stratification Population Sample Produces poor results

Best Practices in Audit Sampling Stratification Break the 384 samples into groups based on population in each group

Best Practices in Audit Sampling Stratification Group Sample Size = Tot Sample / Total Population * Group Population

Best Practices in Audit Sampling Stratification Relatively even sizes of total dollars for each group 139 78 51 54 38 23

Best Practices in Audit Sampling Stratification Perform the testing of the samples

Best Practices in Audit Sampling The challenge You own a cleaning business with 300 employees with revenues of $5.9 million per year, and a Gross profit of $2.3 million. Your service does not clean bathrooms, and you are beginning to notice some customers are going to competitor’s who clean bathrooms. You want to use data analytics to assess whether it would make sense to expand your business to include bathroom cleaning.

The challenge (Continued) Best Practices in Audit Sampling The challenge (Continued) Each employee services an average of 11 customers per week, and spends 2 hours for each cleaning. You receive $36 per cleaning, and you pay your employees 2 hours per cleaning at $10 to $12 per hour, based on experience. You estimate that if you get into the bathroom cleaning business, your employees will need to spend 3 hours per cleaning, and you will need to provide cleaning supplies of approximately $6.00 per cleaning.

Best Practices in Audit Sampling Key Questions How big is the market for this option? Will I maintain my profit margin What is my risk?

Best Practices in Audit Sampling Current State We have gross sales of $5.9 million with payments of $3.6 million for cleaning services We have 300 employees with a Gross Profit of $2.3 million. We have 165,550 customer cleanings performed during the past year

Best Practices in Audit Sampling Current State 165,400 3,597,445 5,936,400 300 2,334,183

Best Practices in Audit Sampling Marketing Research We were able to obtain a list of customer billings for the previous 12 months for 151,543 cleanings that totaled $10.7 million for cleanings that ranged from $60 to $80 per cleaning. We know that this data includes some customers who have their bathroom included in the service. We also know that some of the cleaners are paid for cleaning supplies. We would like to know how many of these cleanings include bathroom service.

Best Practices in Audit Sampling Marketing Research We will pick a random sample of the customer billings and attempt to project to the entire population: The % of billings that include bathroom service The % of total revenue that is for bathroom service The % of billings for bathroom service where the provider had to pay for cleaning service We want our confidence in the projection to be 95%, and we want our sampling error to be less than +/- 5%

Best Practices in Audit Sampling Marketing Research 151,543 $10.7 million

Best Practices in Audit Sampling Marketing Research We will select 383 random samples and project thr results to the population

Best Practices in Audit Sampling Marketing Research

Best Practices in Audit Sampling Marketing Research 383

Best Practices in Audit Sampling Marketing Research

Best Practices in Audit Sampling The Projection It is concluded, with a 95% confidence that 73.8%, +/-5%, of the billings which total $10.7 million include the cleaning of bathrooms. This indicates that we can compete for approximately $7.9 million, +/- $535K in cleaning business that includes bathrooms.

Best Practices in Audit Sampling The Projection Average = $70.33 StdDev = $3.93

Best Practices in Audit Sampling $70.33 $66.40 $62.50 Std = 3.93

Projecting my market share Best Practices in Audit Sampling Projecting my market share If we set our cleaning service rate at $70, we would be competitive with the market, however, If we set our rate at $66.50, approximately 82% of my competition would be higher than me.

Best Practices in Audit Sampling Profitability Re-compute the employee table by making the following changes: Change the hours per cleaning from 2 to 3 Change sales per cleaning to $66.50 Add a $6.00 cost for cleaning supplies for cleaning The results: Total Sales = $11.0 million Total Cost = $6.38 million Gross Profit = $4.6 million

Making Government Work Better The Future State 6,369,222 4,572,145 11,029,025 9/19/2018 Making Government Work Better

Making Government Work Better The Challenge Questions Tel: (857) 242-5773 Email William.bowles@sao.state.ma.us Making Government Work Better 9/19/2018