Statistical Sampling Overview and Principles Alvin Binns

Provider X is identified for billing excessive ambulance services. A decision was made to pull all his/her ambulance services for a specified two years period. Results: - 3,000 claims, 7,000 lines and $1.8M in payments. Scenario

Time Cost Available resources Available staff Reasons for Sampling

What is Sampling? Sampling - is the selection of observations to acquire some knowledge of a statistical universe (population). From the characteristics of samples, we can infer the characteristics of universes, if the sample is representative of the universe.

In order for statistics to be good estimates of parameters, they must, on average, return the value of the universe parameter When the expected value of a statistic equals a universe parameter, we call the statistic an unbiased estimator of that universe parameter How Do I Get a Representative Sample?

How do you ensure that your statistic is an unbiased estimator? How Do I Get a Representative Sample? RANDOMIZATION!!!

A sample that is randomly selected from a universe yields sample statistics that are unbiased estimates of the universe parameters Many software packages, such as SAS and RAT-STATS have a valid random number generator Randomization

Another idea behind random sampling is that each sampling unit has a known probability of being selected Probability Sampling

Universe : An event or things of interest that the researcher wishes to investigate. Eg. All Medicare beneficiaries that received a left heart catheterization from Dr. John Doe between January 1, 2007 and June 30, 2008 paid up to September 30, Sampling Terms

Samples are usually drawn by taking a subset of sampling units from the total universe Sampling units are non-overlapping collection of elements from the universe that cover the entire universe (eg claims, beneficiaries) Sampling Terms

We can infer the values of the universe from the sample by the use of estimation Ideally, we would like gather information from the sample and then estimate that value for the entire universe These estimates calculated from the sample data are called statistics Estimation

Statistics SampleCensus StatisticParameter Estimates ENTITY CHARACTERISTIC

In an simple random sample where we had sampled 100 units out of 1000, suppose we had a $5,000 total overpayment from the sample The Mean Total Overpayment would then be: Estimation

Why Should I Care? HCFA Ruling 86-1 allows the use of statistical sampling for the purpose of estimating a provider’s overpayment to the Medicare trust fund Thus, we can use sampling to estimate overpayments to providers and avoid having to review the entire universe!

CMS guidelines for Statistical Sampling for Overpayment Estimation Program Integrity Manual Section 3.10 Some of the issues addressed are: –Methodologies –Sample Size –Estimation techniques CMS Sampling Guidelines

This replaces and clarifies (for older cases) the old HCFA Sampling Guidelines Appendix (CR 1363) –“This program memorandum (PM) provides clarified guidance and direction for Medicare carriers to use when conducting statistical sampling for overpayment estimation. The attached replaces the prior Sampling Guidelines Appendix for reviews conducted after issuance of this PM. For reviews conducted prior to this issuance, the attached are a clarification to aid interpretation of the earlier instructions, particularly where specific numbers are suggested” Sampling Guidelines

Simple Random Sampling Cluster Sampling Stratified Sampling Other Methodologies Sampling Methodologies

This is the most straightforward method of sampling X number of sampling units are randomly selected from Y total sampling units in the Universe Each sampling unit has an equal probability of being selected Simple Random Sampling

A cluster sample is a probability sample in which each sampling unit is a collection, or cluster, of elements A good example is the random selection of beneficiaries, then selecting all relevant claims from each beneficiary Cluster Sampling

A stratified random sample is one obtained by separating the universe elements into non- overlapping groups, called strata, and then selecting a simple random sample from each stratum An example of this would be samples involving multiple procedure codes, selecting simple random samples from each code Stratified random sampling generally has less sampling variability that other sampling designs Stratified Random Sampling

Stratified Random Sampling – Proportional Allocation Example Universe = 1000 Units Sample = 100 Units

PIM 3.10 states about sample sizes: –“It is neither possible nor desirable to specify a minimum sample size that applies to all situations” –“…real-world economic constraints must be taken into account. As stated earlier, sampling is used when it is not administratively feasible to review every sampling unit in the target universe. In practice, sample sizes may be determined by available resources. That does not mean, however, that the resulting estimate of overpayment is not valid as long as proper procedures for the execution of probability sampling have been followed. A challenge to the validity of the sample that is sometimes made is that the particular sample size is too small to yield meaningful results. Such a challenge is without merit as it fails to take into account all of the other factors that are involved in the sample design” PIM 3.10 – Sample Sizes

CSA procedure: –If we can, we like to pull at least 10% of the universe, however, this is not a rule that is set in stone –We must, at a minimum, pull at least 30 sampling units to satisfy distribution requirements through the central limit theorem PIM 3.10 – Sample Sizes

PIM 3.10 also states: –“In most situations the lower limit of a one- sided 90 percent confidence interval should be used as the amount of overpayment to be demanded for recovery from the physician or supplier. The details of the calculation of this lower limit involve subtracting some multiple of the estimated standard error from the point estimate, thus yielding a lower figure.” PIM 3.10 – Overpayment

It further states that: –“This procedure, which, through confidence interval estimation, incorporates the uncertainty inherent in the sample design, is a conservative method that works to the financial advantage of the physician or supplier. That is, it yields a demand amount for recovery that is very likely less than the true amount of overpayment, and it allows a reasonable recovery without requiring the tight precision that might be needed to support a demand for the point estimate. However, you are not precluded from demanding the point estimate where high precision has been achieved.” PIM 3.10 – Overpayment

What we really do then is calculate the Mean Total Overpayment and subtract a multiple of the standard error from it to achieve the lower level of the confidence interval PIM 3.10 – Overpayment

Below is the formula for the total variance for cluster sampling PIM 3.10 – Overpayment

Look at how the overpayments work: PIM 3.10 – Overpayment OP w/ Small Variance (Large n)OP w/ Large Variance (Small n) Mean Total Overpayment 90% Upper Limit 90% Lower Limit 90% Upper Limit 90% Lower Limit $$ Overpayment

Sample Size Comparison Analysis Variable : CLUSTAMT NMeanStd DevSumMinimumMaximum 2017, , , , , , , , , , , ,603.20

Estimation Of Total Amount Of Refund & It's Lower 1-sided 90% C.I. Difference if sample Size 5-10 beneficiary: $184, Difference in sample Size beneficiary: $207, Sample Size Univ. Size Mean Total Overpayment Std. Error 90% 1-sided Lower Bound 2044$768, , $483, $768, , $276, $806, , $91,617.92

Bottom Line Large Sample Sizes –Use when the expected overpayment is large –Use in high profile cases –Resource intensive –Increase precision even more using stratified sampling plans Small Sample Sizes –Use when the expected overpayment is small –Use in routine, low $ cases –Not as resource intensive –Does not work as well for stratified sampling

Sub-samples It is often beneficial to evaluate a sub- sample before moving to a full statistical sample. (sample size of about 30) Get a good idea of the point estimate (Mean Total Overpayment). Sampling for Consent Settlements.

Summary for Sampling Define the Universe Determine the sampling methodology Create the sampling Frame Determine sample size Create your sample After Sampling review is completed Perform overpayment Projection

Questions? Thank You!