Overview of the Final Report and Findings from the Review of Sampling Methods in Extrapolated New Base-Year Generation Studies May 11-12, 2004

Types of New Base Year Generation Studies Nonextrapolated “Actual” diversion data collected from diversion programs implemented within the jurisdiction Extrapolated Includes diversion data collected from a sample of the non-residential businesses and extrapolated to the entire non-residential business sector

Number of New Base-Year Generation Studies Since April 1998, Board staff have reviewed 184 submitted new base year studies for Board consideration. 143 Nonextrapolated studies 41 Extrapolated studies

Background Board expressed concerns regarding validity of extrapolated studies Working group convened Board staff performed analyses on extrapolation vs. nonextrapolation study methodologies

Extrapolated vs. Nonextrapolated Study Data Comparisons Generation pounds per person per day was 79% higher for extrapolated methodologies Diversion rates for extrapolated studies were significantly higher (53%) than nonextrapolated studies (41%). Average source reduction tonnage in extrapolated studies exceeded nonextrapolated studies by an average of 300%.

Response to Concerns regarding New Base-Year Study Data Base-Year Modification Certification Request form Board staff began on-site verification of new base-year generation study data Board pursued independent, third-party review of extrapolated study methodologies

Scope of Review Review 20 extrapolated studies. Only 2 of the 20 studies reviewed submitted sufficient data. Provide observations and recommendations for improving extrapolated methodologies used. Impact of poor quality or lack of study data.

Findings The findings in the final report address the impact of errors relating to: Study design; Sampling errors; and Technical data components of nonresidential diversion surveys.

Study Design Jurisdictions should list assumptions made in the study design and use sample data to assess their validity and appropriateness. Studies did not include listing of such assumptions. Jurisdictions did not use the study data to test the validity of assumptions.

Sampling Errors Accurate data on the selection of samples is needed for Board staff to fully evaluate the sampling methodology. Lack of documentation regarding sample selection. Studies employed nonrandom sampling methods.

Technical Data Lack of documentation regarding nonresponse. Most jurisdictions provided outlier analysis, but did not address all study design variables. Lack of access to study data.

Additional Analyses Case Study of One Extrapolated New Base- Year Generation Study Review of 35 extrapolated new base-year generation study data Diversion of Top 10 Businesses Compared to All Surveyed Businesses

Commercial Business diversion Represents Approximately 38% of Total Diversion 25% 38% 37% Non Residential Business Audits Res/ADC/Landfill Salvage Residential

New Base Year Case Study: Impact of the Largest Businesses (333) Audited Relative to the Total Business Audits (approx. 7,600) top 10top 20top 30 Business Generators % of Diversion tonnage from Businesses Audits

Diversion of Top 10 Businesses Compared to All Surveyed Businesses By surveying 10 top businesses: 94% of the jurisdictions captured 50% or more of the total diversion; 60% of the jurisdictions captured 80% or more of the total diversion; and 37% of the jurisdictions captured 90% or more of the total diversion.

Maximizing Efficiencies in New Base-Year Generation Studies Many of the jurisdictions would have been better off targeting their largest diverters and submitting a nonextrapolated study. These jurisdictions could have captured most of the industrial/commercial diversion without having to address problems associated with sample selection, nonresponse, outliers, or choosing an estimator.

Future Efforts Board staff will: Update the Board’s Base-Year Modification Request Certification form for extrapolated studies. Update the Board’s Diversion Study Guide to explicitly underscore the importance of documenting all critical aspects of study design, implementation and diversion rate calculation. Develop a Web-based reporting format.

Future Efforts (continued) Board staff will continue to Assist jurisdictions regarding their future new base-year generation studies; and. Stress the benefits of targeting the largest businesses first to measure the non-residential sector.

Summary Board staff were unable to perform a complete statistical review for 18 of the 20 studies due to insufficient study data. Findings support the value of targeting the top 10 businesses.