Criteria for Evaluating Post-2000 Estimates Chuck Coleman Population Estimates Branch Population Division U.S. Census Bureau Prepared for the Spring 2001 FSCPE Meetings, March 27, Washington, DC

Background PEB and some states independently produced April 1, 2000 county population estimates using different data and techniques. Want to determine if state methods should augment PEB’s current method. –Averaged methods also to be considered. Need way to decide which is best. From Post-2000 Estimates Methods Committee.

Goals Minimize bias. –Want estimates errors to be zero “on average.” Maximize accuracy. –Want errors to be “small.” Minimize extreme errors (a.k.a. outliers.) –Want method to work “well” everywhere. Individual goals may be in conflict. –Ex: Most accurate set may have most extreme errors.

Requirements Measures of the criteria Overall decision rule –Accounts for conflicts between individual criteria.

Bias Measure Mean Algebraic Percentage Error (MALPE) Measures mean bias. –“Is the expected value of an error zero?” i indexes the n counties in a state. E i, A i are the estimated and actual populations for county i, respectively.

Accuracy Measures Decided to do separate evaluations using two accuracy measures. –Mean Absolute Percentage Error (MAPE) –Webster’s Rule

Accuracy Measures - MAPE Commonly used. Easy to interpret. Does not account for size of county. –Small counties expected to have larger relative errors.

Accuracy Measures - Webster’s Rule Accounts for county size. Satisfies fairness norms per apportionment literature. Hard to interpret.

Outlier Measure Both MAPE and Webster’s Rule can be viewed as sums of individual loss functions: Sort L i in ascending order. For some , 0 L .

Outlier Measure (cont.) Can be called “anti-  trimmed-mean.” Measures “area” of upper tail of error distribution.   LiLi LL 0

Outlier Measure (cont.) Choice of . –Too low allows small number of extreme values to dominate. In the limit, –Too high causes nonextreme values to swamp measure. Decided to use  = 5%.

Decision Rule Modified Borda Vote on the three criteria For each criterion, a method gets –3 votes if best –2 votes if second-best –1 vote if third-best –0 votes otherwise

Decision Rule (cont.) For each method, votes on all criteria are summed. Most votes wins.

Problems Chosen method may not be statistically distinguishable from others. –Can do Markowitz-Xu (1994) tests on all errors. May not have enough methods to test. –Bias, accuracy pairwise testable. –Extreme values not testable. is purely descriptive statistic. No null hypothesis. Thus, may choose technique without statistical justification.

Summary P2KM Committee created decision rules to select “best” method for estimating county populations within a state. Criteria –Bias –Accuracy –Extreme values Modified Borda Vote on criteria “Best” method may not be statistically so.

References Armstrong, J. Scott (1978), Long-Range Forecasting: From Crystal Ball to Computer, New York: Wiley. Balinski, Michael L. and H. Peyton Young (1982), Fair Representation: Meeting the Ideal of One Man, One Vote: New Haven: Yale University Press. Coleman, Charles D. (1999a), “Nonparametric Tests for Bias in Estimates and Forecasts,” American Statistical Association: 1999 Proceedings of the Business and Economic Statistics Sections, Coleman, Charles D. (1999a), “Metrics for Assessing the Accuracy of Cross- Sectional Estimates and Forecasts,” presented to the 1999 meetings of the Southern Demographic Association, San Antonio, TX, October. Coleman, Charles D. (2000a), “Tests for Differential Bias in Estimates and Forecasts,” presented to the 2000 meetings of the Southern Demographic Association, New Orleans, LA, October. Coleman, Charles D. (2000b), “Webster’s Rule for Assessing the Accuracy of Cross-Sectional Estimates and Forecasts,” manuscript, U.S. Census Bureau. Coleman, Charles D. with Melanie Martindale (2000), “Proposed Outlier Measure: The Anti-Beta Trimmed-Mean,” Proposal to the Post-2000 Estimates Methods Committee by the Outlier Subcommittee.

References Markowitz, Harry M. and Gan Lin Xu (1994), “Data Mining Corrections,” Journal of Portfolio Management, 2,