The Role of Local Specificity in the Interpretation of Small Area Estimation Benmei Liu Scott Gilkeson Gordon Willis Rocky Feuer 2012 FCSM Statistical Policy Seminar December 4, 2012

Outline I.Overview of small area estimation II.The importance of local specificity and how it could affect data use III.An example from a recent project to estimate cancer risk factors and screening behavior IV.Discussion 2

I. Overview of Small Area Estimation (SAE)  The demand for survey estimates for small areas (small geographic areas or domains) has increased in many different areas of application (e.g., income and poverty, education, health, substance use) over the past several decades  The standard direct estimation methods for survey data cannot provide reliable estimates due to the small sample size  Model-based methods that combine information from multiple related sources have been developed to increase the precision 3

Basic SAE Model and Estimates 4

II. The importance of Local Specificity  We label the information about the use of local versus borrowed data based on the SAE techniques as local specificity  We propose that the term local specificity be used as a generalizable and intuitively understandable term for the degree to which local data contribute to the small area estimate for a specified area 5

The importance of Local Specificity (Cont’d)  Local specificity can be an important indicator of fitness for use  We argue that local specificity provides unique information that is not otherwise available  For local data users, a measure of local specificity could be useful  A measure of local specificity was not provided on any of the government websites that release small area estimates data (e.g., SAIPE, NAAL, NSDUH) 6

III. Communicating Local Specificity to End Users: An Example  Combining information from two health surveys to enhance small-area estimation (Raghunathan et al. 2007; Davis et al. 2010)  Project led by National Cancer Institute, with collaboration by:  National Center for Health Statistics  National Center for Chronic Disease Prevention and Health Promotion  University of Michigan  University of Pennsylvania  Information Management Services 7

Motivation for the Project  Cancer screening and risk factor data are of great interest to cancer control planners at the state and sub-state level, but accurate local statistics have been difficult to obtain  Different surveys have different strengths  Combining information from surveys and borrowing strength from other sources (e.g., Census or administrative records) using small area modeling approach could improve small- area estimates 8

Surveys Used  Behavioral Risk Factor Surveillance System (BRFSS) – the largest U.S. survey tracking health conditions and risk behaviors at the state and sub- state level since 1984 Limitations: Potential nonresponse bias; Undercoverage of hhlds without landline phones  National Health Interview Survey (NHIS) – the principal source of information on the health of the civilian noninstitutionalized population of U.S. since 1957 Limitations: Smaller sample size; only includes data on about ¼ of U.S. counties 9

Project Description  Bayesian methods are developed to combine information from the two surveys; also incorporated telephone coverage rates from the Census  National Cancer Institute released estimates for two time periods: and ( - Smoking, mammography, and pap smear - Counties, health service areas, and states  Current work involves including component for cellphone-only households and for the recent periods 10

Focus Group Suggestions  Conducted two focus groups with cancer control planners and public health professionals at the Comprehensive Cancer Control Leadership Institute in June 2010  Recommendations:  Include these estimates within NCI’s State Cancer Profiles website (  The website is a comprehensive system of interactive maps and graphs enabling the investigation of cancer trends at the national, state, and county level  Need a way to describe the differences between the bias- adjusted model-based estimates and existing direct estimates  Data users would appreciate an indicator like local specificity to validate the estimate against local evidence 11

Issues on Communicating Local Specificity 1)How should it be measured? 2)What should it be labeled? 3)What thresholds should be set in assigning values to it? 12

1) Measuring Local Specificity  The bias-adjusted SAE model is complex and lacks an explicit shrinkage factor  The concept of borrowed strength still applies, depending primarily on the combined BRFSS and NHIS sample size within the area  NHIS sample size is confidential. The sample size of the combined sample is close to the BRFSS sample size  BRFSS sample size is published, and alone was the best practical measure of the amount of local data 13

2) Labeling Local Specificity  Presenting the BRFSS sample size as a number along with the estimates didn’t convey the message of local specificity  Developed the term local specificity and selected qualitative (i.e., high, medium, and low) rather than quantitative descriptors 14

3) Assigning Thresholds  Selected BRFSS sample size of 50 as the threshold for low local specificity  Determining break points for the categories of local specificity deserves further study 15

Ratios of model-based county level current mammography screening rate over the bias-corrected BRFSS direct estimate 16

Small area estimates of mammography screening by county in Pennsylvania, with a mini-map showing local specificity 17 Warren county percentage = 65.9 ( ) Westmoreland county percentage = 64.8 ( )

IV. Discussion  Our experience has convinced us that such a measure is critical for end users in their use and interpretation of results  The potential importance of local specificity should not be under-emphasized, given that users demand more from SAEs than from the results of most other statistical models  There is no single computational formula for calculating levels of local specificity that will apply generally across various models and further research is needed  Whenever estimates are based on non-ignorable levels of borrowed strength, it is vitally important to disseminate analyses in such a way that local specificity, as an important index of fitness for use, be conveyed to data users in a clear and unbiased manner 18

Thank you! Contact information: Benmei Liu, Ph.D. Survey Statistician National Cancer Institute 19