1 Understanding and Measuring Uncertainty Associated with the Mid-Year Population Estimates Joanne Clements Ruth Fulton Alison Whitworth

2 Context Improving Migration and Population Statistics (IMPS) Project Quality Strand “Establish quality measures for population statistics” No international precedent for this work

3 Issues Estimates compiled from a wide range of administrative sources plus some survey and Census data Source data subject to sampling and non- sampling errors Lack of independent data with which to corroborate How to estimate each potential error and combine these in one measure?

4 Aim and Objectives Aim Improve understanding, measurement and reporting of the quality of population estimates Objectives –Describing the sources of uncertainty –Developing methods for measuring uncertainty for each issue and combining them into one measure –Eventually feeding findings into ONS quality reports

5 Presentation Outline Summarise population estimates methodology Summarise previous research on quality Detail proposed error measurement methodology –Illustrate by applying to local authority (LA) mid population estimates Outline emerging proposals for further work to achieve robust quality measures

6 Calculating LA Population Estimates e.g. Southampton UA

7 Calculating LA Population Estimates (cont) Adjustment

8 Calculating LA Population Estimates

9 Previous Research Quality of Population Estimates Past experience of inter-censal errors Sampling error and expert opinion of non- sampling error in components of estimates Quality of Population Projections Accuracy of past projections Use of variant projections Simulation methods using error distributions for the components of projections (stochastic forecasting)

10 Proposed Methodology: Initial Assessment of Quality Issues Map out the procedures and data sources used to derive population estimates Identify associated quality issues Identify the importance of these issues

11 LA Population Estimates: Initial Assessment of Quality Issues Brief assessment of the evidence for each component For example: Internal Migration –Relies on GP registration data –Assumes patients reregister within a month of moving (known issue for students leaving university)

12 Proposed Methodology (cont): Detailed Investigation of Quality Issues Quantify uncertainty using statistical theory, empirical evidence and / or expert opinion Both sampling and non-sampling errors

13 LA Population Estimates: Detailed Investigation of Quality Issues Attributing a potential uncertainty range and distribution to each component Not each quality issue Made relatively simplistic distribution assumptions (Normal or Uniform) Assumed same level of uncertainty across LAs

14 LA Population Estimates: Detailed Investigation of Quality Issues Estimating uncertainty relative to size of local authority component For example: Could assume potential error in annual local authority births estimate N(0, X% of estimated births) –Assume similar error distributions by year

15 Proposed Methodology (cont): Overall Quality Measure Mathematically complicated to combine a large number of potential error measures into one quality measure –Errors may be correlated –Distributions not all normally distributed Developed a Simulation methodology

16 LA Population Estimates: Simulation For each local authority randomly generate errors for each component –Using previously developed error distributions Mid-2001 error estimate + Births 01/02 error estimate - Deaths 01/02 error estimate + Internal In-Migrants 01/02 error estimate -Internal Out-Migrants 01/02 error estimate + ….. + Births 02/03 error estimate - Deaths 02/03 error estimate + …

17 LA Population Estimates: Simulation Calculate error in mid-2006 estimate by combining the errors generated for each component in each year up to 2006 Repeat process 1000 times Obtain distribution of potential error in mid local authority estimate

18 Findings: Potential Error Distribution

19 Findings: Measuring Uncertainty in Population Estimates Simulation methodology allows measures of uncertainty to be calculated for population estimates. But, in reality, there is uncertainty in these measures of uncertainty, as… –Only as good as the error assumptions made for each issue / component of change –Very difficult to exactly measure non-sampling error

20 Findings: Key Components of Uncertainty Uncertainty in population estimates related to: –Size of each component –Error distribution assumptions Key components driving uncertainty in LA estimates: –Mid-2001 base population estimate –Internal Migration –International Migration (IPS) –Specific components important in specific LAs e.g. Foreign Armed Forces

21 Extending the Methodology Current assumptions in the estimation of uncertainties are inadequate –Need to examine issues within each component –Consider LA variation in uncertainties within each component Currently focussing on refining error distributions for key drivers of uncertainty within LA estimates –Internal Migration –International Migration

22 Estimating Uncertainty - Internal Migration: Emerging Proposals for Further Work Building upon previous work, investigate uncertainty in estimates related to: Time lags between moving and reregistering Moves not captured by GP registers because patients were not registered when data were extracted The scale of constraining GP register data to NHSCR

23 Estimating Uncertainty - International Migration: Emerging Proposals for Further Work Calculating sampling error of IPS estimates Investigating uncertainty around migrant and visitor switcher estimates Investigate uncertainty within methods used to calculate LA migration estimates from the IPS –For example, in LA emigration model used

24 Future Outcomes of this Work Increased understanding of sources of error in the population estimates and their relative importance Ability to focus resources for research on key sources of uncertainties Additional information which could feed into Quality Reports This work is intended to improve our understanding of the uncertainty in population estimates, rather than provide exact estimates of uncertainty

25 Summary Measuring Uncertainty of Population Estimates Estimating their error margin is complex Detailed quality assessment of each component required to obtain a robust measurement Simulation methods are a plausible approach to approximately measure the overall quality of an estimate Ongoing work on estimating uncertainty in migration components