Record Linkage at the Minnesota Population Center Ron Goeken, Lap Huynh, Tom Lenius, and Rebecca Vick RecordLink Workshop, 2010 University of Guelph, May 24 th 2010

Introduction Overview of linkage process – Prelims vs. final releases Name commonness scores Error rate estimation Weights Looking ahead

Historical Record Linkage – U.S % sample % sample % sample 1880 complete-count % sample % sample % sample % sample

Historical Record Linkage at the MPC Primary goals are to create linked sets that are – Representative – Accurate

Historical Record Linkage at the MPC Representative links – We use a very limited set of variables to predict links to avoid linkage bias Block by birthplace, sex and race Given (first) name Surname (last) name Age

Historical Record Linkage at the MPC Accurate links – If there is more than one ‘potential’ link for a given person we exclude them all – We throw away a lot of potential links

Historical Record Linkage at the MPC Create given and surname and age similarity scores – Jaro-Winkler string similarity algorithm – 20% age difference score We apply name and age similarity thresholds to limit output of potential links

Additional Variables Based on Age Age – Age difference (absolute value, normalized)* – Age categories, in five-year groups*

Additional Variables Based on Name Phonetic Match (binary) – Double Metaphone – NYSIIS* Middle initials (if present) must not conflict (binary)*

Additional Variables Based on Name Name Commonness Scores* – Our answer to incorporating probabilistic information into the process without complete standardization of all name strings. – Proportion of records (by race, birthplace, and sex) in the 1880 data with a Jaro-Winkler score greater than 0.9 – Name commonness score works in tandem with a birthplace density measure, which is the proportion of 1880 records for specific birthplaces (by race and sex)

Classification of links Comparisons that beat the thresholds become ‘potential links’ that are classified as ‘true’ and ‘false’ links by two SVM models – One model includes age variables, the other does not* Link is accepted if both models call it a ‘true’ link and there are no conflicts

Name Commonness Table 6. Distribution of 1870 Records (Males) by Name Commonness Scores

Linkage Rate by Name Commonness

Linkage Rates by Name Commonness and Birthplace Population Size

Table 8. Linkage Rate for Native-Born 1870 Males by Birthplace Rank (number of males by birthplace) and Name Commonness Scores

Occupational Scores and Name Commonness

Estimating error rates Calculate migration rates by different slices of data, e.g. five-year age cats, age difference Split brothers Compare link made in one dataset to link made in another for same group of people Compare to linked set made by another independent source: Pleiades

Selected Linked Household – LINKTYPELAST70FIRST70LAST80FIRST80RELATE70RELATE80AGE70AGE80 household UNDERWOOD NORMAN Head52 household UNDERWOOD MARY UNDERWOOD MARY SpouseHead3349 household UNDERWOOD LUTHER Son11 household UNDERWOOD IRVING UNDERWOOD ERVIN Son 313 primary UNDERWOOD VANDER UNDERWOOD VANDER Son 110 household UNDERWOOD ROSA Daughtr18 household UNDERWOOD CHARLES Son8 household UNDERWOOD ADDI Daughtr5

Weights The weights are based on the linkable population, which is always based on the terminal census year data. Based on an iterative process We capped weight minimums and maximums (min is 1/5 the avg. weight for the subgroup; max is 4 times the avg. weight for subgroup)

Final Release Data Set Size, Males MALE nat-b whitefor-b whiteaf-am

Final Release Data Set, Females FEMALE nat-b white marriedsingleFormerly

Final Release Data Set Size, Couples COUPLE nat-b whitefor-b whiteaf-am

Looking Ahead Hope to alleviate small N problem in the future – Link 1900 and % samples to 1800 complete count – 1850 complete count database currently under construction – Hope to have complete count data for 1860, 1870, and 1900 in the future