Reconstructing historical populations from genealogical data An overview of methods used for aggregating data from GEDCOM files Corry Gellatly Department of History and Art History Utrecht University Workshop on Population Reconstruction IISH Amsterdam, February 2014

1. Overview  Why build a large genealogical database by aggregating hundreds of genealogical data (GEDCOM) files?  Research increasingly requires big data, to:  Understand large-scale population dynamics  between regions  over time  social, biological, cultural and economic aspects  Detect rare or ‘small-effects’  epidemiology (disease and intervention)  inheritance (genetics)  comparative life histories

2. GEDCOM files  Why use GEDCOM files for population reconstruction?  Pros  a standard file structure for representing information about familial relationships and life events  most popular format for storage and exchange of genealogical data  used internationally and widely available online  Cons  it is a highly flexible format that allows users to enter wildly incorrect information (if they wish to)

3. Data aggregation  Single GEDCOM files typically contain only a few hundred individuals, so we import hundreds of files into a single genealogical database  There are broadly 3 steps between import of files and the output, which is usable research datasets 1.Screening (to reject poor quality files) 2.Data cleaning 3.Linkage / de-duplication

4. Screening  Screening is carried out for various errors, e.g.  low mean number of offspring per family  individuals younger than 0 or very old (>110)  impossible relationships (due to age difference between individuals)  individuals occurring as different sexes  missing individuals  If errors are detected, then the file is either:  removed (in the case of obvious errors)  retained for further checking (in the case of ambiguous errors): e.g. where individuals have more than two parents – this can be due to adoption or incorrect family links between individuals

5. Cleaning  Example: date errors  If DOB is 1857  Born to 10 year old mother?  Wife 17 years older?  First of 5 children born at the age of 39?  If DOB is actually 1875  Born to 28 year old mother?  Wife 1 year younger?  First of 5 children born at the age of 21?

6. Dataset extraction  Definition of datasets is driven by research questions:  which timespan?  which region?  do we need complete families?  do we need dates of birth, death, marriage?  The identification of links between genealogies (or removal of duplicate individuals) is done during the process of dataset extraction

7. Linkage, de- duplication  Linkage fields  Day of birth, marriage or death (DOB, DOM, DOD)  Year of birth, marriage or death (YOB, YOM, YOD)  Surname  Given names  Sex  Problems  YOB, YOM, YOD more common than DOB, DOM, DOD (particularly in older data) but less unique to each individual  High inconsistency in recording of given names  Middle names included or excluded  Middle names used instead of first names  Abbreviated names  Nicknames (sometimes in brackets)

8. Linkage, de- duplication  T rade-off between data coverage and quality  Surname, given name, DOB  Low risk of false linkages, but high risk of missing linkages (due to problems with given names) and low data coverage  Surname, DOB  Low risk of false linkages, but low data coverage  Surname, YOB  High risk of false linkages, but high data coverage

9. Group- linking method  I ndividuals are identifiable by those they are related to  This principle is being applied to the problem of genealogical data, in which many records have YOB, but not DOB and given names are somewhat unreliable for linking  Group-linking string

10. Group-link test  T est with single GEDCOM file containing no duplicates 2,082 individuals; 971 marriages; 681 conceptive relationships; 1,913 conceptions

11. Group-link test  Percentage data coverage x Percentage of unique records within that data (÷ 100) gives an estimation of linkage power

12. Missing data  What about missing information?  The information on the siblings of these individuals is probably missing. Why? Because they appeared at marriage  This data is left censored, because these individuals appeared in the data after the event we are measuring (i.e. number and sex of siblings).

13. Missing data  Depending on what type of links we are trying to find, we may want to break up the string  String to link individuals based on their siblings  String to link individuals based on their marriages and children

14. Record de- duplication ( )  De-duplication of 17th century records from the genealogical database  Febrl program (Freely Extensible Biomedical Record Linkage)  17,488 records with Surname and YOB  Indexes  Surname > YOB  Surname > Group-link string 2 (sex + siblings)  Surname > Group-link string 3 (sex + marriages + offspring)  Comparison function  Winkler  Classifier  KMeans

15. Record de- duplication ( )  Results

16. Record de- duplication ( )  Results  Examples of matches in highest weight category (1,914 matches)

17. Record de- duplication ( )  Results  Examples of matches in lower weight category (10,434 matches)

17. Further work  Record linkage  Refine a method of probabilistic data matching that can identify linkages  where typo errors or name variations occur  possible date typos exist  there are missing persons in the family structure  Group-linking algorithm  Using the group-linking string as a start point to then check for existence of birth, marriage and death dates of relatives (where these exist) and performing matches on these variables  Inherently based on probabilistic matching

18. Acknowledgements  Netherlands Organisation for Scientific Research (NWO)  Project number : “Nature or nurture? A search for the institutional and biological determinants of life expectancy in Europe during the early modern period”  Colleagues at Utrecht University  Tine De Moor  Institutions for Collective Action team: