Presentation is loading. Please wait.

Presentation is loading. Please wait.

Using Reported Data as Matching Variables in Record Linkage

Published byBlake Morgan Modified over 5 years ago

Similar presentations

Presentation on theme: "Using Reported Data as Matching Variables in Record Linkage"— Presentation transcript:

1 Using Reported Data as Matching Variables in Record Linkage
By Bill Iwig, Kara Daniel, Tom Pordugal, and Stan Hoge National Agricultural Statistics Service

2 NASS Use of Record Linkage
Match new list sources to the Farm Register Identify duplication within the Farm Register Match Area Frame records to the Farm Register for measuring coverage National Agricultural Statistics Service

3 Record Linkage Procedures
Matching variables are divided into components Matching components are assigned agreement and disagreement weights Records are only compared within blocks Sum of agreement and disagreement weights compared to thresholds National Agricultural Statistics Service

4 Record Linkage System Enhancement
Use data items as matching variables Provided through SuperMatch software feature Parameters allow “close” values to match and be assigned a reduced agreement weight National Agricultural Statistics Service

5 Identifying Duplication on 2002 Census of Agriculture Data File
2.85 million records on the Census Mail List Positive data for 1.1 million at the time of record linkage Numerous steps to eliminate duplication prior to data capture Duplication still exists! National Agricultural Statistics Service

6 National Agricultural Statistics Service
Using Census Reported Data as Matching Variables to Identify Duplication 40 data items used “0” values not considered for matching Fewer than 10 positive values for most records National Agricultural Statistics Service

7 Initial Record Linkage Parameters
Agreement weight = 1 Disagreement weight = 0 “Non-tolerable” percentage difference =11 Sum of weights threshold = 5 National Agricultural Statistics Service

8 Pro-rated Agreement Weight Examples
A = 100, B = 95, Wt = .52 A = 20, B = 19, Wt = .52 A = 20, B = 18, Wt = 0 National Agricultural Statistics Service

9 National Agricultural Statistics Service
Results Approximately 1500 potential duplicates identified Actual number of duplicates less than 500 National Agricultural Statistics Service

10 Recommendations for Effective Application of Data Matching Feature
Evaluate distribution of response differences for true duplicates Evaluate handling of “0” values Highly correlated variables Edited and imputed variables Threshold value for matching National Agricultural Statistics Service

Download ppt "Using Reported Data as Matching Variables in Record Linkage"

Similar presentations

Re-design of the trade in commercial services program in Canada October 2010 OECD Working Party on Trade in Goods and Services.

Re-design of the trade in commercial services program in Canada October 2010 OECD Working Party on Trade in Goods and Services.
Missing data – issues and extensions For multilevel data we need to impute missing data for variables defined at higher levels We need to have a valid.

Missing data – issues and extensions For multilevel data we need to impute missing data for variables defined at higher levels We need to have a valid.
Discussion of topic VI Censuses Work Session on Data Editing Vienna, April 21 st -23 rd 2008 Heather Wagstaff & Thomas Burg.

Discussion of topic VI Censuses Work Session on Data Editing Vienna, April 21 st -23 rd 2008 Heather Wagstaff & Thomas Burg.
Efficient modelling of record linked data A missing data perspective Harvey Goldstein Record Linkage Methodology Research Group Institute of Child Health.

Efficient modelling of record linked data A missing data perspective Harvey Goldstein Record Linkage Methodology Research Group Institute of Child Health.
Efficient modelling of record linked data A missing data perspective Harvey Goldstein Record Linkage Methodology Research Group Institute of Child Health.

Efficient modelling of record linked data A missing data perspective Harvey Goldstein Record Linkage Methodology Research Group Institute of Child Health.
Capturing Sensitive Data & Data Linkage. Capturing Sensitive Data Data Protection Act 1998 (Section 33) – Allows data to be used for research purposes.

Capturing Sensitive Data & Data Linkage. Capturing Sensitive Data Data Protection Act 1998 (Section 33) – Allows data to be used for research purposes.
© 2007 John M. Abowd, Lars Vilhuber, all rights reserved Introduction to Record Linking John M. Abowd and Lars Vilhuber April 2007.

© 2007 John M. Abowd, Lars Vilhuber, all rights reserved Introduction to Record Linking John M. Abowd and Lars Vilhuber April 2007.
Www.stat.gov.lt 20-21 March 2013 ESSnet DWH - Workshop IV DATA LINKING ASPECTS OF COMBINING DATA INCLUDING OPTIONS FOR VARIOUS HIERARCHIES (S-DWH CONTEXT)

March 2013 ESSnet DWH - Workshop IV DATA LINKING ASPECTS OF COMBINING DATA INCLUDING OPTIONS FOR VARIOUS HIERARCHIES (S-DWH CONTEXT)
Data Quality Class 10. Agenda Review of Last week Cleansing Applications Guest Speaker.

Data Quality Class 10. Agenda Review of Last week Cleansing Applications Guest Speaker.
1 The 2010 Census Coverage Measurement Survey Patrick J. Cantwell U.S Census Bureau Annual Meeting of the Association of Public Data Users September 25,

1 The 2010 Census Coverage Measurement Survey Patrick J. Cantwell U.S Census Bureau Annual Meeting of the Association of Public Data Users September 25,
1 Editing Administrative Data and Combined Data Sources Introduction.

1 Editing Administrative Data and Combined Data Sources Introduction.
Data mining and statistical learning, lecture 5 Outline  Summary of regressions on correlated inputs  Ridge regression  PCR (principal components regression)

Data mining and statistical learning, lecture 5 Outline  Summary of regressions on correlated inputs  Ridge regression  PCR (principal components regression)
© John M. Abowd and Lars Vilhuber 2005, all rights reserved Introduction to Probabilistic Record Linking John M. Abowd and Lars Vilhuber March 2005.

© John M. Abowd and Lars Vilhuber 2005, all rights reserved Introduction to Probabilistic Record Linking John M. Abowd and Lars Vilhuber March 2005.
© 2007 John M. Abowd, Lars Vilhuber, all rights reserved Estimating m and u Probabilities Using EM Based on Winkler 1988 Using the EM Algorithm for Weight.

© 2007 John M. Abowd, Lars Vilhuber, all rights reserved Estimating m and u Probabilities Using EM Based on Winkler 1988 "Using the EM Algorithm for Weight.
FINAL REPORT: OUTLINE & OVERVIEW OF SURVEY ERRORS

FINAL REPORT: OUTLINE & OVERVIEW OF SURVEY ERRORS
Household Surveys ACS – CPS - AHS INFO 7470 / ECON 8500 Warren A. Brown University of Georgia February 22, 2011 1.

Household Surveys ACS – CPS - AHS INFO 7470 / ECON 8500 Warren A. Brown University of Georgia February 22,
Effects of Income Imputation on Traditional Poverty Estimates 1987-2007 The views expressed here are the authors and do not represent the official positions.

Effects of Income Imputation on Traditional Poverty Estimates The views expressed here are the authors and do not represent the official positions.
Work Package 5: Integrating data from different sources in the production of business statistics Daniel Lewis Office for National Statistics (UK)

Work Package 5: Integrating data from different sources in the production of business statistics Daniel Lewis Office for National Statistics (UK)
Introduction to Record Linking John M. Abowd and Lars Vilhuber April 2011 © 2011 John M. Abowd, Lars Vilhuber, all rights reserved.

Introduction to Record Linking John M. Abowd and Lars Vilhuber April 2011 © 2011 John M. Abowd, Lars Vilhuber, all rights reserved.
The paired sample experiment The paired t test. Frequently one is interested in comparing the effects of two treatments (drugs, etc…) on a response variable.

The paired sample experiment The paired t test. Frequently one is interested in comparing the effects of two treatments (drugs, etc…) on a response variable.

Similar presentations

Ads by Google