Download presentation
Presentation is loading. Please wait.
Published byEaster Waters Modified over 9 years ago
1
A Comparison of Survey Reports Obtained via Standard Questionnaire and Event History Calendar: Initial Results from the 2008 EHC “Paper” Test Jeff Moore Statistical Research Division Presentation to the ASA/SRM SIPP Working Group Alexandria, VA November 17, 2009
2
Paper prepared by: Jeff Moore, Jason Fields, Gary Benedetto, Martha Stinson, Anna Chan, & Jerry Maples For presentation at: American Association for Public Opinion Research (AAPOR) May 14-17, 2009
3
Overview Background: - SIPP “re-engineering” - event history calendar (EHC) methods Goals & Design of the 2008 EHC Paper Test Preliminary Results Summary / Conclusions / Next Steps
4
SIPP Survey of Income and Program Participation - income/wealth/poverty in the U.S.; program participation dynamics/effects - interviewer-administered; longitudinal - panel length = 3-4 years
5
SIPP Survey of Income and Program Participation - income/wealth/poverty in the U.S.; program participation dynamics/effects - interviewer-administered; longitudinal - panel length = 3-4 years Key Design Feature: - 3 interviews/year, 4-month reference pd.
6
SIPP Re-Engineering Implement Improvements to SIPP
7
SIPP Re-Engineering Implement Improvements to SIPP - reduce costs - reduce R burden - improve processing system - modernize instrument - expand/enhance use of admin records
8
SIPP Re-Engineering Implement Improvements to SIPP - reduce costs - reduce R burden - improve processing system - modernize instrument - expand/enhance use of admin records Key Design Change: - annual interview, 12-month reference pd., event history calendar methods
9
SIPP Re-Engineering Implement Improvements to SIPP - reduce costs - reduce R burden - improve processing system - modernize instrument - expand/enhance use of admin records Key Design Change: - annual interview, 12-month reference pd., event history calendar methods
10
EHC Interviewing Human Memory - structured/organized - links and associations
11
EHC Interviewing Human Memory - structured/organized - links and associations EHC Exploits Memory Structure - links between to-be-recalled events
12
EHC Interviewing Human Memory - structured/organized - links and associations EHC Exploits Memory Structure - links between to-be-recalled events EHC Encourages Active Assistance to Rs - flexible approach to help elicit an autobiographical “story”
13
Evaluations of EHC Methods Many EHC vs. “Q-List” Comparisons - various methods - in general: positive data quality results
14
Evaluations of EHC Methods Many EHC vs. “Q-List” Comparisons - various methods - in general: positive data quality results BUT, Important Research Gaps - data quality for need-based programs? - extended reference period?
15
Paper Test Goals & Design Basic Goal: Can an annual EHC interview collect data of comparable quality to standard SIPP?
16
Paper Test Goals & Design Basic Goal: Can an annual EHC interview collect data of comparable quality to standard SIPP? “Go/No-Go” signal for continued R&D
17
Paper Test Goals & Design Basic Goal: Can an annual EHC interview collect data of comparable quality to standard SIPP? “Go/No-Go” signal for continued R&D Basic Design: EHC re-interview of SIPP sample households
18
Design Details (1) Sample: SIPP 2004 panel interview cases - reported on CY-2007 in waves 10-12
19
Design Details (1) Sample: SIPP 2004 panel interview cases - reported on CY-2007 in waves 10-12 EHC re-interview in 2008, about CY-2007
30
Design Details (2) SIPP Sample Cases in Two Sites - Illinois (all) - Texas (4 metro areas)
31
Design Details (2) SIPP Sample Cases in Two Sites - Illinois (all) - Texas (4 metro areas) Primary Sample Component: 1,096 Wave 10-11-12 Addresses (cooperating wave 11 households) IL:487 TX:609
32
Design Details (3) EHC Questionnaire [handout]
33
Design Details (3) EHC Questionnaire [handout] - paper-and-pencil - 12-month, CY-2007 reference period
34
Design Details (3) EHC Questionnaire [handout] - paper-and-pencil - 12-month, CY-2007 reference period - start with landmark events
35
Design Details (3) EHC Questionnaire [handout] - paper-and-pencil - 12-month, CY-2007 reference period - start with landmark events - subset of SIPP topics (“domains”) - month-level detail
36
Design Details (3) EHC Questionnaire [handout] - paper-and-pencil - 12-month, CY-2007 reference period - start with landmark events - subset of SIPP topics (“domains”) - month-level detail Sample of Addresses, Not People - post-interview clerical match to SIPP
37
Design Details (4) 1096 initial sample addresses
38
Design Details (4) 1096 initial sample addresses Outcomes: - 935 household interviews (91%)
39
Design Details (4) 1096 initial sample addresses Outcomes: - 935 household interviews (91%) - 1,922 individual EHC interviews (99%)
40
Design Details (4) 1096 initial sample addresses Outcomes: - 935 household interviews (91%) - 1,922 individual EHC interviews (99%) - 1,658 EHC Rs matched to SIPP (86%)
41
Design Details (4) 1096 initial sample addresses Outcomes: - 935 household interviews (91%) - 1,922 individual EHC interviews (99%) - 1,658 EHC Rs matched to SIPP (86%) FINAL ANALYSIS SAMPLE: 1,620
42
Primary Evaluation Compare SIPP and EHC Survey Reports
43
Primary Evaluation Compare SIPP and EHC Survey Reports - same people - same time period - same characteristics
44
Primary Evaluation Compare SIPP and EHC Survey Reports - same people - same time period - same characteristics Differences Suggest Data Quality Effects
45
Primary Evaluation Compare SIPP and EHC Survey Reports - same people - same time period - same characteristics Differences Suggest Data Quality Effects (later: use administrative records for a more definitive data quality assessment)
46
Main Research Questions 1.Are responses to Qs about government programs and other characteristics affected by interview method (SIPP vs. EHC)?
47
Main Research Questions 1.Are responses to Qs about government programs and other characteristics affected by interview method (SIPP vs. EHC)? 2.Does the effect of interview method vary across calendar months (especially early in the year vs. late in the year)?
48
Main Research Questions 1.Are responses to Qs about government programs and other characteristics affected by interview method (SIPP vs. EHC)? 2.Does the effect of interview method vary across calendar months (especially early in the year vs. late in the year)? 3.Bottom line: Go? Or no-go?
49
Initial Results 4 Government “Welfare” Programs: Food Stamps Supplemental Security Income (SSI) Temporary Aid to Needy Families (TANF) Women Infants & Children (WIC)
50
Initial Results 4 Government “Welfare” Programs: Food Stamps Supplemental Security Income (SSI) Temporary Aid to Needy Families (TANF) Women Infants & Children (WIC) 4 Other Characteristics: Medicare Social Security employment school enrollment
51
Results in Context
52
Almost All SIPP and EHC Reports Agree
53
Results in Context Almost All SIPP and EHC Reports Agree - all characteristics, all months
54
Results in Context Almost All SIPP and EHC Reports Agree - all characteristics, all months - in general: 97-98% likelihood that a respondent’s SIPP and EHC reports will agree
55
Results in Context Almost All SIPP and EHC Reports Agree - all characteristics, all months - in general: 97-98% likelihood that a respondent’s SIPP and EHC reports will agree - worst case (employment): 92-94%
56
Results in Context Almost All SIPP and EHC Reports Agree - all characteristics, all months - in general: 97-98% likelihood that a respondent’s SIPP and EHC reports will agree - worst case (employment): 92-94% Disagreements are RARE EVENTS
57
Results Summary The details vary, but…
58
Results Summary 3 Patterns:
59
Results Summary 3 Patterns: 1. EHC = SIPP All Year
60
Results Summary 3 Patterns: 1. EHC = SIPP All Year SSI; WIC (IL)
61
Results Summary 3 Patterns: 1. EHC = SIPP All Year equivalent data quality
62
SSI -- % Participation for Each Month of CY2007 According to the SIPP and EHC Reports
63
% participation (% “yes”)
64
SSI -- % Participation for Each Month of CY2007 According to the SIPP and EHC Reports % participation (% “yes”) months of CY 2007
65
SSI -- % Participation for Each Month of CY2007 According to the SIPP and EHC Reports % participation (% “yes”) months of CY 2007 SIPP reports EHC reports
66
SSI -- % Participation for Each Month of CY2007 According to the SIPP and EHC Reports % participation (% “yes”) months of CY 2007 SIPP reports EHC reports 10 percentage pts
67
SSI -- % Participation for Each Month of CY2007 According to the SIPP and EHC Reports 100 percentage pts
68
SSI -- % Participation for Each Month of CY2007 According to the SIPP and EHC Reports
69
Analysis Summary - no “main effect” for method (SIPP = EHC) - no significant method difference in any month
70
WIC (Illinois Only) -- % Participation for Each Month of CY2007 According to the SIPP and EHC Reports
71
Analysis Summary - no “main effect” for method (SIPP = EHC) - no significant method difference in any month
72
Results Summary 3 Patterns: 1. EHC = SIPP All Year SSI; WIC (IL)
73
Results Summary 3 Patterns: 1. EHC = SIPP All Year SSI; WIC (IL) 2. EHC < SIPP All Year
74
Results Summary 3 Patterns: 1. EHC = SIPP All Year SSI; WIC (IL) 2. EHC < SIPP All Year Medicare; Social Security; WIC (TX); Food Stamps (IL)
75
Results Summary 3 Patterns: 1. EHC = SIPP All Year SSI; WIC (IL) 2. EHC < SIPP All Year reduced EHC data quality, but not due to longer recall period
76
MEDICARE -- % Covered in Each Month of CY2007 According to the SIPP and EHC Reports
77
Analysis Summary - significant “main effect” for method (SIPP > EHC) - method difference is constant across months
78
SOCIAL SECURITY -- % Covered in Each Month of CY2007 According to the SIPP and EHC Reports
79
Analysis Summary - significant “main effect” for method (SIPP > EHC) - method difference is constant across months
80
WIC (Texas Only) -- % Participation for Each Month of CY2007 According to the SIPP and EHC Reports
81
Analysis Summary - significant “main effect” for method (SIPP > EHC) - method difference is constant across months
82
FOOD STAMPS (Illinois Only) -- % Participation for Each Month of CY2007 According to the SIPP and EHC Reports
83
Analysis Summary - significant “main effect” for method (SIPP > EHC) - method difference is essentially constant across months
84
Results Summary 3 Patterns: 1. EHC = SIPP All Year SSI; WIC (IL) 2. EHC < SIPP All Year Medicare; Social Security; WIC (TX); Food Stamps (IL)
85
Results Summary 3 Patterns: 1. EHC = SIPP All Year SSI; WIC (IL) 2. EHC < SIPP All Year Medicare; Social Security; WIC (TX); Food Stamps (IL) 3. EHC < SIPP, Early in the Year Only
86
Results Summary 3 Patterns: 1. EHC = SIPP All Year SSI; WIC (IL) 2. EHC < SIPP All Year Medicare; Social Security; WIC (TX); Food Stamps (IL) 3. EHC < SIPP, Early in the Year Only Food Stamps (TX); TANF (TX); employment; school enrollment
87
Results Summary 3 Patterns: 1. EHC = SIPP All Year SSI; WIC (IL) 2. EHC < SIPP All Year Medicare; Social Security; WIC (TX); Food Stamps (IL) 3. EHC < SIPP, Early in the Year Only EHC data quality may suffer due –to longer recall period
88
FOOD STAMPS (Texas Only) -- % Participation for Each Month of CY2007 According to the SIPP and EHC Reports
89
Analysis Summary - no significant “main effect” for method - BUT significant variation by month -- JAN-MAY: SIPP > EHC later months: no difference (reversal?)
90
TANF (Texas Only) -- % Participation for Each Month of CY2007 According to the SIPP and EHC Reports
91
Analysis Summary - no significant “main effect” for method - BUT significant variation by month -- JAN-MAY: SIPP > EHC later months: no difference
92
EMPLOYMENT -- % Working for Each Month of CY2007 According to the SIPP and EHC Reports
93
Analysis Summary - significant “main effect” for method (SIPP > EHC) - BUT significant variation by month -- JAN-AUG (SEP): SIPP > EHC later months: no difference
94
SCHOOL ENROLLMENT -- % Enrolled in Each Month of CY2007 According to the SIPP and EHC Reports
95
Analysis Summary - no significant “main effect” for method - BUT significant variation by month JAN-APR: SIPP > EHC JUN-JUL: SIPP < EHC AUG-DEC: no difference
96
Results Summary 3 Patterns: 1. EHC = SIPP All Year SSI; WIC (IL) 2. EHC < SIPP All Year Medicare; Social Security; WIC (TX); Food Stamps (IL) 3. EHC < SIPP, Early in the Year Only Food Stamps (TX); TANF (TX); employment; school enrollment
97
Field Test Overall Summary
98
Go! Successful “Proof of Concept”
99
Field Test Overall Summary Go! Successful “Proof of Concept” Overwhelming Finding: SIPP-EHC Agreement
100
Field Test Overall Summary Go! Successful “Proof of Concept” Overwhelming Finding: SIPP-EHC Agreement Valuable Lessons to Inform 2010 Test
101
Field Test Overall Summary Go! Successful “Proof of Concept” Overwhelming Finding: SIPP-EHC Agreement Valuable Lessons to Inform 2010 Test Specific Data Comparisons are Instructive
102
Results Implications Pattern 1. EHC = SIPP All Year SSI; WIC (IL)
103
Results Implications Pattern 1. EHC = SIPP All Year SSI; WIC (IL) No evident problems; no reason for concern about data quality in a 12-month EHC interview
104
Results Implications Pattern 2. EHC < SIPP All Year Medicare; Social Security; WIC (TX); Food Stamps (IL)
105
Results Implications Pattern 2. EHC < SIPP All Year Medicare; Social Security; WIC (TX); Food Stamps (IL) Problems with data quality in the Paper Test’s EHC treatment,
106
Results Implications Pattern 2. EHC < SIPP All Year Medicare; Social Security; WIC (TX); Food Stamps (IL) Problems with data quality in the Paper Test’s EHC treatment, but probably not due to EHC method or recall length
107
Results Implications Pattern 2. EHC < SIPP All Year Medicare; Social Security; WIC (TX); Food Stamps (IL) Problems with data quality in the Paper Test’s EHC treatment, but probably not due to EHC method or recall length - less effective screening questions (no D.I.; fewer probes; no local labels)
108
Results Implications Pattern 2. EHC < SIPP All Year Medicare; Social Security; WIC (TX); Food Stamps (IL) Problems with data quality in the Paper Test’s EHC treatment, but probably not due to EHC method or recall length - less effective screening questions (no D.I.; fewer probes; no local labels) - different definitions
109
Results Implications Pattern 2. EHC < SIPP All Year Medicare; Social Security; WIC (TX); Food Stamps (IL) Problems with data quality in the Paper Test’s EHC treatment, but probably not due to EHC method or recall length - less effective screening questions (no D.I.; fewer probes; no local labels) - different definitions Likely fixes in CAPI
110
Results Implications Pattern 3. EHC < SIPP, Early in the Year Only Food Stamps (TX); TANF (TX); employment; school enrollment
111
Results Implications Pattern 3. EHC < SIPP, Early in the Year Only Food Stamps (TX); TANF (TX); employment; school enrollment Most cause for concern; longer recall period may cause reduced data quality in the earlier months of the year
112
Results Implications Pattern 3. EHC < SIPP, Early in the Year Only Food Stamps (TX); TANF (TX); employment; school enrollment Most cause for concern; longer recall period may cause reduced data quality in the earlier months of the year Additional research: - why these characteristics?
113
Results Implications Pattern 3. EHC < SIPP, Early in the Year Only Food Stamps (TX); TANF (TX); employment; school enrollment Most cause for concern; longer recall period may cause reduced data quality in the earlier months of the year Additional research: - why these characteristics? - match to admin records
114
SSI -- % Participation for Each Month of CY2007 According to the SIPP and EHC Reports
115
SSI – Net Bias in the Monthly Estimates Compared to Administrative Records (Survey % “Yes” - Record % “Yes”)
116
SSI – Monthly Rates of Discrepancy With Administrative Records (Total Discrepancies as % of Non-Missing N)
117
FOOD STAMPS (Texas Only) -- % Participation for Each Month of CY2007 According to the SIPP and EHC Reports
118
FOOD STAMPS (Texas Only) -- % Participation for Each Month of CY2007 According to the SIPP and EHC Reports and ADRECS
119
Results Implications Pattern 3. EHC < SIPP, Early in the Year Only Food Stamps (TX); TANF (TX); employment; school enrollment Most cause for concern; longer recall period may cause reduced data quality in the earlier months of the year Additional research: - why these characteristics? - match to admin records (MORE TO DO)
120
Results Implications Pattern 3. EHC < SIPP, Early in the Year Only Food Stamps (TX); TANF (TX); employment; school enrollment Most cause for concern; longer recall period may cause reduced data quality in the earlier months of the year Additional research: - why these characteristics? - match to admin records - understand Paper Test time lag effects
125
. ????
126
Thanks very much! Any questions? contact:jeffrey.c.moore@census.gov 301-763-4975
Similar presentations
© 2025 SlidePlayer.com. Inc.
All rights reserved.