1. Providing Context for Understanding The Data Life Cycle and the National Population Health Survey E. Hamilton IASSIST 2005

2. With thanks to... Chuck Humphrey Gary Catlin Mario Bedard France Bilocq Vicki Crompton Michael Sivyer The NPHS and CCHS project teams — and special thanks to those valiant survey respondents!

3. Research Project Research Objective: to discover evidence of use of the National Population Health Survey Methodology: Using framework of data life cycle, searched established peer-reviewed databases as well as grey literature sources for evidence of data use and knowledge transfer relating to the NPHS

4. What Lies Inside the Black Box? GenesisMaturity Revise Recycle Retire Re-purpose Respect ? The 144 GB Black Box Survey Data & Documentation

5. Data Gap Analysis User Consultation/ Content Determination Questionnaire Development Data Collection Data Processing Dissemination Data Quality Measurement Research & Development Evaluation & Revision User guide and data file release Inside the Black Box Political and social agendas

6. The Data Life Cycle A. Data Gap Analysis Documenting perceived needs and survey expectations B. Administration and Planning Documenting the process U C. NPHS Use U >60 theses; >400 articles in >150 journals D. Revision and Gap Analysis Documenting changes

7. NPHS Background A national longitudinal health survey launched in 1993/4 to fill a significant data gap in health information Intended to provide more comprehensive information on population health status — a key component of Canada ’ s health information infrastructure Dissemination of data and information were deemed a critical part of the survey plan at the outset

8. NPHS Administration Household survey: general and health components Complex survey design using strata and based on LFS frame; Respondents to be followed for 20 years Institutional survey Northern survey Cycle 1 Household survey in the field in June 1994

9. NPHS Data Products YearSurveyMasterShare*PUMFComments 1994/95Household (both files)Yes Suppl. survey data available in all flavours. Dummy file for remote access for Health file InstitutionalYes NorthernYes No 1996/97Household (both files)Yes Longitudinal file for cycle 1 & 2 Dummy files for remote access InstitutionalYes NorthernYes No 1998/99Household (both files)Yes Dummy files for remote access InstitutionalYes No NorthernYes No 2000/01Household (both files)Yes No Longitudinal only; end of PUMFs; dummy files for remote access InstitutionalYes No NorthernYes No

10. NPHS User Guide Background Objectives Survey Content Sample Design Data Collection Data Processing Data Quality Guidelines for Tabulation, Analysis & Release Approximate Sampling Variability Tables Weighting File Usage Plus: Questionnaire Record layout (General and Health) Data Dictionary Derived and Grouped variables CV Tables list

11. Life Cycle Approach Revealed More Interviewer Training manuals Self-study guides for interviewers Expert Group reports Questionnaire Focus group reports Treasury Board documents Comparisons with other surveys Jacknife vs bootstrap Coding concordances And so much more!

12. Training Guide Example During the last 12 months, did EXPERIENCE C happen to you? R.: Well, yes, there was one night, we were drinking and I just … DEP_Q26EE1A Q. At this point the respondent stops talking and you see a tear come to her eyes. What do you do?

13. Training Guide Example Exercise-Minimize Non-Response Tactics: Sad/Upset Respondent: Stop for a minute. Be responsive to the respondent in a supportive way; give the respondent a chance to collect themselves, and help them get on track with the interview. 1.“ That must have been very upsetting ”…..offer to take a break from the interview. 2. Ask if the respondent is able to continue with the interview. 3. If not, offer to continue the interview at a later date.

14. Researcher Needs Why did wording of question change over time? Why this particular question? Why not another wording? Why this content? Why not broader/ more specific? What scales were adopted or modified for use in this survey?

15. What training was given to interviewers? How were difficult issues resolved? What was considered — and what was rejected? What is the relationship between this survey and other surveys of similar subject matter? How has the sample changed (over surveys and over time)? What should I know to analyze this survey appropriately? Researcher Needs

16. Observations There is a rich array of documentation throughout the survey life cycle This documentation should be identified and bundled intellectually with the data Surveys with a repetitive component require special preservation efforts for this rich documentation

17. It is important! Senator Lavoie-Roux: “My questions have already been asked more or less by my colleagues because I was wondering how the data were collected… To me it was really an important issue to know to what extent your statistics could be trusted—I don’t say that in a negative way—since the methods used for data collection are fairly weak. … With the methods that you use, do you believe that we can trust the data? Are you sure that it is accurate? “

18. There is more to come … Thank you for your attention!