COLLECTING DATA: SURVEYS AND ADMINISTRATIVE DATA PBAF 526 Rachel Garshick Kleit, PhD Class 8, Nov 21, 2011

Today Report back on qualitative instruments Purpose Sample Questions Surveys In-person, Paper, telephone, and web Computer assisted Instrument Design Administrative data High Point HOPE VI Evaluation Longitudinal Study Hands-on surveys

Small group work (reprise) Present your instruments Purpose Sample Questions Choose one for group to edit and share Present to large group (today)

Constructing qualitative quantitative instruments Identify research question you are addressing Design sample and timing of collection Borrow or craft questions and protocols Previous surveys for organization Previous evaluations Academic literature Get input on instrument from diverse set of stakeholders Pre-test instrument on diverse group of subjects; perhaps usability study

Survey Basics Surveys are hard to do well Mode will depend on how you can best get to people In-person, paper, telephone, computer assisted, web How can you best access sample? How are they most likely to respond? Computer assisted helps with complex questions or skip patterns or sensitive questions (in-person) Resources required for modes are dramatically different Know what information the respondents will have (and don’t ask about info they don’t have)! Collection, analysis, and communication are critical processes and can be collaborative

Instrument construction Start with topic outline—tie to research questions Question order and grouping matters (interest, sensitivity, answers) Skip patterns can shorten survey and keep relevant Question wording must be clear : Time reference period (last 12 months vs last calendar year) Specific definitions (e.g., family income vs. individual earnings) Ask only about one issue per question Response options are critical: Match options to how you need to use data Label categories clearly for respondents (not data coders!)

Using Administrative Data Know what you want to measure Know what is in the administrative data Who is included? Over what time period? How is information collected? How have the collection or definitions changed over time? How to find out Is there a codebook or data input manual? Can you talk to someone who inputs or analyzes the data? [Watch them work!] Do you know how the data are currently used (gives clues)? Understand and work with the limitations of the data!

High Point HOPE VI Evaluation Longitudinal study to describe what happened to household heads due to redevelopment Pre- and post- redevelopment in-person surveys 2002 and 2008 Attitudes, opinions, family and individual characteristics and outcomes Administrative records Receipt and amounts of TANF, food stamps, employment income, family composition, receipt of housing assistance Tracking records Combine administrative data with on-going contacts with respondents Secondary data Residential building permits, mortgage originations, housing values, crime, businesses opened, U.S. Census GIS matching and mapping Topics Neighborhood relationships, quality, and efficacy, social ties, move decisions, employment, dependence on public benefits, receipt of housing assistance, relocation outcomes

High Point Evaluation In-person survey Initial outline Instrument (T1) Changes in neighborhood quality and social ties, perspective on neighborhood, economic security, health, education Administrative data Check selection bias in survey response Compare sample to population Patterns in relocation and examination of bias in who got to return (process evaluation) Patterns in of income receipt and housing assistance Tracking data Combine with administrative data to examine patterns of relocation GIS Mapping Secondary data Neighborhood effects Comparisons with surrounding east and west neighborhoods, and the city (sometimes) Merged survey, tracking, and administrative data to predict who would move away and control for T1 characteristics in subsequent analyses.

Survey instruments What is the purpose? What is your sample? How will you implement the survey? What is your question design?

Key Points to Keep in Mind Being sure what research question you are asking Using indicators for which questions to develop Design of first question-not too complicated, and specific A hook; how to hook them without intimidating them. (not your interests, the respondents)—salience Defining: how define neighborhood? Tell them? Ask them/self defined. Don’t ask questions you don’t need or make people uncomfortable Race, socioeconomic status (are these important for evaluation) Can prime questions; ask at end. Judging questions Wording questions; Are you familiar with this program vs. to what degree are you familiar. Length. Pop-up: 8 questions—too long, some open-ended. To increase responses, shorten. Ask essential information. Double-sided? Or booklet (what is more approachable?) Web—browser up to date? Pop-up Require as part of program participation? If for program mgmt