AAPOR 69 th Annual Conference Anaheim, CAMay 17, 2014 Casey Langer Tesfaye and Susan White Can a survey of U.S. High Schools be Replaced or Reduced through Web Searches? The Successes and Complications of an Experimental Strategy

AAPOR 69 th Annual Conference Anaheim, CAMay 17, 2014 Background Nationwide Survey of High School Physics Multiphase survey  Sample: public & private HS via NCES  Refreshed quadrennially, Redrawn periodically  Phase 1: schools, Phase 2: teachers 2

AAPOR 69 th Annual Conference Anaheim, CAMay 17, 2014 Phase 1: Schools Does your school offer Physics? Names & addresses of physics teachers 3

AAPOR 69 th Annual Conference Anaheim, CAMay 17, 2014 Background Multimode survey  Mode: paper, web, phone 4

AAPOR 69 th Annual Conference Anaheim, CAMay 17, 2014 Problem 1 Paper survey is really expensive 5

AAPOR 69 th Annual Conference Anaheim, CAMay 17,

AAPOR 69 th Annual Conference Anaheim, CAMay 17, 2014 Problem 2 Schools are oversurveyed One IRB is not enough  More and more schools have custom research applications or processes 7

AAPOR 69 th Annual Conference Anaheim, CAMay 17, 2014 Problem 3 Address lists out of date  Some schools close or move Some schools difficult to reach by phone 8

AAPOR 69 th Annual Conference Anaheim, CAMay 17, 2014 Goal Eliminate paper survey altogether Reduce contact with schools Filter out closed schools faster 9

AAPOR 69 th Annual Conference Anaheim, CAMay 17, 2014 Proposed solution Begin with web searches Call remaining schools 10

AAPOR 69 th Annual Conference Anaheim, CAMay 17,

AAPOR 69 th Annual Conference Anaheim, CAMay 17, 2014 Strategy 3 minute limit Zoned form 12

AAPOR 69 th Annual Conference Anaheim, CAMay 17,

AAPOR 69 th Annual Conference Anaheim, CAMay 17, 2014 Evaluating the effort Responses  Response rate  Data quality  Research process requests $ savings 14

AAPOR 69 th Annual Conference Anaheim, CAMay 17, 2014 Outcome Overall response rate lower, but still over 90% 15

AAPOR 69 th Annual Conference Anaheim, CAMay 17, 2014 Outcome 16 Mode# of Responses% of total Responses Web search106129% % Phone call166346% Total %

AAPOR 69 th Annual Conference Anaheim, CAMay 17, 2014 Data Quality Difficult to measure Many issues to consider 17

AAPOR 69 th Annual Conference Anaheim, CAMay 17, 2014 Training and Streamlining Googling is a skill  Google evolves Search strategies and techniques vary  Evolve over time There is a tradeoff between encouraging creative search strategies and evaluating the quality of the information yielded 18

AAPOR 69 th Annual Conference Anaheim, CAMay 17, 2014 Data Sources varied Sources varied  Search results & search pages (e.g. Physics Olympic teams, activities rosters)  School websites, class websites, department pages, grade-level pages, parental resources  Great schools websites  LinkedIn and other social networking sites 19

AAPOR 69 th Annual Conference Anaheim, CAMay 17, 2014 Quality Control Many schools share the same name  Double check school location Many cities share the same name  Use full/er address to verify Many teachers share the same name  Listings with last names only are common 20

AAPOR 69 th Annual Conference Anaheim, CAMay 17, 2014 Information varied Very different configurations of information for each school  Some easier to navigate, some had more or less information or differently targeted information (e.g. sports teams)  Different definitions (e.g. Upper School) 21

AAPOR 69 th Annual Conference Anaheim, CAMay 17, 2014 Information varied Search results vary  Date of updates varies  Site content varies  Websites go up & down, sometimes cached, sometimes redesigned 22

AAPOR 69 th Annual Conference Anaheim, CAMay 17, 2014 What makes a complete entry? Goal of form was staged  Stage 1: 1. Is physics information available? 2. Is physics teacher information available? 23

AAPOR 69 th Annual Conference Anaheim, CAMay 17, 2014 What makes a complete entry? Goal of form was staged  Stage 1: 1. Is physics information available? 2. Is physics teacher information available?  Stage 2: 3. Then is there any helpful contact information? - What constitutes helpful contact information?  New classification: helpful, but not complete 24

AAPOR 69 th Annual Conference Anaheim, CAMay 17, 2014 Data Quality Every mode of survey has data quality issues  Web searches yielded more inaccurate information that in the past  Inaccurate information leaks into teacher survey  E.g. “I don’t teach physics” “I teach physics, but I have no students this year”  More of a need for a streamlined system for corrections 25

AAPOR 69 th Annual Conference Anaheim, CAMay 17, 2014 Cost Web search, & phone call cheaper than Paper survey, & phone call Web search & phone calls roughly equal (but web had the added benefit of filtering out closed schools faster and reducing contact with schools) 26

AAPOR 69 th Annual Conference Anaheim, CAMay 17, 2014 In Sum… Web searches save $ Web searches lead to less research process requests and potential rejections But… Web searches cannot be the only mode A system that uses web searches:  Must have streamlined procedures for updating incorrect information 27

AAPOR 69 th Annual Conference Anaheim, CAMay 17, 2014 The web continually evolves More information online every day Google continually improving and streamlining search results 28

AAPOR 69 th Annual Conference Anaheim, CAMay 17, 2014 The web continually evolves 29

AAPOR 69 th Annual Conference Anaheim, CAMay 17, 2014 The web continually evolves Training cannot be static  Develop search and quality standards that can evolve over time  Develop a system that encourages web search staff to discuss their strategies  Use these interactions to understand what information you are collecting and where you are collecting it from 30

AAPOR 69 th Annual Conference Anaheim, CAMay 17, 2014 In the future We will continue the web searches  But we will strengthen and streamline the feedback loop 31

AAPOR 69 th Annual Conference Anaheim, CAMay 17, 2014 Thanks! Questions, comments? Casey Langer Tesfaye 32