Lesli Scott Ashley Bowers Sue Ellen Hansen Robin Tepper Jacob Survey Research Center, University of Michigan Third International Conference on Establishment.

Slides:



Advertisements
Similar presentations
1 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation Response Rate in Surveys Key resource: Dillman, D.A.,
Advertisements

Census and Statistics Department Introduction to Sample Surveys.
Market Research Ms. Roberts 10/12. Definition: The process of obtaining the information needed to make sound marketing decisions.
Survey Methodology Nonresponse EPID 626 Lecture 6.
Self-Administered Surveys: Mail Survey Methods ChihChien Chen Lauren Teffeau Week 10.
Brian A. Harris-Kojetin, Ph.D. Statistical and Science Policy
1 Sampling Telephone Numbers and Adults, and Interview Length, and Weighting in the California Health Survey Cell Phone Pilot Study J. Michael Brick, Westat.
Chapter 11: Collecting Data by Communication. Key Issues for Collecting Information by Communication.
ESS Mixed mode experiment ESRA Conference, Ljubljana, July 2013 Alun 18 July 2013.
B121 Chapter 7 Investigative Methods. Quantitative data & Qualitative data Quantitative data It describes measurable or countable features of whatever.
Kevin Deardorff Assistant Division Chief, Decennial Management Division U.S. Census Bureau 2014 SDC / CIC Conference April 2, Census Updates.
© 2003 Prentice-Hall, Inc.Chap 1-1 Business Statistics: A First Course (3 rd Edition) Chapter 1 Introduction and Data Collection.
Quality improvement for asthma care: The asthma care return-on-investment calculator Ginger Smith Carls, M.A., Thomson Healthcare (Medstat) State Healthcare.
© 2004 Prentice-Hall, Inc.Chap 1-1 Basic Business Statistics (9 th Edition) Chapter 1 Introduction and Data Collection.
Chapter 13 Survey Designs
© 2002 Prentice-Hall, Inc.Chap 1-1 Statistics for Managers using Microsoft Excel 3 rd Edition Chapter 1 Introduction and Data Collection.
© 2005 The McGraw-Hill Companies, Inc., All Rights Reserved. Chapter 8 Using Survey Research.
Survey Research & Understanding Statistics
Basic Business Statistics (8th Edition)
Chapter 13 Survey Designs
Chapter 9 Descriptive Research. Overview of Descriptive Research Focused towards the present –Gathering information and describing the current situation.
Introduction of Internet Survey Methodology to the Emirate of Abu Dhabi Andrew Ward, Maitha Al Junaibi and Dragica Sarich.
1 ©IRWIN a Times Mirror Higher Education Group, Inc., company 1997 Collecting and Using Marketing Information.
Survey Designs EDUC 640- Dr. William M. Bauer
EMR 6500: Survey Research Dr. Chris L. S. Coryn Kristin A. Hobson Spring 2013.
Cover Letters for Survey Research Studies
1 Types and Sources of Data UAPP 702 Research Methods for Urban & Public Policy Based on notes by Steven W. Peuquet, Ph.D.
Understanding Web Completion in a Survey of New Businesses David DesRoches Tom Barton Janice Ballou Third International Conference on Establishment Surveys.
CAHPS Overview Clinician & Group Surveys: Practical Options for Implementation and Use AHRQ ANNUAL MEETING SEPTEMBER 18, 2011 Christine Crofton, PhD CAHPS.
Power Point Slides by Ronald J. Shope in collaboration with John W. Creswell Chapter 13 Survey Designs.
Survey Methods By Shivakumaraswamy, K N
Power Point Slides by Ronald J. Shope in collaboration with John W. Creswell Chapter 13 Survey Designs.
Survey Research and Other Ways of Asking Questions
RTI International is a trade name of Research Triangle Institute 3040 Cornwallis Road ■ P.O. Box ■ Research Triangle Park, North Carolina, USA
Electronic reporting in Poland 27th Voorburg Group Meeting Warsaw, Poland October 1st to October 5th, 2012 Central Statistical Office of Poland.
6. Implications for Analysis: Data Content. 1 Prerequisites Recommended modules to complete before viewing this module  1. Introduction to the NLTS2.
Making Sense of the Social World 4th Edition
Questionnaire Design Chapter Nine. Chapter Nine Objectives To understand the role of the questionnaire in the data collection process. To become familiar.
Chapter 12: Survey Designs
Slide 1 Incentives in Surveys with Farmers Third International Conference on Establishment Surveys Montreal, Quebec, Canada June 19, 2007 Slide Kathy Ott.
TATN Survey Results Statewide Survey Results for Quality Indicators for Assistive Technology Developed and provided by Texas Assistive Technology.
 Collecting Quantitative  Data  By: Zainab Aidroos.
Brown, Suter, and Churchill Basic Marketing Research (8 th Edition) © 2014 CENGAGE Learning Basic Marketing Research Customer Insights and Managerial Action.
European Conference on Quality in Official Statistics Session 26: Quality Issues in Census « Rome, 10 July 2008 « Quality Assurance and Control Programme.
Setting Your Survey in Motion & Getting It Done Prepared by: Ali Al-Baggal.
Basic Business Statistics
CHAPTER 12 Descriptive, Program Evaluation, and Advanced Methods.
Evaluation of the Noyce Teacher Scholarship Program 2010 NSF Noyce Conference Abt Associates Inc. July 9, 2010.
Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved.
Chapter Eleven The entire group of people about whom information is needed; also called the universe or population of interest. The process of obtaining.
Marketing Research Approaches. Research Approaches Observational Research Ethnographic Research Survey Research Experimental Research.
Household Surveys: American Community Survey & American Housing Survey Warren A. Brown February 8, 2007.
1 Public Library Use in Oregon Results from the 2006 Oregon Population Survey Oregon State Library March 2007.
Questionnaires Questions can be closed or open Closed questions are easier to analyze, and may be done by computer Can be administered to large populations.
Basic Business Statistics, 8e © 2002 Prentice-Hall, Inc. Chap 1-1 Inferential Statistics for Forecasting Dr. Ghada Abo-zaid Inferential Statistics for.
1 of 29Visit UMT online at Prentice Hall 2003 Chapter 1, STAT125Basic Business Statistics STATISTICS FOR MANAGERS University of Management.
Chapter Eight Questionnaire Design Chapter Eight.
Copyright © 2008 by Pearson Education, Inc. Upper Saddle River, New Jersey All rights reserved. John W. Creswell Educational Research: Planning,
THE MARKETING RESEARCH PROCESS CHAPTER 29 Mrs. Simone Seaton Marketing Management.
Lessons Learned: The Minnesota Young Adult Alcohol Survey American Evaluation Association Annual Conference October 26, 2012.
Chapter 29 Conducting Market Research. Objectives  Explain the steps in designing and conducting market research  Compare primary and secondary data.
Data Collecting Techniques Telephone interviews Traditional telephone interviews involve phoning a sample of respondents and asking them a series.
Planning a Customer Survey Part 1 of 3 Elaine Carlson, Westat Anne D’Agostino, Compass Evaluation & Research.
Using Surveys to Design and Evaluate Watershed Education and Outreach Day 5 Methodologies for Implementing Mailed Surveys Alternatives to Mailed Surveys.
Chapter 1 Introduction and Data Collection
How to Conduct Toileting Trials: A Webinar Course Evaluation
Evaluating Survey Data Collection Methods
Chapter Eight: Quantitative Methods
Business Statistics: A First Course (3rd Edition)
Presentation transcript:

Lesli Scott Ashley Bowers Sue Ellen Hansen Robin Tepper Jacob Survey Research Center, University of Michigan Third International Conference on Establishment Surveys Montreal, Quebec, Canada June 18-21, 2007 Evaluation and Implementation of EDR in School-Based Research

Introduction Survey Research Center conducts national school-based studies of elementary/secondary education and of children and adolescents. Typically, SRC school-based studies include a data collection activity to obtain information about each school from the principal. Focus of this presentation is on assessing the potential effectiveness of using Web surveys to obtain school information from principals in the U.S.

Introduction Currently, most researchers choose the traditional mail mode Use multiple contact – tailored approach (Dillman), scannable instruments, and incentive. Attain % response rates. One SRC research effort over past couple years used Web mode Principals in a large district completed daily activity logs during 3 one-week periods each of three years (70-90% response). The same principals also completed annual Web surveys for three years (70-90% response). These studies provided some evidence that Web surveys can attain acceptable response rates in regional school studies.

Key Research Questions This presentation reports on two very recent studies that help evaluate the ability to transition some national school studies to Web mode. Key research questions considered include: What’s the impact of Web mode on coverage error? What’s the impact of Web mode on sampling error? What’s the impact of Web mode on nonresponse error? What future research is needed?

Two Recent Studies (1) Study 1: Principals’ Use of Internet Study (WEB) In Spring 2007, we administered a brief Web survey. Main purpose was to assess properties of the sample through phases of the study. We developed a national sample of 500 schools randomly drawn from U.S. Department of Education database. We carried out Internet searches and schools calls to obtain principal names and addresses. We sent principals UPS delivered pre-notification letter with $5 incentive then followed with up to 3 contacts.

Study 2: Principals’ Mode Preference Study Our 2 nd study analyzed similar questions in the WEB study and a School Health Policy (MAIL) survey. Main purpose was to learn about principals’ access to Internet and preference for mail or Web mode. Surveys used nationally representative samples of 500 schools covering K-8 th grade (WEB) & 600 schools covering 8 th -12 th grade (MAIL). Two Recent Studies (2)

Web Mode & Coverage Error Coverage error considers whether the list used to create the study sample includes all members of the population and probability of selection is known. Hypothesis: Web mode provides nearly universal access to principals. Principals we can and can’t access are not different.

Coverage – All Principals Listed? U.S. Department of Education provides a census of schools. Recent reports indicate that virtually all schools have high speed Internet access. What about principals? Our studies helped us answer: Do principals use high speed Internet? Can we obtain addresses for principals? How similar are schools where we can deliver to principals and schools where we cannot?

Coverage – Internet Access National Center for Education Statistics report (2006) –100% of schools have Internet. –97% of schools have high speed connections. Answers to questions on our Web and mail surveys showed that principals use high speed connections. Hi-Speed Internet Access At School WEB Study 97% MAIL Study 98% Preliminary Results

Coverage – Available? U.S. Department of Education databases do not include principal name or addresses. Our Web study showed that we could use procedures to obtain names and s. Obtaining Addresses Obtained During 1st Internet Search 55% Obtained by Adding School Calls 90% Successful Delivery to Principal 84% Preliminary Results

Coverage–Similar to Sample? Our study showed no difference in key school characteristics for principals with and without deliverable s. Preliminary Results

Nearly all principals have access to high speed Internet. It is possible to obtain addresses for most principals using a procedure that involves Internet searches and telephone calls to schools. There are no statistically significant differences between schools where we can deliver to principals and those we cannot on key school measures – that is, there is no evidence of coverage bias. Coverage Error - Conclusions

Web Mode & Sampling Error Sampling error considers the precision of estimates that can be made based in part on the number of units included in the random sample. Hypothesis: Web gains more precision for same cost as mail mode. Our study helped us identify the cost drivers for the two modes and we estimated the magnitude of difference between them.

Sampling – Main Cost Factors Web and mail survey costs include: Sample Development Pre-Notification with Incentive Questionnaire Development Application Programming Survey Administration Post Collection Processing

Sampling – Costs (1) (Hypothetical 2-page Qstr) Web Mode (n=500)Mail Mode (n=500) Sample Development NCES Data Internet Search School Calls Preload ~$2,700 NCES Data Purchase Names School Calls Mail Merge ~$1,700 Pre-Notification with Incentive Materials Postage Incentive Labor ~$5,900 Materials Postage Incentive Labor ~$5,900 Paper Questionnaire and Materials Development Questions Provided Format Paper Qstrs Print Questionnaires Print Letters Postage ~$9,500

Sampling – Costs (2) (Hypothetical 2-page Qstr) Web Mode (n=500)Mail Mode (n=500) ProgrammingWeb ~$500DDE ~$500 Survey Administration 1 st 2 nd 3 rd ~$550 1 st Mail Logging 2 nd Mail Logging 3 rd Mail Logging ~$1,800 Data ProcessingData Handling ~$350 DDE Data Handling ~$600 TOTAL~$10,000~$20,000

The cost to conduct a Web survey of principals should almost always be less than the cost to conduct the survey using the traditional mail mode. Cost savings can be used to increase sample size – thus, reducing sampling error. In addition, the marginal cost of adding a case in a Web survey is generally smaller than the cost of adding a case in a mail survey. Sampling Error - Conclusions

Nonresponse error considers whether sample members complete the survey and whether the responders have different characteristics than nonresponders. Hypothesis: Participation in Web surveys is at least as good as mail. Principals who do participate and those who do not look similar on key school characteristics. Web Mode & Nonresponse Error

Nonresponse – How Willing? Our studies helped us answer the following: Are principals receiving requests and choosing to respond to Web surveys? Do principals have a preference for Web or mail surveys? Can we obtain response rates for a Web survey of principals that are comparable to mail? Do those principals who participate in a Web survey look similar to those who do not on key school characteristics?

Nonresponse – Web Survey Experience On average, MAIL Study principals received more than one Web survey request during past year. –Most completed at least one. WEB study principals have participated in Web surveys at even higher rates. WEB Study Principals (%) MAIL Study Principals (%) 0 requests requests requests requests823 Preliminary Results

Nonresponse – Preference Web vs. Mail Most MAIL study and WEB study principals indicated that they prefer Web surveys over mail surveys. Few differences in preference by school characteristics or available principal demographics (e.g., gender).

Nonresponse – WEB Response Rates The response rate achieved in the WEB study was comparable to typical mail survey response rates. The length of time needed to conduct a survey is greatly reduced using the Web compared to traditional mail data collection.

Nonresponse – Similar to Sample? Our study showed no difference in key school characteristics for principals who participated in the WEB Study and those who did not. Preliminary Results

Nonresponse Error - Conclusions Principals are receiving Web survey requests and responding to them. Many principals express a preference for Web surveys over mail surveys. Our experience conducting a Web survey of principals in a random sample of U.S. public schools provides evidence that Web surveys can achieve response rates comparable to mail surveys. Principals who participated in the Web study generally looked similar to those who did not on key school characteristics.

Future Research (1) Measurement Error Research Are Web surveys measuring differently than other modes? What aspects of Web design influence quality of response?

Complexity Found in Establishment Studies What are impacts of introducing long and complex instruments? Will principals input information collected from school files into a Web instrument? Will principals forward portions of Web instruments to other school staff? Future Research (2)

Future Research (3) Some Coverage Error Research Questions: What might we learn if we make contact with principals in schools where we could not obtain deliverable addresses? Are there some specific groups of principals that don’t want to be contacted and if so, what can we learn about them? Are services in schools provided by school, district, state or other entities? How much do principals use home computers for work related activities? How often would they choose to complete work-related surveys at their homes? In addition to principals, can we obtain addresses for multiple informants at the school who might assist with school surveys?

Future Research (4) Experimental Design Ideally, our next step will be to set up research using random assignment to Web and mail modes.

Final Conclusions Special Thanks to: Andrew Hupp Timothy Wright Jackie McBride