PPA 502 – Program Evaluation Lecture 5a – Survey research

Introduction  Government administrators and elected officials love to claim that they possess a profound understanding of their public’s needs, desires, and disaffection.

Introduction  Unfortunately, the administrators and officials are learning that storms of controversy provide meager evidence of the workaday values of the everyday people they govern.  Surveys of the public, conducted following the basic precepts of survey design and analysis, are fast becoming the vehicle for genuine connection to the public will.

Introduction  Uses. –Evaluations of government services. –Changing demographics that may signal shifts in service demand. –Patterns of service utilization. –Problem identification. –Customer service.

Introduction  Surveys have several important qualities. –Anonymity to respondents. –Point of view, characteristics, or use patterns can be characterized with little confusion. –Good surveys provide input from a representative cross-section.

Begin Before the Beginning  The best surveys grow from well-conceived and well-articulated reasons for conducting them.  Resist the temptation to hit the ground running.  Be certain of the purposes of the survey.

Begin Before the Beginning  Identify the appropriate audiences.  Identify the political and personal will for doing the survey.  Determine whether the questionnaire to be developed is better as a one-time or periodic survey.  Think about the usefulness of comparative data.

Getting Started  Convene a steering committee with key stakeholders.  Enlist the help of top government officials or administrators.

Designing the Survey  Sampling. –Choose the appropriate sampling frame: about what population do you wish to generalize? –A sampling plan must give every respondent in the sampling universe an equal chance of ending up in the sample. Simple random sample. Stratified sampling. Stratified random cluster sampling.

Designing the Survey  Targeting the individual in the household. –If no list exists, you may only have addresses or phone numbers. If so, use household member with most recent birthday.

Mail, Phone, or In-person Interviews  The best ways to conduct surveys vary by accuracy, speed, and cost. –Most common are mail and phone surveys.

Mail, Phone, or In-person Interviews CriteriaMailPhoneIn-Person Accuracy Response rate45-55%65-75%75%+ Permits anonymityHighModerateLow Is free from interviewer biasHighModerateLow Handles various question types Long/ complexLowModerateHigh Visual aidsHighLowHigh Ensures question orderLowHigh Permits widest coverage Targets geographic areasHighLowHigh Avoids education biasLowModerateHigh Gives easy access to target populationHigh Moderate Speed of administrationMonthWeekMonth Cost per interview$8-12$15-20>$20

Mail, Phone, or In-person Interviews  Increasing response rates. –Multiple mailings (up to three) with stamped, return address envelope. –Press coverage. –Combination of methods often best: Mail survey with telephone and in-person followup.

Mail, Phone, or In-person Interviews  Selecting sample size. –The size of sample depends on desired precision of estimates. –Generally speaking, if opinions are split as much as possible, than 100 residents will have a margin of error of +/- 10% with 95 percent confidence. Four hundred residents the margin is +/- 5%. –In general, 100 is a good minimum number, especially for subgroups.

Questionnaire Construction  Each question should be judged against the purposes of the survey and the uses to which it will be put.  Steal widely. –National Citizen Survey from International City Management Association and National Research Center.

Questionnaire Construction  Major principles. –Consistency. –Clarity. Vague wording. Double-barreled questions. Assumed knowledge. Overlapping response categories. –Simplicity. Specificity. Brevity (30 min. Phone, 60 min. In-person, 10 page mail). Context sensitivity.

Questionnaire Construction  Major principles. –Security. Demographic at end. General to specific. –Fairness. Option symmetry (balanced responses). Option wording and order. –Background info, pros and cons, opinion. –Randomize pros and cons in a complicated survey.

Conducting the Survey  The survey steering committee. –Double check questionnaire with steering committee.  Frequency of surveys. –For most multipurpose surveys, no more than once per year.  Pretest. –Test on twenty people at random. Ask questions about format and clarity.

Conducting the Survey  Training. –Survey assistants must be trained. All must operate uniformly, asking the questions in the same way, coding in the same way. –Consistent open-ended coding. –10% recontact of survey respondents.  Trying hard and keeping track. –Three contacts by telephone for each number. –Warning and at least two mailings for mail surveys.

Reporting Results  Data analysis. –For most government surveys, percentages, average responses, simple cross-classifications. –The most complicated analysis will be to get accurate population estimates – weighting.  Report writing and presentation. –Executive summary. –Bulleted lists. –Document survey methods in appendix. –Augment tables with bar and pie charts. –Powerpoint for in person presentation.

Hiring a Consultant  Previous experience.  Ability to communicate findings.  Share work with in-house staff.  Intuition.