Survey Methodology Lilian Ma November 6, 2014

Three aspects 1. How questions were designed 2. How data was collected 3. How samples were drawn Probability sampling generally used.

Basic Characteristics/features 1. Their purpose 2. The population they try to describe 3. The source from which they draw samples 4. The design of the way they sample people 5. The use of interviewers 6. The mode of data collection 7. The use of computers in collection of answers

Thoughts to decide on features adopted Think of survey as information source Compare various design features to see how different survey design features permit the surveys to achieve their different purpose

Sampling Error Errors in statistics because of the omission of some persons in the population

(1) Sample survey: CCAT AJC Project on unrepresented Parties Sponsor CCAT AJC Collector CCAT AJC Purpose Main purpose: to self evaluate Access to Justice measure adopted for the benefit of self represented parties appearing before tribunals and agencies. To allow tribunals to check against checklist developed by attendees at CCAT conference in To help agencies and tribunal to get information needed for strategic planning Year started 2014 Target population Tribunal and agencies chairs of all jurisdictions in Canada Sample design: 64 questions grouped under 2 demographics and 14 areas Sample size about 500 subjects Mode of administration Survey monkey, paper survey for testing and promotion Time dimension Fall 2014 Pilot project Tested June 2014 Frequency annual repetition Levels of observation tribunal/agency

(2)Sample survey: SJTO Excellence Project Sponsor SJTO Collector SJTO Purpose Main purpose: to self evaluate quality and performance of SJTO and to collect information needed for strategic planning Year started 2014 Target population SJTO internal subjects and selected external subjects Sample design COAT Excellence Framework combined with client survey, stakeholder survey and staff survey Sample size about 500 subjects ( client survey about 500+ ) Mode of administration computer survey monkey, paper survey Time dimension Fall 2014 Pilot project March 2014 Frequency annual repetition Levels of observation person

What is Survey Methodology SM SM seeks to identify principles about the design, collection, processing, and analysis of surveys that are linked to the cost and quality of survey estimates Quality is defined within a framework labeled the total survey error paradigm. It is both a scientific field and a profession. As a science, it requires multidisciplinary application of –mathematics, statistics on sampling and inferences from sample results to population results. –Psychology re: memory, interview techniques –Computer science for database design, file processing

Some important decisions How will the potential sample members be identified and selected? What approach to contact those sampled. And how effort devoted to try to collect data from those who are hard to reach or reluctant to respond? How much effort devoted to evaluating and testing questions that are asked? What mode to pose questions and collect answers? How much effort devoted to checking the data file for accuracy and internal consistency? What approaches used to adjust the survey estimates to correct fro errors that can be identified? The above decisions will affect the quality of estimates emerging from the survey.

How surveys work to produce statistical descriptions of population: Inference and Errors in surveys, two types of inference If -> mean inference And = means statistical computing, Then Respondent answer to questions -> characteristics of a respondent = characteristics of the sample -> characteristics of the population

Two inferential steps 1. Answers people give must accurately describe characteristics of the respondents 2. The subset of persons participating in the survey must have characteristics similar to those of a larger population. When either of these conditions are not met the survey statistics are subject to “error” meaning deviation of what is attained to what is desired outcome.

Errors “Measurement error” or “error of observation” is the deviation of answer given to a question from the underlying attribute being measured “Errors of nonobservation” is the deviation of a statistic obtained from a sample from that of the full population.

SM is the study that makes survey more or less informative Examine each error Study from a quality perspective Study all the survey design decisions