HOW TO PASS AS EVALUATING STATISTICAL REPORTS

TICK SHEET Achievement 2A1+3A2+3A3 Merit 1M1+3M2 Excellence 1E1+2E2 DESCRIBEEVALUATE JUSTIFY EVALUATION EVALUATE WITH STATISTICAL INSIGHT  Title Source Purpose A1 A3 M1 E1  Population measures A2A3M2E2  Variables A2A3M2E2  Sampling methods A2A3M2E2  Sample size A2A3M2E2  Survey methods A2A3M2E2  Sampling errors A2A3M2E2  Non- sampling error A2A3M2E2  Findings of the survey A2A3M2E2  Summary A1A3M1E1

DESCRIBE LIST AT LEAST 3 OF THE ACTUAL VARIABLES MEASURED CLASSIFY AS: CATEGORICAL ORDINAL QUALITATIVE NUMERICAL AND EVALUATE WRT PURPOSE VARIABLES

POPULATION MEASURES DESCRIBE MEAN MEDIAN MODE QUARTILES STANDARD DEVIATION EVALUATE ADVANTAGES DISADVANTAGES SUGGESTIONS OF WHICH IS THE BEST TO HAVE USED FOR THIS DATA.

SAMPLING METHODS DESCRIBE PROBABILITY SAMPLES Systematic Cluster Stratified Simple Random NON-PROBABILITY SAMPLES Person in the Street Self-Selected Quota sampling EVALUATE ADVANTAGES DISASVANTAGES SUITABILITY WRT PURPOSE ADVANTAGES DISASVANTAGES SUITABILITY WRT PURPOSE

SAMPLE SIZE DESCRIBEEVALUATE WRT ORIGINAL POPULATION SIZE ALLOWABLE SAMPLING ERROR PURPOSE OF REPORT

SURVEY METHODS DESCRIBE FACE TO FACE INTERVIEW TELEPHONE SURVEY COMPUTER-ASSISTED SURVEY SELF-ADMINISTERED SURVEY CLUSTER GROUPS PAPER QUESTIONNAIRE WEB-BASED QUESTIONNAIRE EVALUATE ADVANTAGES AND DISADVANTAGES WRT PURPOSE OF REPORT

SAMPLING ERRORS DESCRIBE SAMPLING ERROR IS AFFECTED BY: SAMPLE SIZE SAMPLE DESIGN VARIABILITY WITHIN THE POPULATION EVALUATE INCREASING SAMPLE SIZE TENDS TO REDUCE SAMPLING ERROR RECOMMEND A DIFFERENT SAMPLING METHOD RECOMMEND HOW TO CHOOSE A MORE REPRESENTATIVE SAMPLE

NON-SAMPLING ERRORS DESCRIBE COVERAGE ERROR RESPONSE ERRORS NON-RESPONSE ERRORS PROCESSING ERRORS ANALYSIS ERRORS. COVERAGE ERROR RESPONSE ERRORS NON-RESPONSE ERRORS PROCESSING ERRORS ANALYSIS ERRORS. EVALUATE EXPLAIN HOW THEY OCCURRED DISCUSS WAYS TO REDUCE SAMPLING ERRORS

NON-SAMPLING ERRORS DESCRIBE The wording of questions, the order in which they are asked and the number and type of options offered can influence survey results. Answers given by respondents do not always reflect their true beliefs because they may feel under social pressure not to give an unpopular or socially undesirable answer. Answers given by respondents may be influenced by the desire to impress an interviewer. If people who refuse to answer are different, with respect to survey issues, from those who respond.This can also happen with people who are never contacted and people who have yet to make up their mind. The wording of questions, the order in which they are asked and the number and type of options offered can influence survey results. Answers given by respondents do not always reflect their true beliefs because they may feel under social pressure not to give an unpopular or socially undesirable answer. Answers given by respondents may be influenced by the desire to impress an interviewer. If people who refuse to answer are different, with respect to survey issues, from those who respond.This can also happen with people who are never contacted and people who have yet to make up their mind. EVALUATE If the excluded or under-represented group is different, with respect to survey issues, then bias will occur. This creates bias because individuals with strong opinions about the survey issues or those with substantial knowledge will tend to be over- represented. Bias can occur because respondents may tend consistently to have views that are more extreme than those of the population in general. Bias will occur.

FINDINGS OF THE SURVEY DESCRIBE EVALUATE

SUMMARY

REMEMBER! EffectivenessMust relate this to the purpose Justification Must include supporting evidence Must always make clear links to the context, including reference to the background information

HOW TO GET A/ M / E Source Summary PurposeDiscusses results with respect to the purpose Features  Population measure and variables  Sampling methods  Survey methods  Sampling and possible non-sampling errors  Sample size EffectivenessMust relate this to the purpose JustificationMust include supporting evidence Must always make clear links to the context, including reference to the background information Achievement Achievement with MeritAchievement with Excellence F Identifies & comments on at least 3 different features Same as Achieved but adds statistical evidence from the report and discusses the processes used Same as Merit but: examines the quality of the survey process (loaded questions, questions changing from year to year, framing of questions to get a positive response, adequate sample size) examines contextual information (small footnotes with important information eg small response for a question, further clarification necessary, discussion on some data omitted) When discussing the purpose and results there are further questions such as: are there supporting studies?, is there funding tied to this report which may promote bias?, is there other data available? Discussion of how the report could be improved such as: change data representations and displays, not change questions between years, continue the survey over a longer time.