EVALUATING STATISTICAL REPORTS HOW TO PASS AS 91266 EVALUATING STATISTICAL REPORTS

EVALUATE WITH STATISTICAL INSIGHT TICK SHEET   Achievement 2A1+3A2+3A3 Merit 1M1+3M2 Excellence 1E1+2E2 DESCRIBE EVALUATE JUSTIFY EVALUATION EVALUATE WITH STATISTICAL INSIGHT Title Source Purpose A1 A3 M1 E1 Population measures A2 M2 E2 Variables Sampling methods Sample size Survey methods Sampling errors Non- sampling error Findings of the survey Summary

VARIABLES DESCRIBE LIST AT LEAST 3 OF THE ACTUAL VARIABLES MEASURED CLASSIFY AS: CATEGORICAL ORDINAL QUALITATIVE NUMERICAL AND EVALUATE WRT PURPOSE

POPULATION MEASURES DESCRIBE EVALUATE MEAN MEDIAN MODE QUARTILES STANDARD DEVIATION ADVANTAGES DISADVANTAGES SUGGESTIONS OF WHICH IS THE BEST TO HAVE USED FOR THIS DATA.

SAMPLING METHODS DESCRIBE EVALUATE ADVANTAGES DISASVANTAGES PROBABILITY SAMPLES Systematic Cluster Stratified Simple Random NON-PROBABILITY SAMPLES Person in the Street Self-Selected Quota sampling ADVANTAGES DISASVANTAGES SUITABILITY WRT PURPOSE

SAMPLE SIZE DESCRIBE EVALUATE WRT ORIGINAL POPULATION SIZE ALLOWABLE SAMPLING ERROR PURPOSE OF REPORT

SURVEY METHODS DESCRIBE EVALUATE FACE TO FACE INTERVIEW TELEPHONE SURVEY COMPUTER-ASSISTED SURVEY SELF-ADMINISTERED SURVEY CLUSTER GROUPS PAPER QUESTIONNAIRE WEB-BASED QUESTIONNAIRE ADVANTAGES AND DISADVANTAGES WRT PURPOSE OF REPORT

SAMPLING ERRORS DESCRIBE EVALUATE SAMPLING ERROR IS AFFECTED BY: SAMPLE SIZE SAMPLE DESIGN VARIABILITY WITHIN THE POPULATION INCREASING SAMPLE SIZE TENDS TO REDUCE SAMPLING ERROR RECOMMEND A DIFFERENT SAMPLING METHOD RECOMMEND HOW TO CHOOSE A MORE REPRESENTATIVE SAMPLE

NON-SAMPLING ERRORS DESCRIBE EVALUATE EXPLAIN HOW THEY OCCURRED COVERAGE ERROR RESPONSE ERRORS NON-RESPONSE ERRORS PROCESSING ERRORS ANALYSIS ERRORS. EXPLAIN HOW THEY OCCURRED DISCUSS WAYS TO REDUCE SAMPLING ERRORS

NON-SAMPLING ERRORS describe evaluate The wording of questions, the order in which they are asked and the number and type of options offered can influence survey results. Answers given by respondents do not always reflect their true beliefs because they may feel under social pressure not to give an unpopular or socially undesirable answer. Answers given by respondents may be influenced by the desire to impress an interviewer. If people who refuse to answer are different, with respect to survey issues, from those who respond .This can also happen with people who are never contacted and people who have yet to make up their mind. If the excluded or under-represented group is different, with respect to survey issues, then bias will occur. This creates bias because individuals with strong opinions about the survey issues or those with substantial knowledge will tend to be over- represented. Bias can occur because respondents may tend consistently to have views that are more extreme than those of the population in general. Bias will occur.

FINDINGS OF THE SURVEY DESCRIBE EVALUATE

SUMMARY

REMEMBER! Effectiveness Must relate this to the purpose Justification Must include supporting evidence Must always make clear links to the context, including reference to the background information

How to get a/ m / e Source Summary Purpose Achievement Achievement with Merit Achievement with Excellence F Identifies & comments on at least 3 different features Same as Achieved but adds statistical evidence from the report and discusses the processes used Same as Merit but: examines the quality of the survey process (loaded questions, questions changing from year to year, framing of questions to get a positive response, adequate sample size) examines contextual information (small footnotes with important information eg small response for a question, further clarification necessary, discussion on some data omitted) When discussing the purpose and results there are further questions such as: are there supporting studies?, is there funding tied to this report which may promote bias?, is there other data available? Discussion of how the report could be improved such as: change data representations and displays, not change questions between years, continue the survey over a longer time. Source Summary Purpose Discusses results with respect to the purpose Features Population measure and variables Sampling methods Survey methods Sampling and possible non-sampling errors Sample size Effectiveness Must relate this to the purpose Justification Must include supporting evidence Must always make clear links to the context, including reference to the background information