Quality and purpose: quality in quantitative methods Angela Dale University of Manchester

Quality in survey methods A survey is of adequate quality if it can answer specified question with an adequate degree of accuracy it is fit for purpose If it can provide an answer at a 1% level of confidence when only 5% is required, then it may not be cost effective

Standards for surveys Random sample – best bet of avoiding bias in sample; but need adequate sampling fame Adequate numbers for required subgroups – depends on level of accuracy needed Good response rate – but extent of bias is crucial Questionnaire must be well developed well designed questions; good flow; salient

Can a scoring system work? A scoring system can indicate overall strength on all these dimensions, but: Higher quality costs more Increased spend may have diminishing returns But there will be a level below which a survey would have very low value/low credibility Need to find a way of assessing quality against requirement

Analysis issues Results based on analysis need to reflect the quality of the survey Importance of making clear the level of accuracy of the results The quality of the analysis is also of great importance An excellent survey may be badly analysed , methods may be used poorly; interpretations incorrect inappropriate assumptions about causality may be made Claims may go beyond what the data can support

Can a poor survey have value? A very poor survey may still have value: if analysed with care and weaknesses recognised If only very limited conclusions, that can be justified, are drawn Where there is no better alternative

Conclusions It is of value to have a reference framework for assessing quality But equally important to use it critically and with care The key to quality lies not just in good data but in ensuring that claims made can be supported by the data