2. A pilot study examining criteria used to select drugs for hospital, provincial and national formularies J Robertson, D Newby, T Pillay, E. Walkom

3. About the study  Questionnaire survey  Members of Australian PBAC/ESC  Members of 6 provincial PTCs in South Africa  Participants in pharmacoeconomics short course  Objectives of the study  Establish importance of 22 criteria for selecting drugs for inclusion in national, provincial or hospital formularies  Compare ratings of criteria between 3 groups surveyed

4. Results  Clinical factors  Efficacy and safety were most important factors  Availability of other treatment options important  Quality of life less important than efficacy, safety  Cost factors  Cost-effectiveness important criterion in SA  Cost offsets ranked more highly in SA  Pharmacological factors varied by setting  Other factors  External pressure sometimes an influence in all settings

5. Our intentions  Survey instrument  Decision-making criteria, information sources, understanding of clinical & economic terms  Australia: national, hospital P&T committee  Feedback responses on importance of criteria  Explore complexity using hypothetical scenarios  South Africa: provincial PTCs  ? Discuss scenarios with committee members  Others: opportunistic  Short course participants, other contacts  ? Survey only

6. Variable response rates  Pilot in Australian hospital area PTC (postal)  2/17 responses, no interest in phase II  SA PTCs, (completed as part of meeting)  100% response rate but limited time to complete  SA PTC (started in meeting, to return by post)  0% response, Relevance of participating?  Threatening questions? Lack of interest?  Modified for Australian PBAC/ESC (postal)  9 PBAC, 2 ESC members responded  Short course (completed as part of workshop)  100% response rate

7. Scientifically rational description of decision-making  Survey did not focus on specific examples  Importance of criteria across all decisions  Could not identify when criteria may be more or less important  Could not capture complexity of decisions  ? Expected responses, consistent with NDP  Modified survey for PBAC/ESC  How important is criterion for decision-making?  How important should it be for decision-making?

8. Uncertainty in interpretation of terms  Survey in English; pilot tested Australia, SA  Aware of ‘difficult’ knowledge questions  Assessing capacity of committee, not individuals  Modified approach for more recent surveys  No chance to clarify questions, responses  Meaning of cost-effectiveness (Doubilet 1986)  Cost saving  Effective  Cost savings with equal or better outcomes  Having additional benefits worth additional cost

9. Failed to capture complexity of decision- making and engage decision-makers  Opinions and views of whole committee may not be sum of individual views  Not account for group dynamics  Time pressure to complete complex survey may compromise quality of responses  Work not of sufficient interest or perceived importance to encourage responses  Threatening nature of knowledge questions may have discouraged completion of survey

10. Our lessons  Survey too complex; tried to do too much  Knowledge questions were threatening  Not appropriate method for capturing complexities of decision-making process  Need examples to anchor questions  Trade-off quantity vs quality of information  Surveys - larger numbers, variety of settings, low cost  Jenkings & Barber SocSciMed 2004;58(9):1757-1766 observed meetings, analysed taped discussions, assessed local context