MIE2009 Good Evaluation Practice Workshop Pirkko Nykanen 1 Guidelines for Good Evaluation Practices in Health Informatics - A shared networked initiative EFMI and IMIA Coordinators Pirkko Nykänen, Tampere University Jytte Brender, Aalborg University

MIE2009 Good Evaluation Practice Workshop Pirkko Nykanen 2 Motivation Need to have good practice guidance for evaluation No single, global methodology exists Good practice guidelines should serve as: – Framework to design evaluation studies, to select methodologies, to conduct studies – Support for health professionals and users to understand evaluation and contribute to evaluation studies

MIE2009 Good Evaluation Practice Workshop Pirkko Nykanen 3 Objectives Generic practical guidelines that provide evaluators, users, health professionals with structured, scientifically justified, comprehensive and understandable set of rules for good practice –To design and carry out evaluation studies in health informatics domain

MIE2009 Good Evaluation Practice Workshop Pirkko Nykanen 4 Objectives – practically Guidelines: – Criteria and aspects, both in quantitative and qualitative terms, to consider at each evaluation stage – Carry out evaluation of the specific aspect, criteria at specific stage – Design and manage the evaluation study Guidelines from all stakeholders viewpoints: third party evaluators, users, health professionals, managers, decision makers, health economists,...

MIE2009 Good Evaluation Practice Workshop Pirkko Nykanen 5 How GEP-HI guidelines have been developed Existing guidelines development principles Existing literature and material on evaluation studies, methodologies, reported evaluation experiences, guidelines for good clinical practice, codes of ethics, good implementation practices critical study and assessment, review Spiral, consensus making approach: –Coordinators - editorial team - external experts by HISEVAL website / mailing list –Revised versions – comments – input – revisions… –Presentations / workshops in MIE, MEDINFO- confereneces

MIE2009 Good Evaluation Practice Workshop Pirkko Nykanen 6 Exploration ReportBuyer Team: iterative discussion and consensus Selection of Methods Internal Report First Study Design ReportBuyer Study Design Including: Goal and Mandate ReportBuyer Participants Internal Reports Interim Reports Buyer (?) Participants Final Study Report Buyer Participants Publication 1... n Study Implementation The Study Project Controlling & Risk Management Receiver C Weßel GEP-HI Suggestion for a Flowchart First start GEP-HI guidelines

MIE2009 Good Evaluation Practice Workshop Pirkko Nykanen 7 First study design Study exploration Operational. of methods Detailed Planning Evaluation Implement. Funder / client Proj. Managem. Proj. Managem. Proj. Managem. Proj. Managem. Eval. consultant Eval. consultant Eval. consultant Eval. consultant Object organisation Evaluation team Evaluation team Evaluation team Evaluation team Information need AgreementDesign outline MethodPlans & procedures Finalisation of the evaluation study. GEP_HI guidelines flowchart Project control and risk managem ent

MIE2009 Good Evaluation Practice Workshop Pirkko Nykanen 8 GEP-HI guidelines Study exploration – Starting question First study design –Preliminary design Operationalisation of methods – methodological approach, methodical aspects Detailed study plan – detailed planning Evaluation study implementation – Actual accomplishment of the planned evaluation study Project control and risk management – Good project management practices Finalisation of the evaluation study – Accounting, archiving, reporting of evaluation study results

MIE2009 Good Evaluation Practice Workshop Pirkko Nykanen 9 Status now and progress Guidelines-paper has been accessible on iig.umit.at/efmi-website since 2006 Comments and input received Final revision is ongoing –Final version will be available by end 2009 Comments and input received will be incorporated and the guidelines finalised for publication

MIE2009 Good Evaluation Practice Workshop Pirkko Nykanen 10 Contact Feedback and input paper accessible: iig.umit.at/efmi Editorial team: Elske Ammenwerth, Nicolette de Keizer, Jan Talmon, Michael Rigby, Jytte Brender, Pirkko Nykänen

MIE2009 Good Evaluation Practice Workshop Pirkko Nykanen 11 A walk-through of the key guidelines issues Pirkko Nykänen, Tampere University Jytte Brender, Aalborg University

MIE2009 Good Evaluation Practice Workshop Pirkko Nykanen 12 First study design Study exploration Operational. of methods Detailed Planning Evaluation Implement. Funder / client Proj. Managem. Proj. Managem. Proj. Managem. Proj. Managem. Eval. consultant Eval. consultant Eval. consultant Eval. consultant Object organisation Evaluation team Evaluation team Evaluation team Evaluation team Information need AgreementDesign outline MethodPlans & procedures Finalisation of the evaluation study. GEP_HI guidelines flowchart Project control and risk managem ent

MIE2009 Good Evaluation Practice Workshop Pirkko Nykanen 13 Guidelines summary 1.Study exploration 2.First study design 3.Operationalisation of methods 4.Detailed study plan and project plan 5.Evaluation study implementation 5.1 Project control and risk management 6.Finalisation of the evaluation study

MIE2009 Good Evaluation Practice Workshop Pirkko Nykanen 14 Phase 1: Study exploration focuses on the starting question of the evaluation study 1.1The information need 1.2Primary audience 1.3Identification of the buyer / sponsor / study funding party 1.4The context of the evaluation study 1.5A first identification of stakeholders 1.6A first identification of (external) consultants 1.7A first sketch of the setting 1.8First exploration of evaluation methods to be used 1.9Exploring the restrictions of study execution and publication 1.10Budget 1.11Ethical, moral and legal issues 1.12Result of Study Exploration= Outline, draft design of the study 1.13Formal accept to proceed to the next phase

MIE2009 Good Evaluation Practice Workshop Pirkko Nykanen 15 Phase 2: First study design focuses on the preliminary design of the evaluation study 2.1Elaboration of the rationale for the study 2.2Key evaluation issues/questions 2.3Budget 2.4Establishment of the design team 2.5Stakeholder analysis/Social Network analysis 2.6Study constraints 2.7Methods 2.8Organisational setting, the study context 2.9Technical setting, the study context 2.10 Participants from the organisational setting 2.11 Material and immaterial resources 2.12 Time and timing 2.13 Risk analysis 2.14 Ethical, moral and legal issues 2.15 Strategy for reporting and dissemination of results 2.16 Result of First Study Design = a more detailed evaluation study plan

MIE2009 Good Evaluation Practice Workshop Pirkko Nykanen 16 Phase 3: Operationalisation of methods focuses on making the design and evaluation methods concrete and compliant with the organizational setting and the information need, while taking into account the known pitfalls and perils 3.1Study type 3.2Approach 3.3Assumptions 3.4Pitfalls and perils 3.5Skills 3.6Frame of reference 3.7Timing 3.8Justification 3.9Usability studies 3.10Outcome measures 3.11Quality Control on data (measures) 3.12Participants 3.13Study flow 3.14Result of Operationalisation of Methods = Decisions on the methodological and methodical approach 3.15Ethical, moral and legal issues = required approvals, ethical committee permissions achievable and prepared

MIE2009 Good Evaluation Practice Workshop Pirkko Nykanen 17 Phase 4: Detailed study plan and project plan focuses on providing plans, prescriptions and procedures detailed to the level necessary for the specific study 4.1Project management 4.2Evaluation activity mapping 4.3Quality management 4.4Risk management 4.5Communication strategy 4.6Recruitment of necessary additional staff 4.7Result of Detailed Study Plan and Project Plan = Practical plans with incorporated tools. Upon approval of plans the evaluation study can now be started

MIE2009 Good Evaluation Practice Workshop Pirkko Nykanen 18 focuses on activities related with the actual accomplishment of the designed evaluation study 5.1Establishment of the frame of reference 5.2Observation of changes 5.3Quality control of findings 5.4Interpretation of observations 5.5Continuous project management, quality management and risk management 5.6Regular reports 5.7Final result of Evaluation Study Implementation = The final evaluation report summarising the study plan and results, and presenting conclusions Phase 5: Evaluation study implementation

MIE2009 Good Evaluation Practice Workshop Pirkko Nykanen 19 Phase 5.5 Project controlling and risk management Focuses on the good project management practices and risk management specifically for an evaluation study -Dedicated project management tools -Quality management in relation to the objectives and milestones of the study > Develop a quality management plan -Risk management -Prospectively during planning -Retrospectively with the risks realised

MIE2009 Good Evaluation Practice Workshop Pirkko Nykanen 20 Phase 6: Finalisation of the evaluation study focuses on accounting, archiving of materials, and reporting of evaluation studies in terms of the STARE-HI guidelines 6.1 Accountings 6.2 Reports and publications 6.3 Archiving 6.4Reporting guidelines 6.5Reporting scope 6.6The reporting message 6.7Authorship 6.8.Ethical and moral aspects 6.9 Preparation of reports / publications - Follow STARE-HI guidelines / Statement on reporting of evaluation studies in health informatics, IJMI 78, 2009, Usability documentation in the context of the EU regulations and standards

MIE2009 Good Evaluation Practice Workshop Pirkko Nykanen 21 Generalisability of the guidelines Designed for all types of evaluation study types, also applicable on usability evaluation study planning The guidelines are for – to plan an evaluation study – to carry out an evaluation study The guidelines are NOT for –Which methods to use – How to use the methods Handbooks available, see e.g. Brender 2006