1. An initiative of the International Rare Diseases Research Consortium-IRDiRC Spanish Rare Diseases Registries Research Network - SpainRDR https://spainrdr.isciii.es Josep Jiménez Servei Català de la Salut Madrid, March 2013 Quality assessment and WP4. Quality assessment and ethical and legal issues

2. WP4. Aims SpainRDR - Spanish Rare Diseases Registries Research Network – https://spainrdr.isciii.es An initiative of the International Rare Diseases Research Consortium-IRDiRC It will also develop a framework to ensure data protection and compliance with ethical issues, as well as patient participation in all network activities. This WP seeks to optimise the quality of population health registries for rare diseases in Spain, by ensuring that participatory regions and centres follow pre-established procedures aimed at increasing homogeneity and minimising inaccuracy and incompleteness of data.

3. WP4. Tasks T17 Analysing the legal framework of health registries and translating this to RD registries T18 Establishing the approriateness of and degree of agreement between ethical principles and pthe issues raised by the development of the National RD Registry, under the overall supervision of the IIER Ethics ommittee, taking its ethical guidelines for RD registry reporting as reference SpainRDR - Spanish Rare Diseases Registries Research Network – https://spainrdr.isciii.es An initiative of the International Rare Diseases Research Consortium-IRDiRC T15 Developing SOPs for data collection and validation T16 Defining a quality assessment framework to improve data accuracy and completeness. A set of indicators to assess quality of registries will be drawn up in accordance with the EPIRARE quality indicator project

4. SpainRDR - Spanish Rare Diseases Registries Research Network – https://spainrdr.isciii.es An initiative of the International Rare Diseases Research Consortium-IRDiRC T15 Developing SOPs for data collection and validation

5. SpainRDR - Spanish Rare Diseases Registries Research Network – https://spainrdr.isciii.es An initiative of the International Rare Diseases Research Consortium-IRDiRC Developing SOPs for data collection and validation T15. Developing SOPs for data collection and validation Topics involved in a whole RD registry process Background and fundamental bases General and specific aims Geographical and long-term settings Target and study populations Design Case-definition Case inclusion and exclusion criteria Sources of cases List of covariables Data model and data dictionary Identification of disease codes and covariables codification Statistical analysis plan Operating manual (including SOPs and report forms) Training Pilot testing Data collection Data storing, access and safety Data delivery Quality control procedures Reporting Governance Patient involvement Long-term funding Ethical and legal issues External assessment Building consensus and synergies for the registration of rare diseases patients in Europe: the EPIRARE project. Leadership by Domenica Taruscio, CNMR, ISS

6. SpainRDR - Spanish Rare Diseases Registries Research Network – https://spainrdr.isciii.es An initiative of the International Rare Diseases Research Consortium-IRDiRC Developing SOPs for data collection and validation T15. Developing SOPs for data collection and validation The operating manual should contain a set of standardized procedures that should describe all type of activities regarding data collection, cleaning, storage, management, analyses and reporting. Difficulties and troubles have to be prevented, and alternate options should be included in each procedure. Each Autonomous Community (AC) should develop a set of SOPs adapted to its context, but common elements need to be defined to ensure homogeneity The SOPs should be reviewed and updated periodically The quality assurance process is based on the SOPs

7. SpainRDR - Spanish Rare Diseases Registries Research Network – https://spainrdr.isciii.es An initiative of the International Rare Diseases Research Consortium-IRDiRC Developing SOPs for data collection and validation T15. Developing SOPs for data collection and validation 1. Policy and procedure maintenanceSchedule for reviewing and updating, responsibilities, communication of changes,… 2. General documentationGeneral registry information, project report, key contacts, agreements (partners, patient registries,…), advisory boards, committees, legislation, … 3. Catalogues and classificationsRD list and codifications, data sources, common data elements, variable dictionary,… 4. Registry operating system and data management Hardware, software, data security,… 5. Data processing operationsReporting requirements, case ascertainment, case completeness, data entry, resolution of conflicting information, death clearance, consolidation of data, processing management reports,… 6. Ethic rulesConfidenciality, informed consent,… 7. Quality controlQuality assessment plan Operating manual

8. SpainRDR - Spanish Rare Diseases Registries Research Network – https://spainrdr.isciii.es An initiative of the International Rare Diseases Research Consortium-IRDiRC SpainRDR Network Flowchart AC RD Registry (RD Aplication)) External Reporting Sources (Patient Registries, Health Registries, etc) Integrated data environment Consolidated data environment AC RD Registry AC File SpainRDR File AC Procedures: A.Analysis of each data source B.Integration of sources and data management C.Consolidation of the AC database D.Generation of the exchange file G. Correction of errors and updating the AC registry CC Procedures: E. Analysis of each AC exchange file F. Generation of the response file Data analysis environment (source) I SpainRDR IIER A A B C D E F G Autonomous Community (AC) SpainRDR Coordination Center

9. SpainRDR - Spanish Rare Diseases Registries Research Network – https://spainrdr.isciii.es An initiative of the International Rare Diseases Research Consortium-IRDiRC Developing SOPs for data collection and validation T15. Developing SOPs for data collection and validation AC SOPs Data entry: Data structure User’s manual Procedures manual Data sources: Source analysis Data integration and management procedures Consolidation of the database Security document CC SOPs Importation of AC files Generation of AC errors and queries file External data sources Data process and management Security document Correction of errors and queries response AC registry updating Generation of the AC exchange file

10. SpainRDR - Spanish Rare Diseases Registries Research Network – https://spainrdr.isciii.es An initiative of the International Rare Diseases Research Consortium-IRDiRC Developing SOPs for data collection and validation T15. Developing SOPs for data collection and validation AC SOPs Data entry: Data structure User’s manual Procedures manual Data sources: Source analysis Data integration and management procedures Consolidation of the database Security document CC SOPs Importation of AC files Generation of AC errors and queries file External data sources Data process and management Security document Correction of errors and queries response AC registry updating Generation of the AC exchange file AC diversity: -Health care models -Information systems COMPLEXITY HETEROGENEITY RD list Multiplicity of sources: -Interoperability -Pacient identification -RD codification -Values conflicts -Access to the data source Common elements are needed to ensure homogeneity

11. SpainRDR - Spanish Rare Diseases Registries Research Network – https://spainrdr.isciii.es An initiative of the International Rare Diseases Research Consortium-IRDiRC Developing SOPs for data collection and validation T15. Developing SOPs for data collection and validation SOP structure Cover: SOP Ref No SOP title Category Version Date issued Valid until Author(s) Approved by Modification history Contents: Purpose Scope Responsibility Procedure References Appendices

12. SpainRDR - Spanish Rare Diseases Registries Research Network – https://spainrdr.isciii.es An initiative of the International Rare Diseases Research Consortium-IRDiRC T16 Defining a quality assessment framework to improve data accuracy and completeness

13. SpainRDR - Spanish Rare Diseases Registries Research Network – https://spainrdr.isciii.es An initiative of the International Rare Diseases Research Consortium-IRDiRC Defining a quality assessment framework T16. Defining a quality assessment framework Data quality Features and characteristics of a data set, that bear on its ability to satisfy the needs that result from the intended use of the data. Quality Assurance (QA) Activities undertaken before data collection to ensure that the data are of the highest possible quality at the time of collection Quality Control (QC) Activities undertaken during and after data collection aimed at identifiying and correcting sources of data errors Quality Assessment (Qass) Process of quality evaluation of the consolidated database Quality Results (QR) Value or set of values resulting from applying a data quality measure Quality Indicators (QInd) List of quality measurements

14. SpainRDR - Spanish Rare Diseases Registries Research Network – https://spainrdr.isciii.es An initiative of the International Rare Diseases Research Consortium-IRDiRC Defining a quality assessment framework T16. Defining a quality assessment framework Quality DesignQuality Process QA (Quality Assurance)QC (Quality Control)QAss (Quality Assessment)QR (Quality Results) Quality assurance of: 1.Data:  Completeness of cover  Completeness of detail  Accuracy of detail  Accuracy of reporting  Accuracy of interpretation 2.Registry procedures (Procedure Manual) 3.Computerized systems 4.Security and confidentiality 5.Pilot Phase Design Necessity of:  Case-definition and case- inclusion/case-exclusion criteria  Sources of cases (analysis of sources of information)  List of co-variables, data dictionary and codification  Rules and documentation (Procedural Manual + Pilot Phase)  Good coding systems (only one code is allocated for each appropriate term)  Standards (Ex: maximum % tolerable error in sex, dates,...) Describe rules and procedures to measure and evaluate data quality in the following areas: 1.Completeness 2.Comparability 3.Validity of accuracy 4.Timeliness 5.Data source evaluation 6.Security and confidentiality Quality control measures can be:  Continuous: formal on-going program which forms part of the registry standard procedures (computer checks for data quality)  Occasional: ad hoc survey to address specific questions of data quality Achieved high-quality data. Periodic quality assessment of quality control activities. Finding and solving problems: Problems in case selection and case ascertainment Errors in coding, data entry, data transformation, accuracy Data consistency across sites and over time International errors Data security and confidentiality Follow-up (ex.: mortality) Timeliness (ensure quality data) Reporting Coordination  Internal and external assessment of procedures and quality control strategies  Use of standardized data report form (DRRF) 1.Validity of results 2.Timeliness (Data delivery) 3.Reporting  Internal and external validity: o Internal: without bias (accuracy of measurements) o External: consistency and coherence of results  Use of standardized data reporting QInd (Quality Indicators) Registry Establishment IndicatorsData Collection Indicators –Data accuracy and data completeness-Data result Indicators Goal: 1.Provide assessment of quality and disparity 2.Provide baselines to track progress 3.Identify information gaps 4.Emphasize interdependence 5.Emphasize interdependence of quality and disparities 6.Promote awareness and change

15. SpainRDR - Spanish Rare Diseases Registries Research Network – https://spainrdr.isciii.es An initiative of the International Rare Diseases Research Consortium-IRDiRC Defining a quality assessment framework T16. Defining a quality assessment framework Quality assessment plan Based on (or included in) the operating manual Pre-requisites for quality assessment Quality assurance activities: case-definition and inclusion criteria, minimum set of data, data dictionary, data collection protocols, data checks, training and motivation, … Quality control activities: assessment of completeness, accuracy of detail, validation checks, consistency checks, errors correction,… Quality assessment activities: data audits, variability analysis (by site and over time), validity audits, completeness of cover, … Quality improvement methods: feedback of quality reports, feedback of results and recommendations, timeliness, reporting, coordination,… Occasional ad-hoc measures Quality indicators list

16. SpainRDR - Spanish Rare Diseases Registries Research Network – https://spainrdr.isciii.es An initiative of the International Rare Diseases Research Consortium-IRDiRC Defining a quality assessment framework T16. Defining a quality assessment framework Arts DGT, De Keizer NF, Scheffer GJ. Defining and Improving Data Quality in Medical Registries: A Literature Review, Case Study, and Generic Framework. J Am Med Inform Assoc. 2002;9:600–11

17. SpainRDR - Spanish Rare Diseases Registries Research Network – https://spainrdr.isciii.es An initiative of the International Rare Diseases Research Consortium-IRDiRC Defining a quality assessment framework T16. Defining a quality assessment framework Register evaluation: Assessment of the public health importance of the topic (RD) Description of the systems used and their objectives Description of the uses and other outputs of the register Description of the resources used to operate the system Evaluation of the system against each of the register attributes Summary of whether the system is meeting its objectives and any modifications required Examination at intervals of the value of the register to ensure that the objectives still hold and being met.

18. SpainRDR - Spanish Rare Diseases Registries Research Network – https://spainrdr.isciii.es An initiative of the International Rare Diseases Research Consortium-IRDiRC Defining a quality assessment framework T16. Defining a quality assessment framework Define Purpose Develop Study Plan Design Protocol (SOPs development) Conduct Study Identify Improvement Opportunities Evaluate Improvements Implement Improvements (SOPs improvement) Report Study Findings QA QC QAss QR QA QC QAss Quality Control Quality Improvement Analyze Study Procedures Analyze Study Data QInd

19. SpainRDR - Spanish Rare Diseases Registries Research Network – https://spainrdr.isciii.es An initiative of the International Rare Diseases Research Consortium-IRDiRC T15/T16 Pilot study evaluation Assessment of proceduresMarch 2013 Set of common elements for developing SOPs Review of procedures and development of new ones Discussion and approval April-Juny 2013 Proposal of a quality indicators list Design of quality improvement methods Discussion and approval April-Juny 2013 Workplan proposal

20. Thank you very much! SpainRDR - Spanish Rare Diseases Registries Research Network – https://spainrdr.isciii.es An initiative of the International Rare Diseases Research Consortium-IRDiRC