Brussels, 20 March 2013 Bart Vannieuwenhuyse 1 Topic 2 - Quality Metrics Bart Vannieuwenhuyse Senior Director Health Information Sciences Janssen R&D

Brussels, 20 March 2013 Bart Vannieuwenhuyse 2 Scope of the project –  Purpose – “improving”  “what gets measured, gets done”  What are “Quality Metrics”?  A “metric” is a measure.  “Quality” is something a “customer” defines.  A “Quality Metric”, therefore, is a measure of quality as defined by the customer. NOTE 1: A “customer” might be defined as anybody with an expectation of receiving something of value in exchange for something else of value, either external to or internal to an organization. NOTE 2: Not all “Metrics” are “Quality Metrics” Topic 2 – Quality Metrics

Brussels, 20 March 2013 Bart Vannieuwenhuyse 3 Contributing projects Topic 2 – Quality Metrics

Brussels, 20 March 2013 Bart Vannieuwenhuyse 4 Convergence challenges  Define scope – agree on areas with highest need  “Internal” vs “External” application of metrics  Potential opportunities to leverage (tbd)  Improving efficiency of collaboration in project  Process to improve project deliverables  Measuring quality of (external) data  Identifying quality of (sub)contractors Topic 2 – Quality Metrics

Brussels, 20 March 2013 Bart Vannieuwenhuyse 5 Topic 2 – report back Quality Metrics – domains: – Project quality Quality of deliverables – internal “peer review” generally adopted Project management – “on time – on budget” generally adopted – Project impact Uptake of solutions – need for further development of metrics (e.g. Service registry using text mining in BioMedBridges) Scientific impact – publications, possibility to further improve on speed and breadth of sharing results Societal / health care impact – need for further development of more standardized approaches – Data quality …

Brussels, 20 March 2013 Bart Vannieuwenhuyse 6 Data Quality “Data quality is the end product of a whole process” Type of Use (Care – Research) Context of creation Quality of Solution Quality of Usage Metrics 1Metrics 2 “All elements need to be of the right quality” A Rolls Royce with 3 wheels is a crappy car

Brussels, 20 March 2013 Bart Vannieuwenhuyse 7 Data quality - process Context of data creation – meta-data – Should be made explicit – Provenance must be clear “medical context” - clarity on reimbursement and “medical practice” Clarity on who created the data – Mapping to common ontologies Type of use drives selection of data – Data should be fit for intended use – Care vs Research – Options to select data sources on available meta data

Brussels, 20 March 2013 Bart Vannieuwenhuyse 8 Data quality - metrics Quality of solution – metrics 1 – Adopt existing standards e.g. ISO SDLC like approach (engineering) Functional suitability- Reliability Performance efficiency- Security Compatibility- Maintainability Usability- Portability – STEEEP – Safe Timely Efficient Effective Equitable Patient-centered (IOM – US) Quality of usage – metrics 2 Effectiveness- Freedom of risk Efficiency- Context coverage User satisfaction

Brussels, 20 March 2013 Bart Vannieuwenhuyse 9 Data quality - dimensions Accuracy Quantitative vs Qualitative data (origin of data) Benchmarking to check accuracy (TransForm, OMOP, EU-ADR) Completeness Needed granularity – data available? (TransForm selection tool) “Longitudinality” – length of available Hx Timeliness Data “freshness” – latest update Reliability Who created the data – who is responsible Trustworthiness – traceability (versioning, time-stamping) Structured - Unstructured

Brussels, 20 March 2013 Bart Vannieuwenhuyse 10 Next steps Data Quality Metrics community – Convene individuals from all EU projects dealing with re-use of existing data – Consolidate existing approaches across EU projects – share current solution – Classifications of data quality metrics – check availability of ISO standards for eHealth data – if not, consider developing one? (ISO 8000 general data quality) – Consolidate available quality standards of solutions (e.g. ISO 25000) – Recommendation for projects to focus on data quality even before projects starts – Develop common approaches to evaluating data quality – “benchmarking” analogy of computer chips // radar-graph – Have guidelines on Data quality – e.g. when creating new data / attention to meta-data (training) – Develop and share analytical methods that deal with “imperfect data” Data quality is a journey And even the longest journey starts with the first step