1. FAIR Metrics RDA 10
Luiz Bonino – - September 21, 2017

2. Fair data principles Findable: Accessible: Interoperable: Reusable:
Findable:
F1. (meta)data are assigned a globally unique and persistent identifier;
F2. data are described with rich metadata;
F3. metadata clearly and explicitly include the identifier of the data it describes;
F4. (meta)data are registered or indexed in a searchable resource;
Accessible:
A1. (meta)data are retrievable by their identifier using a standardized communications protocol;
A1.1 the protocol is open, free, and universally implementable;
A1.2. the protocol allows for an authentication and authorization procedure, where necessary;
A2. metadata are accessible, even when the data are no longer available;
Interoperable:
I1. (meta)data use a formal, accessible, shared, and broadly applicable language for knowledge representation.
I2. (meta)data use vocabularies that follow FAIR principles;
I3. (meta)data include qualified references to other (meta)data;
Reusable:
R1. meta(data) are richly described with a plurality of accurate and relevant attributes;
R1.1. (meta)data are released with a clear and accessible data usage license;
R1.2. (meta)data are associated with detailed provenance;
R1.3. (meta)data meet domain-relevant community standards;

3. Rapid Adoption of Principles
Developed and endorsed by researchers, publishers, funding agencies, industry partners. As of Sept 2017, 120+ citations since 2016 publication Included in G20 communique, EOSC, H2020, NIH, and more…
G20:
EOSC:
H2020:

4. How do we assess compliance to the FAIR principles?
Principles identify what needs to be there, but they don’t tell what is necessary and/or sufficient
They also don’t tell you how to achieve FAIR
Going beyond the principles requires some thought about what constitutes FAIRness and how do we measure it.
Michel Dumontier -

5. What is fairness?
FAIRness reflects the extent to which a digital resource addresses the FAIR principles as per the expectations defined by a community of stakeholders.
Michel Dumontier -

6. A metric is a standard of measurement.
What is a metric?
A metric is a standard of measurement.
It must provide clear definition of what is being measured, why one wants to measure it.
It must describe the process by which you obtain a valid measurement result, so that it can be reproduced by others. It needs to specify what a valid result is.
Michel Dumontier -

7. How fair things are?
FAIRness of repositories: IDCC17 Practice Paper “Are the FAIR Data Principles fair?” by Alastair Dunning, Madelein de Smael, Jasmin Böhmer
DANS FAIR metrics -
NIH Commons Framework Working Group on FAIR Metrics
EC’s EOSC FAIR Metrics Group …

8. Fair metrics group - fairmetrics.org
Michel Dumontier
Univ. Maastricht
Susanna-Assunta Sansone
Univ. Oxford
Peter Doorn
DANS
Mark Wilkinson
U.P Madrid
Erik Schultes
DTL
Luiz Bonino
DTL/LUMC

9. motivation
We believe that increasing the FAIRness of digital resources will maximize their discovery and reuse. An assessment can provide the feedback needed to characterize and improve the FAIRness of a digital resource. To evaluate the FAIRness of a digital resource we need metrics.

10. objective
We seek to develop a set of core metrics that can be utilized in a computational infrastructure to automatically assess the FAIRness of any digital resource.

11. principles for FAIR metrics
Clear: understand what is meant
Realistic: possible to report on what is being asked of them
Discriminating: can distinguish the FAIRness of the resource
Measurable: assessment can be objective, quantitative, machine-interpretable, scalable and reproducible
Universality: applicable to all digital objects

12. milestones Establish charter (done)
Establish guidelines for the development of the FAIR metrics (done)
Establish a metric proposal form (done)
Develop a set of core FAIR metrics (ongoing)
Prepare and discuss a Release Candidate for proposed FAIR metrics and a prototype implementation (Sept 2017)
Community-wide implementation and review (Oct-Nov, 2017)
Revision of FAIR metrics and documentation (Jan-Feb 2018)
Prepare and discuss a Final Recommendation and Reference Implementation the core set of FAIR metrics (March 2018)

13. process For each FAIR principle: Propose one or more possible metrics
Fill out the information in the metric proposal form
Discuss the merits of the proposal
does it conform to the FAIR metric guidelines? (clear, realistic, discriminating, measureable, universal)
is it atomic or does it comprise a number of different or complementary aspects?
Iteratively refine and test the metric proposal until consensus is achieved
~3.5hrs per metric * 15 principles =  62.5h * 6 people = 315 person hrs

14. Jul/17: framework document Jul/17: FAIR Metrics Proposal Form
deliverables
Jul/17: framework document
Jul/17: FAIR Metrics Proposal Form
Sept-Oct/17: draft recommendation and webinar
Sept-Oct/17: Computational framework for automated evaluation of FAIR metrics
Mar/18: final recommendation
Mar/18: reference implementation of the computational framework for automated evaluation of FAIR metrics

15. FAIR Metrics form Metric identifier Metric name
To which principles does it apply?
What is being measured?
Why should I measure it?
What must be provided?
How to measure it?
What is a valid result?
For which digital resource(s) is this relevant?
Examples of its application across types of digital resource
Comment
Michel Dumontier -

16. Example of a FAIR Metric
F1 (meta)data are assigned a globally unique and persistent identifier Aspect: Identifier Persistence Rationale: An identifier can be used to find, access, and reuse a resource. As such, it must be available to users in the longest term possible otherwise we will not be able to perform those functions with the identifier in hand. Relevant FAIR Principles: F,A,I,R Metric: Availability of data management plan, which includes a section dealing with continuity and contingencies related to the persistence of identifiers. The value of the metric is true or false. Procedure: Check and verify the URL in the resource metadata points to a data management plan with continuity section. Document should follow a community standard, or recommend a basic structure.
Michel Dumontier -

17. Example of a FAIR Metric

18. Example of a FAIR Metric

19. Q&A/contact info
Luiz Bonino
Skype: luizolavobonino