Peer-Reviewer Guidance Data Seal of Approval Hervé L’Hours DSA Conference Amsterdam. 24 September, 2014

Overview The Seal is awarded through the independent adjudication of a Peer Reviewer appointed by the DSA Board. Assembly Members commit to peer-review a maximum of three repositories per year per DSA repository. Currently primarily by Board members Peer-reviewer expansion acknowledged as critical for scaling Peer-reviewer training acknowledged as critical for community confidence

“Peer-Review”? Never a good sign if Wikipedia has a disambiguation page for the term…. “the terminology has poor standardization and specificity, particularly as a database search term.” Trust-based review at a distance, based on evidence statements and supporting (ideally publically available) documentation But not a full ‘Audit’ in ISO17021 terms

ISO17021 ISO Standards are guided by ISO17021 “Requirements for Bodies Providing Audit and Certification of Management Systems” “The overall aim of certification is to give confidence to all parties that a management system fulfils specified requirements. The value of certification is the degree of public confidence and trust that is established by an impartial and competent assessment by a third-party.”

ISO17021 Where we differ Community-based collaboration, not legal entity No site visits Language Further changes to the DSA approach will be guided by advice from within the community and from wider collaboration such as that with WDS through RDA.

Who Peer-Reviews You? “A Peer Reviewer must have completed at least one self- assessment, which resulted in the award of the DSA without outsourcing compliance to any guidelines, and be a member of staff of a General Assembly member“ “The Board will attempt to match peer reviewers with the appropriate subject matter of the repository, but this is not a requirement” “The Board will attempt to match peer reviewers with the relevant language of the repository, but this is not a requirement.”

Guidance for Peer Reviewers Does the self-assessment statement correspond to the Guideline in question? (Also refer to the supporting text for applicants below each Guideline.) Are the links to supporting documentation available publically? Do you agree with the self-assessed compliance level and is the overall evidence sufficient to award the DSA? Have abbreviations been explained? In responding to the self-assessment provide helpful comments rather than specific questions where possible.

Process “A peer reviewer appointed by the DSA Board will either confirm the self-assessment evidence and compliance levels for each Guideline or reject them and provide relevant comments to be addressed by the applicant. The comment/response process will continue until the reviewer can decide whether to award the DSA. In the event of a dispute the applicant may appeal to the Board.” Impartial Confidential until the DSA is awarded.

E. Appeals 1. An organisation not awarded the Seal after peer review [] may appeal to the Board in writing. 2. The appeal must clearly state the rationale behind the appeal. 3. The Board will review the appeal rationale and take a decision on a course of action with an 85% majority vote of the Board required. The action may include, but not be limited to: (a) affirming the earlier decision or (b) restarting the application process, with a new peer reviewer assigned.

Evidence Transparency Link to publicly available documentation Or deadline for public release English or short summary in English Reviewers Guide: “Topics for discussion and inclusion are suggested but they are neither exhaustive nor prescriptive”

Challenges to Come How do we know what is Appropriate? Sufficient? Impartiality Consistency Competence Openness and responsiveness to criticism

Challenges: Outsourcing “To take account of the increasingly distributed and service- based nature of modern repositories, the DSA Board expanded the possibility of outsourcing to all Guidelines. This decision will be monitored over time and may be amended in future in cooperation with the DSA community. Applicant information relevant to outsourcing is requested in the ‘Repository Context’ section and must form part of the evidence for each applicable Guideline.”

Solutions Public Forum for evidence and discussion Sector-specific guidance Work on standardisation of evidence base from preservation plans to licencing and access criteria

Peer-Reviewer Guidance Data Seal of Approval Hervé L’Hours DSA Conference Amsterdam. 24 September, 2014