USING DATA MANAGEMENT PLANS as a RESEARCH TOOL for IMPROVING DATA SERVICES in ACADEMIC LIBRARIES Jake Carlson, Patricia Hswe & Susan Wells Parham Amanda Whitmire, Lizzy Rolando & Brian Westra IASSIST 2015 Minneapolis, MN 2-6 June 2015

Amanda Whitmire Jake Carlson Patricia M. Hswe Susan Wells Parham |  Lizzy Rolando |  Brian Westra D A R T Team 5 June DART Project

Acknowledgements Amanda Whitmire | Oregon State University Libraries Jake Carlson | University of Michigan Library Patricia M. Hswe | Pennsylvania State University Libraries Susan Wells Parham | Georgia Institute of Technology Library Lizzy Rolando | Georgia Institute of Technology Library Brian Westra | University of Oregon Libraries This project was made possible in part by the Institute of Museum and Library Services grant number LG D A R T Team 5 June 20153

4

5

6

7

transition slide 5 June 20158

Levels of data services the basics DMP reviewworkshopswebsite mid-level dedicated “research services” metadata support facilitate deposit in DRs consults high level infrastructuredata curation From: Reznik-Zellen, Rebecca C.; Adamick, Jessica; and McGinty, Stephen. (2012). "Tiers of Research Data Support Services." Journal of eScience Librarianship 1(1): Article June 20159

Informed data services development 5 June Survey

Informed data services development 5 June SurveyDCPs

Informed data services development 5 June SurveyDCPsDMPs DMP

5 June DART Premise DMP Research Data Management needs practices capabilities knowledge researcher

5 June

5 June DART Premise Research Data Management needs practices capabilities knowledge Research Data Services

5 June DART Premise

5 June We need a tool

5 June Solution: an analytic rubric Performance Levels Performance Criteria HighMediumLow Thing 1 Thing 2 Thing 3

NSF Directorate or Division BIOBiological SciencesENGEngineering DBIBiological InfrastructureCBET Chemical, Bioengineering, Environmental, & Transport Systems DEBEnvironmental BiologyCMMICivil, Mechanical & Manufacturing Innovation EFEmerging Frontiers OfficeECCSElectrical, Communications & Cyber Systems IOSIntegrative Organismal SystemsEECEngineering Education & Centers MCBMolecular & Cellular BiosciencesEFRIEmerging Frontiers in Research & Innovation CISEComputer & Information Science & Engineering IIPIndustrial Innovation & Partnerships ACIAdvanced Cyberinfrastructure CCFComputing & Communication Foundations GEOGeosciences CNSComputer & Network SystemsAGSAtmospheric & Geospace Sciences IISInformation & Intelligent SystemsEAREarth Sciences EHREducation & Human Resources OCEOcean Sciences DGEDivision of Graduate EducationPLRPolar Programs DRL Research on Learning in Formal & Informal Settings MPSMathematical & Physical Sciences DUEUndergraduate EducationASTAstronomical Sciences HRDHuman Resources DevelopmentCHEChemistry DMRMaterials Research DMSMathematical Sciences PHYPhysics 5 June division-specific guidance

5 June SourceGuidance text NSF guidelines The standards to be used for data and metadata format and content (where existing standards are absent or deemed inadequate, this should be documented along with any proposed solutions or remedies) BIO Describe the data that will be collected, and the data and metadata formats and standards used. CSE The DMP should cover the following, as appropriate for the project:...other types of information that would be maintained and shared regarding data, e.g. the means by which it was generated, detailed analytical and procedural information required to reproduce experimental results, and other metadata ENG Data formats and dissemination. The DMP should describe the specific data formats, media, and dissemination approaches that will be used to make data available to others, including any metadata GEO AGS Data Format: Describe the format in which the data or products are stored (e.g. hardcopy logs and/or instrument outputs, ASCII, XML files, HDF5, CDF, etc).

5 June Advisory Board Project team testing & revisions Feedback & iteration Rubric

5 June Performance Level Performance CriteriaComplete / detailed Addressed issue, but incomplete Did not address issue Directorates General Assessment Criteria Describes what types of data will be captured, created or collected Clearly defines data type(s). E.g. text, spreadsheets, images, 3D models, software, audio files, video files, reports, surveys, patient records, samples, final or intermediate numerical results from theoretical calculations, etc. Also defines data as: observational, experimental, simulation, model output or assimilation Some details about data types are included, but DMP is missing details or wouldn’t be well understood by someone outside of the project No details included, fails to adequately describe data types. All Directorate- or division- specific assessment criteria Describes how data will be collected, captured, or created (whether new observations, results from models, reuse of other data, etc.) Clearly defines how data will be captured or created, including methods, instruments, software, or infrastructure where relevant. Missing some details regarding how some of the data will be produced, makes assumptions about reviewer knowledge of methods or practices. Does not clearly address how data will be captured or created. GEO AGS, GEO EAR SGP, MPS AST Identifies how much data (volume) will be produced Amount of expected data (MB, GB, TB, etc.) is clearly specified. Amount of expected data (GB, TB, etc.) is vaguely specified. Amount of expected data (GB, TB, etc.) is NOT specified. GEO EAR SGP, GEO AGS

5 June Mini-reviews 1 & 2 23

5 June

Inter-rater reliability 5 June

5 June Wherein I try not to put you to sleep. Inter-rater reliability

A primer on scoring 5 June X = T + E Very helpful excerpts from: Hallgren, Kevin A. “Computing Inter-Rater Reliability for Observational Data: An Overview and Tutorial.” Tutorials in Quantitative Methods for Psychology 8, no. 1 (2012): 23–34.

A primer on scoring 5 June X = T + E Observed Score True Score Measurement Error

A primer on scoring 5 June X = T + E If there were no error noise Observed Score True Score Measurement Error

A primer on scoring 5 June X = T + E Could be issues of: internal consistency test-retest reliability inter-rater reliability Observed Score True Score Measurement Error

A primer on scoring 5 June Var(X) = Var(T) + Var(E) Variance in Observed Scores Variance in True Scores Variance in Errors

Inter-rater reliability 5 June “IRR analysis aims to determine how much of the variance in the observed scores is due to variance in the true scores after the variance due to measurement error between coders has been removed.” Hallgren, Kevin A. “Computing Inter-Rater Reliability for Observational Data: An Overview and Tutorial.” Tutorials in Quantitative Methods for Psychology 8, no. 1 (2012): 23–34.

Inter-rater reliability 5 June “IRR analysis aims to determine how much of the variance in the observed scores is due to variance in the true scores after the variance due to measurement error between coders has been removed.” Hallgren, Kevin A. “Computing Inter-Rater Reliability for Observational Data: An Overview and Tutorial.” Tutorials in Quantitative Methods for Psychology 8, no. 1 (2012): 23–34. If IRR = 0.80: 80% of Var(X) is due to Var(T) 20% of Var(X) is due to Var(E) Var(X) = Var(T) + Var(E)

Measures of IRR 5 June Percentage agreement | not for ordinal data; overestimates agreement 2.Cronbach’s alpha | works for 2 raters only 3.Cohen’s kappa | used for nominal data; works for 2 raters only 4.Fleiss’s kappa | for nominal variables 5.Intra-class correlation (ICC) | perfect!

5 June Intra-class correlation (ICC) Variance due to rated subjects (DMPs) ICC = (Variance due to DMPs + Variance due to raters + Residual Variance) 6 variations of ICC – must choose carefully based on study design Shrout PE, Fleiss JL. Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin. 1979; 86(2):420–428. McGraw KO, Wong SP. Forming inferences about some intraclass correlation coefficients. Psychological Methods. 1996; 1(1):30–46.

Intra-class correlation (ICC) 5 June ICC_results <- icc(ratingsData, model="twoway", type="agreement", unit="single") “two-way” | vs. one-way; raters are random & DMPs are random “agreement” | vs. consistency; looking for absolute agreement b/w raters “single” | vs. average; single ratings are used, not averages of ratings

ICC: consistency vs. agreement 5 June Rater 2 = 1.5 x Rater 1Rater 2 always rates 4 points higher than Rater 1 Rater 2 = Rater 1

Intra-class correlation (ICC) 5 June ICC_results <- icc(ratingsData, model="twoway", type="agreement", unit="single") “two-way” | vs. one-way; raters are random & DMPs are random “agreement” | vs. consistency; looking for absolute agreement b/w raters “single” | vs. average; single ratings are used, not averages of ratings

Inter-rater reliability 5 June Mean = | Median = Standard Deviation = Mean = | Median = Standard Deviation =

Inter-rater reliability 5 June Mean = | Median = Standard Deviation = Mean = | Median = Standard Deviation =

Inter-rater reliability 5 June Mean = | Median = Standard Deviation = Mean = | Median = Standard Deviation = 0.112

5 June poor fair good excellent