JOURNAL METRICS & EVALUATION Chan Li Sr. Data Analyst California Digital Library ALA 2012 Midwinter

BIG DEALS CHALLENGES AND ISSUES  Big Deals reinforce publisher monopolies, but also provide broader access to a wide variety of journals.  Journals expenditures are far outpacing library budgets

ANALYZE JOURNAL VALUE: VALUE- BASED STRATEGY  Use objective metrics to compare the value of scholarly journals, e.g. usage, cost, citations, etc.  Collect, create, organize, and assign quantitative value rankings to 8,500 UC licensed journals, based on subject categories  The process is objective and quantifiable and is based both on cost and value with a goal of leading to more sustainable prices overall

CDL OVERALL WEIGHTED VALUE METRIC  Definition: Overall Weighted Value Metrics measures a title’s relative UC value against the other UC licensed titles within the same subject categories from utility, cost effectiveness and quality perspectives.  Metrics included:  Subjects Categories Measurement CategoriesMetrics UtilityUC Usage UC Citations QualitySNIP Impact Factor Cost EffectivenessUC Cost Per Use UC Cost Per SNIP

OVERALL WEIGHTED VALUE ALGORITHM: BENCHMARK VALUES  Benchmark Values  Median values: resistant to extreme numbers  Benchmark values are different across subjects  Multiple benchmark values for some metrics Usage: median, lower quartile Cost per use: median, ILL cost

OVERALL WEIGHTED VALUE ALGORITHM : DISCIPLINARY DIFFERENCES MATTER IN JOURNAL EVALUATION SubjectsMedian UsageMedian Impact Factor Life and Health Sciences Medical and Health Sciences 1, Plant Biology Physical Science and Engineering Mathematical Sciences Physical Chemistry1, Social ScienceHistorical Studies Psychology Arts and HumanitiesPhilosophy Creative Arts and Writing 192N/A

OVERALL WEIGHTED VALUE ALGORITHM: OVERALL WEIGHTED VALUE SCORES CategoryMetricScore Weight UtilityUsage25% Citation15% QualitySNIP15% Impact factor15% Cost Effectiveness Cost Per Use15% Cost Per SNIP15% Value Scores are assigned at journal title level Not all the metrics are weighted equally: Score is based on whether a title’s values are above or below the benchmark values. Titles with no data available will be assigned with lowest scores Value categories are assigned: lowest, low, medium and high

OVERALL WEIGHTED VALUE ALGORITHM: UC INNOVATIONS IN JOURNAL EVALUATION  Composite score using multiple vectors of value  Utility  Quality  Cost-effectiveness  Value categories are assigned based on scores: lowest, low, medium and high  Subject differences are counted  Relative to local context  “How much value does UC derive from this journal compared to other journals that we license in the same discipline?”  Others can use our methodology, but not our rankings

CDL OVERALL WEIGHTED VALUE ALGORITHM: CDL DATA ANALYSIS PROCESS Assign Scores to UC 8,500+ licensed Titles All UC licensed Titles- -- 8,500+titles, 36 journal publishers Data Collection Subject UC usage UC citation SNIP Impact Factor Cost Per Use Cost SNIP Other metrics Subject UC usage UC citation SNIP Impact Factor Cost Per Use Cost SNIP Other metrics Metrics Assign Values Median Values for each metric by specific subject Data Analysis 8,500 Titles Value Results Compare titles against median values

CDL OVERALL WEIGHTED VALUE ALGORITHM: APPLICATIONS  Identify high value titles to UC libraries across all the publishers  Identify high value titles by publisher  Compare overall publisher packages values  Identify the publishers that provide better value in a specific subject area

DISCUSSION Chan Li: