Presentation is loading. Please wait.

Presentation is loading. Please wait.

Academic Analytics at CU-Boulder

Similar presentations


Presentation on theme: "Academic Analytics at CU-Boulder"— Presentation transcript:

1 Academic Analytics at CU-Boulder
CU-Boulder – AcAnalytics liaison Lou McClelland Planning, Budget and Analysis, Institutional Research Sept : Short version presented to IA advisory board June : Minor additions and edits June : Presented longer version to Provost and CFO, Russ Moore and Kelly Fox PBA: L:\IR\Reports\GuideGR\AcAnalytics\CU-BoulderAcAnalytics201409Short.pptx

2 AcAnalytics does data about scholarly activity
5/14/2019 CU-Boulder IR, LMcC, CU-BoulderAcAnalytics201409

3 AcAnalytics “Comparative” Database
Data meet criteria such as articles that are peer reviewed Annual “release” in comparative time window – for 2012 annual release Faculty Academic Year (Nov ) Journal Articles Citations Conference Proceedings Books Grants Awards Varies by award and ranges from the previous 10 to 50 years Summary: 4-5 years through release year, books 10 years What’s on the portal is generally from the comparative database 5/14/2019 CU-Boulder IR, LMcC, CU-BoulderAcAnalytics201409

4 AcAnalytics “Full” Database
Extends before and after the most recent release Generally back to 2003 or 2004, not all time Recent updates Generally not every person, source, week, but some are; e.g. Science Differential speed for different clients under consideration Includes scholarly work that’s matched but doesn’t meet criteria, even in comparison time window E.g. book reviews, training grants May include tentative matches with match score 5/14/2019 CU-Boulder IR, LMcC, CU-BoulderAcAnalytics201409

5 AcAnalytics data delivery
Standard web portal Mostly comparative, for PhD programs or “departments” Graphs, listings, journals and grants market share, downloads Users must be “credentialed” Data mart Full and comparative data for all schools, all units, plus CU-Boulder people, in 56 tables. Got access 3/ Work in progress. Super restricted access. Custom files, feeds, and pulls, including VIVO format 5/14/2019 CU-Boulder IR, LMcC, CU-BoulderAcAnalytics201409

6 Metrics about a department or PhD program
Starting point: Units in the discipline and people in each unit Volume – e.g. number of articles Count for each co-author, but only once for a unit Per capita – articles per person Intensity – citations per article, dollars per grant “Percent who” -- Number, percent who have any Compare within discipline because levels of scholarly activity differ systematically and dramatically by discipline E.g., articles/person ranges over disciplines from <1.0 to >15 AcAnalytics usual: Per-capita plus some intensity, some “pct who” 5/14/2019 CU-Boulder IR, LMcC, CU-BoulderAcAnalytics201409

7 CU-Boulder IR, LMcC, CU-BoulderAcAnalytics201409
However, reputation relies on volume! Volume and per-capita aren’t necessarily the same Plot: Physics depts in top half on both measures 5/14/2019 CU-Boulder IR, LMcC, CU-BoulderAcAnalytics201409

8 AcAnalytics at UCB -- Players relationships roles
5/14/2019 CU-Boulder IR, LMcC, CU-BoulderAcAnalytics201409

9 CU-Boulder annual submission
Key responsibility of school, liaison Overhauled 2012 submission entirely based on client work, issues Took several months Considerable consultation with AcAnalytics Results on next slide 2013 submission 85% in To do: Finish, ready for 2014 submission 5/14/2019 CU-Boulder IR, LMcC, CU-BoulderAcAnalytics201409

10 Results for the 2012 database, CU-Boulder
Database entity Old way New way Why Unique individuals (~850 RA/SRAs in both) 2,098 2,149 Corrected omissions Units – “Departments” 43 63 Added research institutes, museum, and omissions; split business Units – PhD programs 47 58 Added new since 2008; split business Units - Other Undifferentiated 2 Libraries plus one Unique AA disciplines 68 (of 172 possible) 105 To cover new and to cover peers Unit x disc. combos 114 314 Some research insts have 15+ People x units 2,213 4,100 Some people have 5+ 5/14/2019 CU-Boulder IR, LMcC, CU-BoulderAcAnalytics201409

11 CU-Boulder major clients – Overview
Comparing CU-Boulder units to those elsewhere Program review fall ’12, ‘13, ’14, ‘15 Academic prioritization – from program review fall ‘12 Units: Math, SLHS A&S dean’s office Gathering data on individuals, not comparisons Faculty Info System (FIS) team Engineering dean’s office 5/14/2019 CU-Boulder IR, LMcC, CU-BoulderAcAnalytics201409

12 CU-Boulder Clients: Academic program review
Fall 2012 – social sciences – per-capita comparisons – delivered very late – some interest, use Fall School of Education used in self-study Fall 2014 – ITP only possible unit Fall 2015 – Music and A&S Humanities These involve three of AcAnalytics’s weakest areas: Creative activity, book chapters, and departments without PhD programs LMcC should meet with these chairs fall 2014 on potential of AcAnalytics data for their self-studies, then work over ‘14-15 to assemble 5/14/2019 CU-Boulder IR, LMcC, CU-BoulderAcAnalytics201409

13 CU-Boulder Clients: Math (and applied math)
Chair asks LMcC for help. Triggers: Academic Prioritization low rating; A&S use of AcAnalytics data; presumed effect of CU-Boulder unusual 2-dept org Lou checked in AcAnalytics data Separate depts IS unusual All AAU’s have math depts; only 18% have applied math as a dept Therefore combined for us and others, then compared A combined CU-Boulder math/applied math dept would stand far higher than either math or applied math did in 2010 data, or data. See next slide 5/14/2019 CU-Boulder IR, LMcC, CU-BoulderAcAnalytics201409

14 Academic Prioritization score
Math (and applied math): Results Academic prioritization research measure: AcAnalytics per-capita, converted to percentile in group Discipline match and dept organization matter! Up to Math Applied Math Combined Academic Prioritization score 100% 2012 (81%) 5 80% 4 60% 2012 (47%) 2010 (45%), 3 40% 2012 (38%) 2 20% 2010 (18%) 1 5/14/2019 CU-Boulder IR, LMcC, CU-BoulderAcAnalytics201409

15 CU-Boulder Clients: Deans’ offices
A&S dean Portal user from 9/2013 LMcC asked for a meeting, provided some caveats and links No other interaction Big user! To do Find out more, help out more Engineering dean’s office Asked 4/15/14 for h-indices for fall 2012 people in EN depts/programs Delivered same day, from data mart, with info on definitions and use And invited Engineering to come back for broader info 5/14/2019 CU-Boulder IR, LMcC, CU-BoulderAcAnalytics201409

16 UCB Clients: FIS (Faculty Info System) team
Interest is in full data (not comparative), by person, at detail level Individual journal articles, awards, books, book chapters Including tentative matches To feed FIS, make some public via VIVO AcAnalytics national workshop, here, Feb ‘14, focusing on detail data Awards: Progress, promising Publications: Trickier due to Elements, other sources, volume, match issues, need for feedback loops on matches ORCID IDs will help When FIS has hundreds, we’ll submit to AcAnalytics for their use 5/14/2019 CU-Boulder IR, LMcC, CU-BoulderAcAnalytics201409

17 CU-Boulder IR, LMcC, CU-BoulderAcAnalytics201409
Ongoing issues requiring collaboration with AcAnalytics and other schools Discipline matches – comparability E.g. Business, Education, SLHS, Math Interdisciplinary depts & programs, including research institutes RA/SRA’s – We report all to AcAnalytics, and can ID in our data AcAnalytics first matched activity to RA/SRA’s for 2012 db No cross-school comparability or ability to check Can you partition change from release A to release B into real vs. improved coverage and improved matching? 5/14/2019 CU-Boulder IR, LMcC, CU-BoulderAcAnalytics201409

18 Internal CU-Boulder issues
Goals in use w depts and programs Tradeoffs between understanding for improvement, and “scoring” Who needs something comparable across departments? PhD programs? Will units seek or use data for understanding if they’re not scored, or forced? Appropriate discipline matches (e.g., math/applied math) Use of volume vs. per-capita vs. “pct who” measures AcAnalytics usual is per capita with “pct who” and intensity Boulder Applied Math is high due to high “pct who” measures Academic reputation may relate most strongly to volume E.g., Number of citations. E.g., World University rankings are 90% volume Better to use multiple measures, trickier to visualize 5/14/2019 CU-Boulder IR, LMcC, CU-BoulderAcAnalytics201409

19 May organize UCB users/priorities by level of analysis
1: Articles, books, awards, grants FIS team (Faculty Info System), VIVO, library Matching issues fall here 2: People Counts, detail, collaborations Finding people working on a topic Salary equity, retention, deans, chairs 3: Units – departments, schools/colleges, PhD programs Explore/understand own Comparisons to other schools and within CU-Boulder 4? Interdisciplinary units and programs 5/14/2019 CU-Boulder IR, LMcC, CU-BoulderAcAnalytics201409

20 CU-Boulder IR, LMcC, CU-BoulderAcAnalytics201409
Some next steps Finish 2013 submission, prep for 2014 Music and A&S Humanities for 2015 program review FIS team support Internal outreach, priority-setting Enable and support portal users including deans’ offices Other users and uses including beta products, data mart Continue work with AcAnalytics on issues Website 5/14/2019 CU-Boulder IR, LMcC, CU-BoulderAcAnalytics201409


Download ppt "Academic Analytics at CU-Boulder"

Similar presentations


Ads by Google