1. Academic Analytics at CU-Boulder
CU-Boulder – AcAnalytics liaison Lou McClelland
Planning, Budget and Analysis, Institutional Research
Sept : Short version presented to IA advisory board
June : Minor additions and edits
June : Presented longer version to Provost and CFO, Russ Moore and Kelly Fox
PBA: L:\IR\Reports\GuideGR\AcAnalytics\CU-BoulderAcAnalytics201409Short.pptx

2. AcAnalytics does data about scholarly activity
5/14/2019
CU-Boulder IR, LMcC, CU-BoulderAcAnalytics201409

3. AcAnalytics “Comparative” Database
Data meet criteria such as articles that are peer reviewed
Annual “release” in comparative time window – for 2012 annual release
Faculty Academic Year (Nov )
Journal Articles
Citations
Conference Proceedings
Books
Grants
Awards Varies by award and ranges from the previous 10 to 50 years
Summary: 4-5 years through release year, books 10 years
What’s on the portal is generally from the comparative database
5/14/2019
CU-Boulder IR, LMcC, CU-BoulderAcAnalytics201409

4. AcAnalytics “Full” Database
Extends before and after the most recent release
Generally back to 2003 or 2004, not all time
Recent updates
Generally not every person, source, week, but some are; e.g. Science
Differential speed for different clients under consideration
Includes scholarly work that’s matched but doesn’t meet criteria, even in comparison time window
E.g. book reviews, training grants
May include tentative matches with match score
5/14/2019
CU-Boulder IR, LMcC, CU-BoulderAcAnalytics201409

5. AcAnalytics data delivery
Standard web portal
Mostly comparative, for PhD programs or “departments”
Graphs, listings, journals and grants market share, downloads
Users must be “credentialed”
Data mart
Full and comparative data for all schools, all units, plus CU-Boulder people, in 56 tables. Got access 3/ Work in progress. Super restricted access.
Custom files, feeds, and pulls, including VIVO format
5/14/2019
CU-Boulder IR, LMcC, CU-BoulderAcAnalytics201409

6. Metrics about a department or PhD program
Starting point: Units in the discipline and people in each unit
Volume – e.g. number of articles
Count for each co-author, but only once for a unit
Per capita – articles per person
Intensity – citations per article, dollars per grant
“Percent who” -- Number, percent who have any
Compare within discipline because levels of scholarly activity differ systematically and dramatically by discipline
E.g., articles/person ranges over disciplines from <1.0 to >15
AcAnalytics usual: Per-capita plus some intensity, some “pct who”
5/14/2019
CU-Boulder IR, LMcC, CU-BoulderAcAnalytics201409

7. CU-Boulder IR, LMcC, CU-BoulderAcAnalytics201409
However, reputation relies on volume! Volume and per-capita aren’t necessarily the same Plot: Physics depts in top half on both measures
5/14/2019
CU-Boulder IR, LMcC, CU-BoulderAcAnalytics201409

8. AcAnalytics at UCB -- Players relationships roles
5/14/2019
CU-Boulder IR, LMcC, CU-BoulderAcAnalytics201409

9. CU-Boulder annual submission
Key responsibility of school, liaison
Overhauled 2012 submission entirely based on client work, issues
Took several months
Considerable consultation with AcAnalytics
Results on next slide
2013 submission 85% in
To do: Finish, ready for 2014 submission
5/14/2019
CU-Boulder IR, LMcC, CU-BoulderAcAnalytics201409

10. Results for the 2012 database, CU-Boulder
Database entity
Old way
New way
Why
Unique individuals (~850 RA/SRAs in both)
2,098
2,149
Corrected omissions
Units – “Departments” 43 63
Added research institutes, museum, and omissions; split business
Units – PhD programs 47 58
Added new since 2008; split business
Units - Other
Undifferentiated 2
Libraries plus one
Unique AA disciplines
68 (of 172 possible)
105
To cover new and to cover peers
Unit x disc. combos
114
314
Some research insts have 15+
People x units
2,213
4,100
Some people have 5+
5/14/2019
CU-Boulder IR, LMcC, CU-BoulderAcAnalytics201409

11. CU-Boulder major clients – Overview
Comparing CU-Boulder units to those elsewhere
Program review fall ’12, ‘13, ’14, ‘15
Academic prioritization – from program review fall ‘12
Units: Math, SLHS
A&S dean’s office
Gathering data on individuals, not comparisons
Faculty Info System (FIS) team
Engineering dean’s office
5/14/2019
CU-Boulder IR, LMcC, CU-BoulderAcAnalytics201409

12. CU-Boulder Clients: Academic program review
Fall 2012 – social sciences – per-capita comparisons – delivered very late – some interest, use
Fall School of Education used in self-study
Fall 2014 – ITP only possible unit
Fall 2015 – Music and A&S Humanities
These involve three of AcAnalytics’s weakest areas: Creative activity, book chapters, and departments without PhD programs
LMcC should meet with these chairs fall 2014 on potential of AcAnalytics data for their self-studies, then work over ‘14-15 to assemble
5/14/2019
CU-Boulder IR, LMcC, CU-BoulderAcAnalytics201409

13. CU-Boulder Clients: Math (and applied math)
Chair asks LMcC for help. Triggers:
Academic Prioritization low rating; A&S use of AcAnalytics data; presumed effect of CU-Boulder unusual 2-dept org
Lou checked in AcAnalytics data
Separate depts IS unusual
All AAU’s have math depts; only 18% have applied math as a dept
Therefore combined for us and others, then compared
A combined CU-Boulder math/applied math dept would stand far higher than either math or applied math did in 2010 data, or data.
See next slide
5/14/2019
CU-Boulder IR, LMcC, CU-BoulderAcAnalytics201409

14. Academic Prioritization score
Math (and applied math): Results Academic prioritization research measure: AcAnalytics per-capita, converted to percentile in group Discipline match and dept organization matter!
Up to
Math
Applied Math
Combined
Academic Prioritization score
100%
2012 (81%) 5
80% 4
60%
2012 (47%)
2010 (45%), 3
40%
2012 (38%) 2
20%
2010 (18%) 1
5/14/2019
CU-Boulder IR, LMcC, CU-BoulderAcAnalytics201409

15. CU-Boulder Clients: Deans’ offices
A&S dean
Portal user from 9/2013
LMcC asked for a meeting, provided some caveats and links
No other interaction
Big user! To do Find out more, help out more
Engineering dean’s office
Asked 4/15/14 for h-indices for fall 2012 people in EN depts/programs
Delivered same day, from data mart, with info on definitions and use
And invited Engineering to come back for broader info
5/14/2019
CU-Boulder IR, LMcC, CU-BoulderAcAnalytics201409

16. UCB Clients: FIS (Faculty Info System) team
Interest is in full data (not comparative), by person, at detail level
Individual journal articles, awards, books, book chapters
Including tentative matches
To feed FIS, make some public via VIVO
AcAnalytics national workshop, here, Feb ‘14, focusing on detail data
Awards: Progress, promising
Publications: Trickier due to Elements, other sources, volume, match issues, need for feedback loops on matches
ORCID IDs will help
When FIS has hundreds, we’ll submit to AcAnalytics for their use
5/14/2019
CU-Boulder IR, LMcC, CU-BoulderAcAnalytics201409

17. CU-Boulder IR, LMcC, CU-BoulderAcAnalytics201409
Ongoing issues requiring collaboration with AcAnalytics and other schools
Discipline matches – comparability
E.g. Business, Education, SLHS, Math
Interdisciplinary depts & programs, including research institutes
RA/SRA’s – We report all to AcAnalytics, and can ID in our data
AcAnalytics first matched activity to RA/SRA’s for 2012 db
No cross-school comparability or ability to check
Can you partition change from release A to release B into real vs. improved coverage and improved matching?
5/14/2019
CU-Boulder IR, LMcC, CU-BoulderAcAnalytics201409

18. Internal CU-Boulder issues
Goals in use w depts and programs
Tradeoffs between understanding for improvement, and “scoring”
Who needs something comparable across departments? PhD programs?
Will units seek or use data for understanding if they’re not scored, or forced?
Appropriate discipline matches (e.g., math/applied math)
Use of volume vs. per-capita vs. “pct who” measures
AcAnalytics usual is per capita with “pct who” and intensity
Boulder Applied Math is high due to high “pct who” measures
Academic reputation may relate most strongly to volume
E.g., Number of citations. E.g., World University rankings are 90% volume
Better to use multiple measures, trickier to visualize
5/14/2019
CU-Boulder IR, LMcC, CU-BoulderAcAnalytics201409

19. May organize UCB users/priorities by level of analysis
1: Articles, books, awards, grants
FIS team (Faculty Info System), VIVO, library
Matching issues fall here
2: People
Counts, detail, collaborations
Finding people working on a topic
Salary equity, retention, deans, chairs
3: Units – departments, schools/colleges, PhD programs
Explore/understand own
Comparisons to other schools and within CU-Boulder
4? Interdisciplinary units and programs
5/14/2019
CU-Boulder IR, LMcC, CU-BoulderAcAnalytics201409

20. CU-Boulder IR, LMcC, CU-BoulderAcAnalytics201409
Some next steps
Finish 2013 submission, prep for 2014
Music and A&S Humanities for 2015 program review
FIS team support
Internal outreach, priority-setting
Enable and support portal users including deans’ offices
Other users and uses including beta products, data mart
Continue work with AcAnalytics on issues
Website
5/14/2019
CU-Boulder IR, LMcC, CU-BoulderAcAnalytics201409