Analytics: Planning Considerations Jacqueline Bichsel, John Fritz, Margie Jantti, and Celeste Schwartz with Sondra Smith EDUCAUSE 2012 | Preconference Seminar 10P | November 6, 2012

Seminar Agenda Introductions Analytics and the maturity index John Fritz and academic analytics at UMBC Break Margie Jantti and the Wollongong library cube Celeste Schwartz and data informed decisions at MCCC Closing questions, comments, and evaluations

Seminar Speakers Jacqueline Bichsel Senior Research Analyst, EDUCAUSE John Fritz Assistant Vice President, Instructional Technology & New Media  University of Maryland, Baltimore County  Margie Jantti University Librarian, University of Wollongong  Celeste M. Schwartz Vice President for Information Technology and College Services  Montgomery County Community College  Sondra Smith Director, Analytics Outreach and Education, EDUCAUSE 

(Selections from) a Year of Analytics http://www.educause.edu/library/analytics “SLIDER BAR” TOOL ANALYTICS SPRINT ECAR ANALYTICS STUDY

Seminar Agenda Introductions Analytics and the maturity index John Fritz and academic analytics at UMBC Break Margie Jantti and the Wollongong library cube Celeste Schwartz and data informed decisions at MCCC Closing questions, comments, and evaluations

The 2012 Study of Analytics in Higher Education Jacqueline Bichsel, Ph The 2012 Study of Analytics in Higher Education Jacqueline Bichsel, Ph.D. Senior Research Analyst, ECAR

Defining Analytics

EDUCAUSE Definition of Analytics Analytics is the use of data, statistical analysis, and explanatory and predictive models to gain insights and act on complex issues.

Analytics Is a Priority

But Analytics Use Is Lagging …particularly in areas of administrative and faculty activities

Higher Education’s Progress

ECAR Analytics Maturity Index

Benchmark Your Institution Measure your progress on the ECAR Analytics Maturity Index http://www.educause.edu/library/resources/2012-ecar-study-analytics-higher-education

Seminar Agenda Introductions Analytics and the maturity index John Fritz and academic analytics at UMBC Break Margie Jantti and the Wollongong library cube Celeste Schwartz and data informed decisions at MCCC Closing Q & A and evaluations

About Blackboard @ UMBC Began using Bb in Spring 2000 Current version: 9.1, SP6 Adoption 95% of all students 75% of all instructors 65% of all courses 350 Communities Analytics Initiatives 2012 Bb A4L LFT & Implementation 2011 Learning Analytics in Gateway Courses sub-grant 2010 Adopted iStrategy for analysis of all Bb courses 2009 Code release at BbWorld 2009 2008 Check My Activity for Students available 2007 www.umbc.edu/blackboard/reports Support Staff: 2 FTE (Admin & Support) 1 Server Admin John

www.umbc.edu/blackboard/reports

Bb Activity by Grade Distribution

New Method: Bb A4L (formerly iStrategy) BbA4L Data Warehouse (Queries) Blackboard (Activity) Student Admin. (Grades, Demographics) Jeffrey Check My Activity (Students) Most Active Bb Courses (Faculty) www.umbc.edu/blackboard/reports

Accesses by Grade (SP2012)

New Reports

Evolving CMS Use by Faculty User & Document Management (Pull) Password-protected class & group space Attach or Copy/Paste Documents (expiration) Communications (Push) Announcements Email, Messages Discussion & Chat Assessments (Push & Pull) Electronic assignment delivery & collection Quizzing, Surveys, Course Usage Adaptive release of content based on prior student action.

Tim Hardy & ECON 122 Tim attends hybrid course design workshop One of six faculty funded by the Provost’s FA2009 pilot to deliver a 50/50 hybrid course. ·    Joined the SU2009 cohort of the Alternate Delivery Program (ADP) ·    Attended the January 2009 Hybrid Course Redesign Workshop. ·    His UMBC Blackboard ranking for student-only activity has increased dramatically: FA2010 (1st overall, 2,970 average hits per student) * See note below * SP2010 (1st overall, 1,345 average hits per student) FA2009 (1st overall, 1,666 average hits per student) SU2009 (4th overall, 842 average hits per student) SP2009 (36th overall, 366 average hits per student) FA2008 (532nd overall, 78 average hits per student) SU2008 (38th overall, 334 average hits per student) SP2008 (76th overall, 494 average hits per student) FA2007 (151st overall, 326 average hits per student) Note: His ECAC329 “Cost Accounting” course registered 5,536 avg. hits per student, 1st among all UMBC Bb courses (not just undergrad) for FA2010. SP2011 (53.7%) FA2010 (49.4%) SP2010 (47.7%) FA2009 (46%)

Grade Center Impact on Activity Building & Scaling Analytic Capacity 1/30/2012 Grade Center Impact on Activity We’ve also used student activity to identify and promote effective practices. For example, we’ve seen that courses using the grade center tend to generate more student activity than courses that don’t. Research by Educause confirms that students value checking grades and access to practice quizzes more than any other function. But this can’t simply mean that faculty should assign and grade more student work. So, how do we scale up the feedback function? Purdue University & Gardner Institute

Personal Analytics in Action

Can IT Architect User Choices? A good book on how "choice architecture" creates default choices that are anything but neutral. RECAP Record Evaluate Compare Alternative Prices – or Performance

Demo & Practice Bb Analytics for Learn (BA4L) Instance

Seminar Agenda Introductions Analytics and the maturity index John Fritz and academic analytics at UMBC Break Margie Jantti and the Wollongong library cube Celeste Schwartz and data informed decisions at MCCC Closing Q & A and evaluations

Impact and effect Margie Jantti, University Librarian A study on library use and student performance Margie Jantti, University Librarian University of Wollongong Library, Australia, 2012

There are many approaches to evaluating the value of libraries. As an academic library, the focus is on the transformative power of information There are many approaches to evaluating the value of libraries.

Limited focus on the transformative power of information Typical measures Satisfaction measures Rankings Contingency valuation Information literacy assessment Usage rates Limitations Scale, e.g. samples versus population Subjective Often one-dimensional Limited focus on the transformative power of information

Problem statement I: Does a student’s academic performance improve through the use of library information resources?

Problem statement II: How do we know which students make little or no use of library information resources?

Libraries and other units produce lots of discrete data

The challenge was to build a relational database (or cube) books ereadings databases ebooks student grades

Data sources for the Cube Student data – PIU Loans – snapshot exported weekly Electronic resource usage – ezyproxy logs

Eresources: >Authentication logs (ezyproxy) >Monitor use in 10 minute blocks (144 ten minute periods p/day) >When used – database name captured Other benefits logs are updated weekly

Figure 1. Correlation between electronic resource usage and student grades

Electronic resources: ejournals, ebooks, ereadings etc Undergraduate student resource usage (Faculty of Commerce) Table 1. Weighted Average Marks (WAM) for Undergraduate Students (Faculty of Commerce) Electronic resources: ejournals, ebooks, ereadings etc

Is it perfect? >Some limitations in correlations >Arbitary measures; business rules >Many external factors affect grades, e.g. academic influence

But……. A first at UOW Library: >for true integration of data silos >getting answers to our problem statements >evaluating our communication and intervention strategies >for a new way of demonstrating the value of the Library

QUESTIONS?

Seminar Agenda Introductions Analytics and the maturity index John Fritz and academic analytics at UMBC Break Margie Jantti and the Wollongong library cube Celeste Schwartz and data informed decisions at MCCC Closing questions, comments, and evaluations

Data-Informed Decision-Making throughout the Institution Celeste M. Schwartz, Ph. D. Vice President for Information Technology and College Services

What is the Question. What Data can help answer the Question What is the Question? What Data can help answer the Question? What Actions will be recommended based on the data?

Astra – Room Scheduling

Student Success This office uses Dashboards as opposed to the Reports we have reviewed.

Enrollment Analysis

Help Desk Weekly Lab Usage Monday Tuesday Central Library – 70% utilization at 12:00pm(noon) Wednesday 10/3/2012 West Library – 70% utilization at 11:00am Monday 10/2/2012 CH Tutoring – 30% utilization at 10:00am Monday 10/1/2012 West Library Laptops – 20% utilization at 7:00pm Monday 10/1/2012 Central Library Laptops – 20% utilization at 12:00pm(noon) Thursday 10/4/2012 Monday Tuesday

Total number of tickets created: 456 Summary of Weekly calls by Status & Assignment.

Help Desk Feedback Report Date Representative Email Problem Resolved? Service Rating Comments 10/1/2012 5:02:02 PM robert gehring Not provided Yes Very Satisfied Representative was very helpful, and answered all my questions. 10/1/2012 6:43:32 PM Robert Gehring He was fast, polite and extremely helpful. Thank you. 10/2/2012 3:07:37 PM Jasmyne Smith   10/3/2012 4:37:26 PM Ryan Foster Satisfied Ryan did well to resolve my loss of available drives. It did take almost two full weeks to resolve this issue after first being reported on 9/20/12. 10/4/2012 10:08:00 AM Jasmin 10/4/2012 11:55:31 AM Jennifer Great. Thanks 10/4/2012 1:44:50 PM Jen dalder@mc3.edu This is the third time I've contacted the help desk. On each occasion, you were EXTREMELY patient and helped me resolve the problem. Additionally, you do it WITHOUT making me feel like an idiot! Thank you! 10/5/2012 8:53:06 AM Kathy Miller Kathy is always willing to listen and ask questions until everyone is clear on what the difficulty/need is! Many, many thanks to Kathy and Mary Lou! 10/6/2012 11:39:35 AM Robert Robert is remarkable and we are very fortunate to have him. Because it was so late at night when my computer crashed, he was able to talk me through re-booting so that I could finish the class and not waste time and inconvenience all my students. Sincere thanks, Lee

What is the Question. What Data can help answer the Question What is the Question? What Data can help answer the Question? What Actions will be recommended based on the data?

Now it’s Your Turn What Actions will you recommend based on the Data? Read the daily enrollment email from the VP for Student Affairs based on the daily updated warehouse data. Take ten minutes with your team to discuss recommended actions Each team will report out

Data-Informed Decision-Making throughout the Institution QUESTIONS Cschwartz@mc3.edu

Seminar Agenda Introductions Analytics and the maturity index John Fritz and academic analytics at UMBC Break Margie Jantti and the Wollongong library cube Celeste Schwartz and data informed decisions at MCCC Closing questions, comments, and evaluations

Thank You. jbichsel@educause.edu fritz@umbc.edu margie@uow.edu.au cschwartz@mc3.edu srsmith@educause.edu