1. IR’s role in empowering student advising through Learning Analytics
Stefano Fiorini, Lead Research Management Analyst
Pallavi Chauhan, Data and Research Analyst
INDIANA UNIVERSITY BLOOMINGTON BLOOMINGTON ASSESSMENT AND RESEARCH

2.
“ The Bloomington Assessment and Research office supports faculty, programs, campus initiatives, and the student experience by providing research and analyses for data-driven decisions and continuous improvement.”

3. Administrative Support Research Capabilities
Infrastructure and IT Support
Indiana University
Bloomington
Research Capabilities
Wealth of Data
Faculty and Staff
Students

4. Learning Analytics
“ […] the most dramatic factor shaping the future of higher education is something that we can’t actually touch or see: big data and analytics.”
George Siemens (2011)

5. Learning Analytics at IUB
with a plethora of data and rich research expertise available at IUB we are using a ground-up, top-down and middle-out approach to fully engage in producing useful analytics that inform the institution across all levels (students, courses, programs, institution as a whole) to achieve student success

6. The Project
Assist advisors in helping students follow successful course pathways through curriculum
Augment advisor's knowledge with predictive algorithms designed to identify risk
Provide actionable information to advisors

7. Utilizing Machine Learning (ML) Algorithms to Support Student Success

8. Problem Statement
Vended Products utilizing ML algorithms are not being used on campus for decision-making
It lacks specificity

9. Uncovering the Problem Statement
Black box
Lack of transparency
Hard to interpret
Solutions/Actions suggested too diffused
Predictions in all cases/too many false positives
Created lack of trust in data
It does not engage the user in a smart and intuitive way

10. Direction & Project Goals
Build advisors’ trust in data and understanding of predictive analytics
Foster research-motivated advising
Build analysts’ understanding of advising processes and needs
Design a user-centric prediction tool for analytics and insight
Participatory approach to development

11. Participatory Action Research
Academic
Advisors
Research
Analysts
Identify the purpose of ML analytical systems from key stakeholders
What does it mean to you? What value does it add to student advising? How does it support student success?

12. Participatory Approach
Partnered with University Division Advising at IU Bloomington
~10,000 UDIV students
1 Advisor: Students
~350 majors

13. Design Cycle – Phase1

14. Learning Analytical (LA) Tool Context
The student context
For our analyses, low grades are considered C and below
Low grades are rare on our campus (12% of all grades)
Interest in low grades because studies show that low grades increase the odds of dropping out (Feng Et Al. 2017)

15. Learning Analytical (LA) Tool Development
Design parameters
Multiple course pathways taken by student
Individual course contribution
Homogeneity of majors
Optimize model (above random) to identify students at risk of getting low grades
Predicted
Actual
Avg
High
Low

16. Insights from Phase 1 Assessment goal: expose convergence of LA Tool prediction with Advisor’s risk Identification
Strong Convergence
Advisors helped identify and code recurring of themes that serve to signal potential struggles in any course enrollment
Performance in similar courses was the most important factor that advisors use to identify potential struggle
Behavior/Motivational issues and Test scores ranked the lowest in indicating potential struggle

17. Quantitative convergence
LA Tool Prediction
Advisors Assessment
Safe
At Risk
Total
131 (46%)
153 (54%)
284
30 (21%)
113 (79%)
143
161
266
427

18. Comparison with actual students performance
Advisors evaluation
LA Tool Prediction
Actual Student Performance
Safe
At Risk
Total
226 (80%)
58 (20%)
284
142 (50%)
58 (41%)
85 (59%)
143
19 (13%)
124 (87%)
427
161
266

19. Design Cycle - Phase 2

20. Phase 2 (Pilot)

21. Phase 2 (Pilot)

22. Insights from Phase 2 Assessment goal:  Evaluate information quality and ease of information access 
52 % of students predicted at risk received a low final grade in the course or withdrew
Other themes were identified to be addressed in the next phase include
Academic Probation
GPA Trends (Cumulative GPA/Term GPA)
W’s or Repeated Courses
Design functionalities/Ease of Use
Impact of Co-enrollments
Need for development of an all-encompassing advisor portal that has mechanism for outreach to student

23. Design Cycle - Phase 3

24. Phase 3

25. Initial Insights from Phase 3 – Next Steps
Feedback from advisors to identify convergence of predictions with midterm grades/early evaluation
Work with advisors to map out the who what and how of outreach
Other emerging themes:
Include High School Data
Include W’s
Explore ways to directly capture outreach done by advisors and feed into the algorithms to improve prediction – cross classification

26. Participatory Approach Outcomes
User-center identification of roles and development needs of LA Tools
Respond to data provision needs
Respond to contextual information needs
Promotes data driven culture for advising students
Engagement with Advisor Research Group
Numerous questions from Advisors to Analysts
New LA tool development objectives
 Anchor the tool development, assessment and its use to intentional advising practices

27. sfiorini@indiana.edu pallchau@indiana.edu
Open Discussion...
Special thanks to University Division Advisors for their collaboration!
Our contacts: