1. Taking Student Success to Scale. High Impact Practices in the States

2. Assessment as Culture Management & Professional Development
Jerry Daday
Executive Director
Center for Innovative Teaching & Learning
Western Kentucky University

3. Criteria for high quality hips
Expectations set at appropriately high levels
Intentional (clear Essential Learning Outcomes (ELOs); structured experience
Significant investment of time and effort
Preparation, orientation, and training
Interaction with faculty and peers
Experience with diversity
Frequent and constructive feedback
Periodic and structured opportunities for reflection
Relevance through real world applications (i.e. hands-on experience)
Public demonstrations of competence
Kuh & O’Donnell (2013)

4. outcomes Transactional Outcomes Retention Persistence GPA
Essential Learning Outcomes
AAC&U LEAP Essential LO
Value rubrics used for assessment (Value rubrics used for assessment)
Deep Learning
Applied Learning
Community/relationship building

5. HIPs Inventory
Last spring, we conducted an inventory of HIPs used in 53 of our majors at WKU
88% of these majors offer a writing-intensive course
77% offer a capstone experience
66% offer collaborative/project based assignments
64% offer common intellectual experience
54% offer global-learning
43% offer undergraduate research & creative activities

6. HIPs Inventory
On average, departments reported that students receive 5 HIPs within each major 1 major = 10 HIPs 2 majors = 9 HIPs 3 majors = 8 HIPs

7. Validity Are we measuring what we think we are measuring?
Before we can accurately/validily measure a concept (like a HIP), we need to engage in . . .
Conceptualization: process of defining a concept
Operationalization: process of specifying measures/indicators of a concept

8. Value of taxonomies Explicitly define HIPs
Ensure fidelity of the HIP by expressing purpose/intent
Essential Learning Outcomes (ELOs)
Specific attributes that define the HIP
Guide professional development of faculty & staff
PD tool for improvement
Validation of activity’s impact for tenure & promotion and annual review
Precursor to assessment effort
Must define parameters before meaningful assessment of student learning can take place

9. Value of taxonomies Programs within a department or university
Taxonomies may be used to guide the develop of
Individual courses
First-Year-Experience
Living-Learning-Communities
Undergraduate Research
Service Learning
Programs within a department or university
Internships & Study Abroad
University-wide initiatives

10. Example: CSU Service Learning (Slide 1)

11. Example: CSU Service Learning (Slide 2)

12. Intermission

13. HIP: What Works and How Do We Know?
Jennifer Merriman, PhD
Executive Director Research
August 25, 2017

14. Why should we care about program evaluation?
If it is not working, why keep doing it?
Why should we care about program evaluation?
$$$$

15. Overview HIP Theory of Action Heterogeneity of Programs
Revised Theory of Action
Claims
Formative and Summative Evaluation
Fidelity of Implementation
Developing an evaluation to help determine if your program is HIGH IMPACT

16. Improved Student Outcomes
Theory of Action
HIP
Improved Student Outcomes
The causal logic of a program’s impact on outcomes

17. Heterogeneity of Practices
Documentation of the specific practices yields more accurate evaluation of impact
What do we mean by HIP?
How is it operationalized?

18. Variation in which practices offered
Institution A B C
First-Year Seminar and Experiences √
Common Intellectual Experiences
Learning Communities
Writing-Intensive Courses
Collaborative Assignments and Projects
Undergraduate Research
Diversity/Global Learning
Service Learning
Internships
Capstone Courses and Projects

19. Variation in practice operationalization
Institution
Collaborative Assignments and Projects A B C
Study Groups √
Team-based assignments
Cooperative projects
Meet twice a year
Meet once a month
Meet weekly
Professor-led
Student-led

20. Improved Student Outcomes
Theory of Action
HIP
Improved Student Outcomes
Which practices?
Implemented in what ways?
For which students?
In what context?
Which outcomes?
Why and how will these practices influence student outcomes?
How does implementation affect outcomes?
Are outcomes seen for all students?
In all contexts? 20

21. Revised Theory of Action
Specificity and context matter
Revised Theory of Action
Urban institution serving many 1st gen students
Student Outcomes
Engagement
GPA
Retention
Other?
Team-based assignments
Meet weekly throughout semester
Student-led
Collaborative Assignments and Projects
Specific implementation details here
Specifics
Internships
Undergraduate Research
All Students
Only Policy Students
Only STEM
Students

22. Black Box vs Specific Practices
Which practices actually drive outcomes?
A black box approach might mask effectiveness, OR heterogeneous effects might yield null effects on average
Black Box vs Specific Practices
Student Outcomes:
GPA
Team-based assignments
Meet weekly throughout semester
Student-led
Collaborative Assignments and Projects
Specific implementation details here
Specifics
Internships
Undergraduate Research
All Students
Only Policy Students
Student Outcomes:
None
Only STEM
Students
Student Outcomes:
None

23. Claims Too Vague Example Claims
Students who participate in HIP are more successful
Traditionally underserved minority students who successfully complete an undergraduate research project in a STEM field at 4-year public IHEs are 8 times more likely to go on to graduate school.
Too Vague
Figure out the kinds of claims you want to make BEFORE you begin developing your evaluation.
The desired claims will drive evaluation design and data requirements.

24. Formative vs Summative Evaluation
Evaluation Type
Definition
Uses
Examples
Formative
Evaluates a program during  development in order to make early improvements
Helps to refine or improve program
When starting a new program
To assist in the early phases of program development
How well is the program being delivered?
What strategies can we use to improve this program?
Summative
Provides information on program effectiveness
Conducted after the completion of the program design
To help decide whether to continue or end a program 
To help determine whether a program should be expanded
to other locations
Should this program continue to be funded?
Should we expand these services to all other after-school programs in the community?

25. Fidelity of Implementation
The extent to which the delivery of an intervention adheres to the program model as intended by the developers of the intervention
If null program effects are found it may be that the program is not effective OR that the program was not implemented with fidelity
Five dimensions:
Adherence: Program adherence refers to the extent to which program components are delivered as prescribed by the model. Adherence indicators can include program content, methods, and activities.
2) Exposure: Program exposure (i.e., dosage) is the amount of program delivered in relation to the amount prescribed by the program model. Exposure can include the number of sessions or contacts, attendance, and the frequency and duration of sessions.
3) Quality of delivery: Quality of delivery reflects the manner in which a program is delivered. Aspects of delivery quality can include provider preparedness, use of relevant examples, enthusiasm, interaction style, respectfulness, confidence, and ability to respond to questions and communicate clearly.
4) Participant responsiveness: Participant responsiveness refers to the manner in which participants react to or engage in a program. Aspects of participant responsiveness can include participants’ level of interest in the program; perceptions about the relevance and usefulness of a program; and their level of engagement, enthusiasm, and willingness to engage in discussion or activities.
5) Program differentiation: Program differentiation is the degree to which the critical components of a program are distinguishable from each other and from other programs.
Fidelity of Implementation

26. intermission

27. Action Steps Describe your HIP programs carefully:
Which programs do we offer?
Who participates in the programs?
How is the program operationalized?
What are the expected outcomes for each program and why do we think we will see that outcome?
This sets up the Theory of Action
What claims do we want to make from the evaluation?
Different stakeholders have different claims they want to make internally and externally—vet the claims with all pertinent groups
Is the program still is development, continuously changing, or fully developed?
This will determine whether to undertake formative or summative evaluation
Data availability
How will we standardize our metrics? Are all data available for the evaluation? When will the data be available? When do we need to report findings?
What is the counterfactual?
When you talk about program outcomes you need to think carefully about the comparison group—improved student success compared to what?
What can I do next?

28. Key Questions
Taxonomies can be used for faculty professional development to ensure HIPs are implemented with fidelity
Taxonomies can also be used to set up program evaluation. For example . . .
Is there a difference in outcomes based on these degrees?
Does a “low intensity” HIP have the same impact as a “high intensity HIP”?
What should we track for outcomes?
What do you wish to measure? (transactional? essential learning? Deep or applied learning?)
What about the 8 criteria of a HIP (Kuh & O’Donnell 2013)?
Should these be what we are measuring as attributes across all HIPS?
Or should attributes vary across HIPs?

29. Taking student success to scale. High Impact Practices in the States.

30. National Model and Laboratory for Student Success