1. Keston Fulcher, PhD James Madison University
Program Assessment Workshop for Staff A Presentation for Palo Alto College 1/10/2018
Keston Fulcher, PhD
James Madison University

2. What is Assessment?
Assessment is a bureaucratic exercise that has little bearing to our work.
OR…
Assessment is a process that compellingly tells our story: The goals we’ve set; the strategies we’ve used to accomplish them; how we’ve affected our students and institution; how we’ve learned from our failures; how we’ve celebrated our successes.

3. A Note about Improvement
The Damon Dash Story….

4. Overview Admin vs. Academic Program Asmt Outcomes for Participants
Contrast outcomes and actions in an assessment framework
Distinguish among beginning, developing, good, and advanced assessment reporting
Discuss the fundamentals of identifying problems, collecting baseline data, systematically addressing the issues, and then re-assessing to determine success

5. Admin vs. Academic Asmt
Academic Units: Focus typically on student learning outcomes
Administrative Units: Focus typically on effectiveness/efficiency (more on this later)
Administrative Units: Much more diverse

6. Outcome vs. Action
(Intended) Outcome: A desired IMPACT of your office’s efforts
Action: What you do to achieve outcome
The two are often confused. However, sometimes the distinction is not clear cut.

7. Example 1 Action Intended Outcome
Two pills every four hours for three days
Intended Outcome
My temperature will drop from 101 to 98.6
Cause - Effect

8. Example 2 Intended Outcome I will lose 20 pounds Action I will
exercise 30 minutes four days a week and
eat approximately 2000 calories a day
Intended Outcome
I will
lose 20 pounds
Cause - Effect

9. Example 3 Intended Outcome
Action
Realtors will attend summer retreat on better sales tactics.
Realtors, on average, will show 50 houses per year.
The marketing office will advertise via Google, the weekly real estate digest, etc.
Intended Outcome
The real estate office will sell 10 million dollars worth of homes in 2018 (2 million more than 2017)
Cause - Effect

10. Outcome or Action?
The maintenance crew will keep the lawn mowed.

11. Outcome or Action?
The assessment director will meet with all unit coordinators on campus.

12. Example 4 (A PAC Flashback)
The year, 2008
Top Song:
Top Movie:

13. Top News Story 2008
Image From:

14. 2008 PAC FT 4-yr FTIC Graduation Rate
11.2%

15. Segue to Outcome/Action
(Intended) Outcome: The 2012 FT PAC cohort will have a 4-yr FTIC graduation rate of 22.4%. Note, 2008 rate was 11.2%
What actions did you take to get here….

16. In Business Parlance
Outcomes = LAG measures
Actions = LEAD measures

17. The Assessment Cycle 1) State Intended Outcomes 2) Map to Actions
3) Select Methods
4) Analyze and Interpret Results
5) Report to Stakeholders
6) Use Results for Improvement
I may have this slide come up between each step mentioned, with the appropriate step maximized.

18. Step 1: State Service-Level Outcome
Service-Level Outcomes (SLOs) are the desired effect your unit/program wishes to have.
All bullets at once

19. Step 2: Map Actions to Outcomes
Identify actions you intend to take to achieve desired outcome
Referred to as Program Theory
How the design of a program should theoretically affect outcomes
All bullets at once

20. Step 3: Select Methods
Select instruments and data collection strategies to gather evidence about outcomes.
First bullet and sub-points on first click, second bullet on second click

21. Step 4: Analyze and Interpret Results
What did you find, and what does it mean relative to your outcomes and actions?
All bullets at once

22. Step 5: Reporting to Stakeholders
Identify stakeholders and what they want and need to know.
All bullets at once

23. Step 6: Using Results for Improvement
Program/unit improvement based on evidence is the main purpose of assessment

24. O.k….Let’s Pivot to Assessment Reporting
Four hypothetical assessment reports; handing out one at a time; each progressively better
Report format inspired after PAC’s
Purpose: Aligned with…Distinguishing among beginning, developing, good, and advanced assessment reporting
Context:
Unit: The Health Center
Outcome: Flu shots
ACTIVITY

25. On Learning Improvement
A Pig Never Fattened Because It was Weighed

26. Fundamental Considerations of Improvement
It’s about relationships and trust
Leadership within the unit
Clarity of shared vision regarding improvement
Tight relationship between actions and outcomes
Development of staff
Assessment built around (pre and post) actions
Contextual factors
It’s about Relationships and Trust
Improvement mindset, not defensiveness
Cohesiveness of faculty
Ability to work on a common project
Leadership within the program
Champions of improvement effort
Dept. head must be supportive
Good project management
Clarity of vision and development of faculty
What EXACTLY is it that faculty want to improve
Do faculty currently have expertise to teach this area? If not, how do we help develop those skills

27. ACTIVITY Practice Scenario… Pretend you are an improvement consultant
The director of the Health Center approaches you…
You are trying to size up the program and its readiness for an improvement project.
What questions should you ask?
For example, what do you want to improve?
Work with your partner to generate questions.
10 minutes.
Let’s role play
ACTIVITY

28. Summary and Questions Outcomes for Participants
Contrast outcomes and actions in an assessment framework
Distinguish among beginning, developing, good, and advanced assessment reporting
Discuss the fundamentals of identifying problems, collecting baseline data, systematically addressing the issues, and then re-assessing to determine success

29. fulchekh@jmu.edu References
First Pig Paper:
Second Pig Paper: