1. Farris Child Dan Chandler
Transforming Assessment With Direct Evidences of Learning Brigham Young University
Farris Child
Dan Chandler
Introductions FF

2. Session Outcomes Attendees will be able to:
Describe the differences between direct and indirect evidences of learning.
Identify instances where either direct or indirect evidences would be appropriate.
Explain why direct evidences of learning are important.
Show how changing the assessment of an outcome can lead to direct verses indirect evidence.
How many of us are involved with advisement assessment?
How many of us are currently assessing outcomes?
Who already knows the difference between direct and indirect evidences of learning?
Who is using both evidences with their advisement assessment?
Who isn’t sure what we’re talking about?
Two fold reason for these questions
1. To help us get started
2. As an example – indirect assessment tool gathering indirect evidence
Those who raised your hand – DO I have any proof
I can assume that most likely yes- but I don’t know for sure
One of you who raised your hand- Could you please tell me what is
the difference between direct and indirect
Example of Direct
Seasoned Assessor – Consider Direct Evidences of Learning
Beginning Assessor – Considerations and some ideas of how you can assess your outcomes FD

3. Summary of Indirect and Direct Evidences of Learning
Capture perceptions or opinions
Capture signs that students/ advisors are probably learning or have the skill sets being assessed
Less clear and less convincing
Requires those being assessed to demonstrate their knowledge or skills
Tangible, visible, self explanatory evidence of what has and hasn’t been learned or what has or hasn’t been done
Clear and convincing
Suskie (2009) defines indirect evidences of learning as, “evidence consisting of proxy signs that students are probably learning”.
Aiken-Wisniewski et. al. (2010) states that. Indirect measures (another word for evidences of learning) often capture the opinions and perceptions about student learning or process outcomes but are not used to directly measure the desired outcome.
Suskie defines direct evidence of learning as, “Tangible, visible, self- explanatory, and compelling evidence of what exactly students have and haven’t learned.”
Aiken-Wisiewski et.al explain that, “Direct measures capture student learning directly through concrete behavioral expressions of the desired outcome. They prompt students to demonstrate their knowledge or skills.” DD

4. Examples of Direct and Indirect Evidences of Student Learning Outcomes
Surveys
Questionnaires
Interviews – asking questions which responses can’t be validated
Questions asking if a student has the skills or knowledge regarding an outcome (Yes / No type questions)
Direct Observations
Pre test / Post test
Evidence we can see which clearly demonstrates skill sets or knowledge = no doubt
Questions asking the student to explain, identify, define, list, etc.
Possibly multiple choice questions (depends on how the outcome is written)
Examples (PDOS
Indirect
Focus Groups
Usage Statistics
Surveys & Questionnaires
Direct
Internal review- observations
Performance on case student problems
External review by qualified individuals
*Aiken-Wisniewski et.al. (2010)
+After discussing this slide have audience explain to their neighbor the difference between direct and indirect evidences of learning-
OBSERVE- As I’m observing this is a direct evidence of learning DD

5. Examples of Indirect and Direct Evidences of Outcomes
Ask advisors if they participated in professional development
Or ask how many hours of professional development they acquired.
Look at an advisor’s transcript, signed CEU form, certificate of attendance, etc.
Please see you handout for more specific definitions DD

6. Examples of Indirect and Direct Evidences of Outcomes
Use a Likert scale asking students the extent to which they know what needs to be done in order to get off academic probation
Have the student explain to their advisor how to get off academic probation
OR ask a student to correctly identify which statement describes how to get off academic probation (out of several options)
OR ask the student to explain how to get off academic probation on a survey with an open ended question
Please see you handout for more specific definitions DD

7. Examples of Indirect and Direct Evidences of Outcomes
Ask student if they know what they need to do in order to move forward with their career goals
Advisors use a rubric with defined levels identifying how prepared the student is with their career goals (graduate school, internship, and employment preparedness, etc.)
Ask student to list the next steps for their career goals
Please see you handout for more specific definitions DF

8. How do you decide which tools to use?
Focus Groups
Questionnaires
Interviews
How do you decide which tools to use?
Direct
Observations
Pre-test/post-tests
Surveys FF

9. Importance of Direct Evidence
Fusch (2013) suggests we should move beyond surveys by having students create learning products which allow us to form a more direct assessment of what students have learned
Campbell (2005) states, “We must gather evidence from multiple sources. Evidence must reflect both direct and indirect measures, and be both quantitative and qualitative.”
Suskie (2009) proposes that, “No assessment of knowledge, conceptual understanding, or thinking or performance skills should consist of indirect evidence alone.”
Both indirect and direct assessment methods are important.
I propose that if we have evidence from both types- stronger more viable assessment portfolio
Much of advisement assessment tends to be more indirect
Stronger evidence - FF

10. Transforming Indirect to Direct Questions
Indirect – Ask yes/no, Likert type, etc. questions
Direct – Ask students to identify, locate, interpret, etc.
OR- use open ended questions requiring the student to explain, list, recall, describe, etc.
Direct- can be more difficult to go through all the open ended responses but can be a great evidence.
If you would like more the direct, but without going through the responses asking students to identify, locate, interpret, etc. may work depending on the outcome FD

11. Question Type Examples
Indirect –
Outcome = Students can identify what they must do in order to get off probation
Question = Do you know how to get off probation (Yes or No)
OR- Please indicate the level to which you understand what you need to do in order to get off probation
Direct
Question = How does a student get off probation? (open ended or have them find and select the correct response out of several possibilities).
Because the outcome key word is identify – Have a multiple choice question with several possibilities and one correct option. If the student can identify the correct option then this would be an example of a direct evidence.
Direct ?
Outcome = Students can explain / identify what they must do in order to get off probation DD

12. Question Type Examples
Indirect –
Outcome = Students can find their progress report
Question = Do you know where to go to find your progress report?
Direct
Question = Which of the following steps (in order) would you take in order to access your progress report? (This question would list several possibilities and require the student to identify which steps they would take.)
OR we could ask the student to, “Please show me how to find your progress report.”
Indirect = They are telling us they can but we can’t prove it- no concrete evidence
Direct= They had to correctly identify or explain
Direct
Outcome = Students can find their progress report DF

13. Challenge
Think about a Student Learning Outcome or Process and Delivery Outcome that you trying to assess:
How could we gather direct evidence for this outcome?
How could we gather indirect evidence for this
outcome? FF

14. References
Aiken-Wisniewski, S., Campbell, S., Higa, L., Kirk-Kuwaye, M., Nutt, C., Robbins, R. & Vesta, N. (2010). Guide to Assessment in Academic Advising Second Edition. Monograph Series Number 23. National Academic Advising Association.
Suskie, L. (2009). Assessing Student Learning. 2nd Ed. San Francisco, CA. John Wiley & Sons, Inc.
Campbell, S. (2005, December). Why do assessment of academic advising? Academic Advising Today, 28(4). Retrieved from Academic Advising Today: Do-Assessment-of-Academic-Advising.aspx
Fusch, D. (2013). Assesing Student Learning Outcomes: Surveys Aren’t Enough. Retrieved from Academic Impressions Web Site: surveys-arent-enough?qq=18798a430987mW
Robbins, R. & Zarges, K.M. (2011). Assessment of Academic Advising: A Summary of the Process. Retrieved from NACADA Clearinghouse of Academic Advising Resources Web Site:  of-academic-advising.aspx