1. Telling Our Story About Our Students and Engaging Others in Doing the Same
Prospect For Success Charrette was a segue to today’s discussion and activities about
-sharing the impact we have on student learning
-making linkages to concepts throughout the curriculum
Discussion with the business school student who was able to articulate the impact assignments had on him and his future
These are the type of stories we want to share that demonstrate student learning
In the world of assessment, other measures dominate the agenda
-the Feds want accreditors to choose outcomes and measure of quality for institutions
-states want to know if money given is being well spent
How do you think these stakeholders are suggesting that we assess the quality of a student learning?

2. Why we need to tell our own story about student learning?
Measures of quality are imposed
Standardized exams (comparison institutions)
Return on Investment measures (e.g., job placement and median starting salaries
Program progression measures (e.g., retention and completion rates)
Additional scrutiny is required
making student work available to outside parties
including outside parties in the review of student work
What story are we telling about how we impact student learning?
Where is this information published?

3. Voluntary System of Accountability (VSA)
Public universities to supply straightforward, comparable information on the undergraduate student experience through a common web report
Three objectives
Demonstrate accountability, transparency
Support innovation in the measurement, reporting of student learning outcomes
Use as a search tool for informational and comparison purposes

4. Student Learning Outcome (SLO) Component of the VSA
Use results from one of the required instruments to assess campus-level outcomes
ETS Proficiency Profile critical thinking and written communications sub-scores
CLA performance and analytic writing tasks
ACT CAAP critical thinking and writing essay modules
Results can be no more than three years old
Chosen outcomes (critical thinking and writing) and measure of quality for institutions
What stories did we tell about the results and use of results from the CLA in 2009 and the ETS HEIghten exam in 2015?

5. Stories Reported on the VSA
CLA longitudinal study
Results were posted but they were not useful in contributing to campus discussions about student learning outcomes.
The faculty have not found the results of the study useful in directing improvements.
ETS Proficiency Profile in 2014
Results and comparisons to participating institutions were posted to our College Portrait.
Test results were shared with the deans and department chairs with the recommendation that departments identify learning activities in the curriculum that are linked to the critical thinking competencies and provide greater emphasis or infuse additional learning activities to strengthen students' critical thinking skills.
Demonstrated accountability but was not very informative to the public about student learning
9:36 – 10:06 Spellings about UNC Charlotte (Can we excerpt this? If not, see script below.)
UNC Charlotte looks like University of Houston. A big urban university, providing opportunity in an affordable way to first-generation students. Incredible connectedness with the business community. Great base of support with non-alums because they understand what this institution means for the vibrancy of the city.

6. The Story Reported on the University’s Website
Student Achievement
Retention and Graduation Rates
Teacher Licensing Exam Results
National Survey of Student Engagement Results
Post-Graduation Survey Results
Enrollment
Another story we tell the public
Some of these measures were chosen because the are part of the federal requirement accreditation compliance.
These are limited in the way we tell our story about student learning.

7. How can we tell our story better?
Program Learning Outcome examples:
Africana Studies
College of Education
Respiratory Therapy
Computing and Informatics
University Writing Programs
Rochelle to make handout #1. John and Cathy should have examples.

8. VSA’s New Student Learning Outcome Component
SLO reporting is no longer restricted to specific instruments
The National Institute of Learning Outcomes Assessment (NILOA) Transparency Framework was adopted as the preferred reporting method
Greater flexibility in assessing student learning
Greater clarity in messaging to stakeholders
Required to report on the components of the Transparency Framework on the University’s website
After raising concerns and providing suggested changes, VSA has changed it requirements.
Not tell the story through the lens of a chosen measure
We have the opportunity to tell our story using the outcomes and measures that we value
Today’s purpose is to develop processes to help us tell our story about the quality of student learning using the NILOA Transparency Framework.

9. NILOA Transparency Framework
The NILOA Transparency Framework lays out a process for institutions to make evidence of student accomplishment accessible, useful and meaningful to various audiences. 
This overview/big picture looks similar to our current SLO assessment cycle.
National Institute for Learning Outcomes Assessment. (2011). Transparency Framework. Urbana, IL: University of Illinois and Indiana University, National Institute for Learning Outcomes Assessment (NILOA). Retrieved from:

10. Benefits of the Transparency Framework
Standardized exam is no longer required to assess campus-level outcomes
Supports regional and discipline compliance requirements for SLOs
The following are already in place:
Student Learning Outcome Statements, Assessment Plans and Activities, Evidence of Student Learning , and Use of Student Learning Evidence for Programs
Assessment Resources
NILOA framework allows us to tell our story in a way that standardized test and program progression measures do not
However, there are components of the Framework that we will have to address.

11. NILOA Framework Readiness
Dimension
Readiness to meet the Criterion
Student Learning Outcomes Statements
Campus-level Assessment Activities
Student learning outcomes statements are clearly stated
Campus-level assessment information shared with internal and external stakeholders
Student learning outcomes statements are integrated
Presentation format and accessibility of campus assessment information
Student learning outcomes statements are prominently posted and available to students
Participation in and sharing of information regarding campus-level assessment activities
Campus-level Assessment Plan
Campus-level Evidence of Student Learning
Comprehensive institutional-level assessment plan
Engagement in monitoring and compiling campus-level assessment results and analysis
Alignment between institution-level assessment plan and program-level assessment plans
Distribution and sharing of campus-level results of student assessment
Stakeholder involvement in the development and review of assessment plans
Integration of campus-level results with measures used at other levels to create complete picture of student learning
Campus-level Assessment Resources
Use of Campus-level Evidence of Student Learning
Centralized assessment resources availability and use
Integration of campus-level results with measures used at other levels to guide campus decision-making
Faculty and staff development activities to promote campus-level assessment best practices availability and use
Evidence of use of assessment results from all levels of campus provided to stakeholders
Campus policies and procedures recognizing and providing support for faculty and staff assessment activities
Communication of changes made as a result of assessment evidence from all levels of campus
Communication of outcomes from changes made as a result of assessment evidence from all levels of campus
Green = available or no outside input necessary
Light Gold = moderately difficult, as a process or collaborative efforts are needed
Dark Gold = difficult, as a process or collaborative efforts are needed
What is in green is based on program SLO plans and reports and resources in place. This is about 30%.
Why are some components orange? The following data is missing:
-Campus-level outcomes, assessment plans and activities, evidence of student learning, and use of evidence (in place of standardized exam results)
-Alignment and integration of campus-, program-, and course-level outcomes (mapping process)
-Engagement of student affairs, academic services, and external stakeholders in the process (broader participation and transparency)
-Publication of student learning components (demonstrate transparency)
Over 40% is moderately difficult to achieve (light orange)
A little over 30% will require some significant effort to achieve (dark orange)

12. Outcome Linkages
Peeling an onion with it layers but everything is connected at the base.
Will this require more work?
Yes, but a better method of addressing calls for accountability and using non-meaningful, snapshots of data.
I have presented the conceptual framework.
What does this look like in practice? Who are the petals of the onion? - Cathy will discuss
How do we make the components of the Framework all green?
-John will take us through an exercise to turn one of the components green
-Cathy will discuss the activity we have designed to generate ideas about how to address the gaps
-You are not solving the problem today but you can share with and provide guidance to those who will be responsible
The ability exists to make these linkages in the new software, Compliance Assist, using a mouse click. .

14. Why the need for campus-level outcomes?
Faculty & Staff are able to
Communicate skills to the public in terms they understand and agree upon
Help students connect the dots between courses
Make linkages between the various initiatives and their impact
Students are able to
Make connections between general education and academic major outcomes
Make connections between curricular and co- curricular outcomes
John will address along with the rank order activity and discussion

15. Prioritizing and Ranking Campus level Outcomes
Critical Thinking
2. Ethical Reasoning
3. Global Learning and Intercultural Competence
4. Information Literacy
5. Inquiry
6. Oral Communication
7. Quantitative Literacy
8. Written Communication

16. Campus-Level Assessment Plan
The plan consists of:
campus-level student learning outcomes with measures that are integrated with program- and course-level student learning outcomes and measures
an assessment plan for collecting, analyzing, disseminating, and reviewing evidence of campus-level student learning and evidence-based decision making
involvement of external and internal stakeholders in:
development of the assessment plan
review of assessment data
decisions for improving student learning
publishing to website
Cathy will address

17. NILOA Framework Readiness
Dimension
Readiness to meet the Criterion
Student Learning Outcomes Statements
Campus-level Assessment Activities
Student learning outcomes statements are clearly stated
Campus-level assessment information shared with internal and external stakeholders
Student learning outcomes statements are integrated
Presentation format and accessibility of campus assessment information
Student learning outcomes statements are prominently posted and available to students
Participation in and sharing of information regarding campus-level assessment activities
Campus-level Assessment Plan
Campus-level Evidence of Student Learning
Comprehensive institutional-level assessment plan
Engagement in monitoring and compiling campus-level assessment results and analysis
Alignment between institution-level assessment plan and program-level assessment plans
Distribution and sharing of campus-level results of student assessment
Stakeholder involvement in the development and review of assessment plans
Integration of campus-level results with measures used at other levels to create complete picture of student learning
Campus-level Assessment Resources
Use of Campus-level Evidence of Student Learning
Centralized assessment resources availability and use
Integration of campus-level results with measures used at other levels to guide campus decision-making
Faculty and staff development activities to promote campus-level assessment best practices availability and use
Evidence of use of assessment results from all levels of campus provided to stakeholders
Campus policies and procedures recognizing and providing support for faculty and staff assessment activities
Communication of changes made as a result of assessment evidence from all levels of campus
Communication of outcomes from changes made as a result of assessment evidence from all levels of campus
Green = available or no outside input necessary
Light Gold = moderately difficult, as a process or collaborative efforts are needed
Dark Gold = difficult, as a process or collaborative efforts are needed

18. NILOA Transparency Framework Gaps
Dimension
NILOA Criterion Statement
Breakout Group
Student learning outcomes statements are integrated
Student learning outcomes statements at the campus-level are integrated with student learning outcomes statements from the college-, program-, and course-level.
Team #1
Comprehensive institutional-level assessment plan
The institution has a comprehensive assessment plan at the institution-level that includes common assessment activities for all students designed to provide evidence of student learning across campus.
Participation in and sharing of information regarding campus-level assessment activities
Faculty, staff, students, and external stakeholders such as employers or graduate school admissions professionals actively participate in the decision processes related to campus-level assessment.
Team #2
Engagement in monitoring and compiling campus-level assessment results and analysis
Faculty, staff, students, and external stakeholders such as employer or graduate school admissions professionals participate in the monitoring and/or compiling of campus-level assessment results and analysis.
Integration of campus-level results with measures used at other levels to guide campus decision-making
Campus-level assessment results have been considered in combination with assessment results at other levels of the institution to guide decision-making related to changes in policies and practices that may lead to improved student learning.
Campus policies and procedures recognizing and providing support for faculty and staff assessment activities
Campus policies and procedures regarding faculty and staff review processes (e.g., tenure and promotion, performance reviews, etc.) are structured to provide support and/or recognition for faculty and staff working to improve or advance their assessment practices.
Alignment between institution-level assessment plan and program-level assessment plans
Institutional-level assessment is integrated with and scaffolds from program-, course-, and student-level assessment data. The review processes from all levels of assessment provides supporting evidence for all learning outcomes at the institutional-level.
Team #3
Stakeholder involvement in the development and review of assessment plans
Stakeholders from programs and departments, including student affairs and students, are involved in the development and review of assessment plans, results, and recommendations.
Cathy to explain group assignments, team member roles,
Why external stakeholder participation? Include the perspective of alumnae performance from those who hire our students or accept them into graduate school.
Why include other internal stakeholders? Units not in the Colleges also support the student learning process.

19. Breakout Groups/Lunch/Report Out
Team 1 - (Salon E) Integration of campus-level outcomes and developing a campus-level assessment plan   
Team 2 - (Salon C) Involving external partners in campus-level assessment processes and changes to internal policies and practices
Team 3 - (Salon D) Identifying campus-level measures and a review process and involving internal stakeholders

20. Publishing Student Learning Outcome Assessment Plans
Dimension
NILOA Criterion Statement
Short Description
Campus-level assessment information shared with internal and external stakeholders
Campus-level assessment information relating to assessment plans, processes, and results are available to and proactively shared with a variety of on-campus and off-campus stakeholders.
Assessment plans, processes, and results
Evidence of use of assessment results from all levels of campus provided to stakeholders
Stakeholders from both inside and outside the institution are regularly provided with evidence that institutional decision-making is appropriately guided by assessment results from multiple levels of campus assessment.
Linkages of decision making to assessment results
Communication of changes made as a result of assessment evidence from all levels of campus
Changes made as a result of assessment results are communicated to both internal and external campus audiences, including students. Communications include information on evidence supporting the need for change.
Implemented changes
Communication of outcomes from changes made as a result of assessment evidence from all levels of campus
Changes made as a result of assessment results are monitored and evaluated. Outcomes from changes are communicated to both internal and external campus audiences, including students. Communications include information on evidence used to evaluate the change.
Improvements based on changes
Distribution and sharing of campus-level results of student assessment
Campus-level assessment results are shared proactively with faculty and staff on campus in ways that facilitate their discussion and application (e.g., not simply made available as a report on a website).
Communication Methods
Presentation format and accessibility of campus assessment information
Campus-level assessment information is presented in easily accessible formats that are understandable to a variety of audiences both on and off campus. The presentation of assessment information is intentionally designed to promote appropriate use and interpretation of results.
AM session developed the processes for
-Identifying campus-level outcomes and assessment cycle
-Alignment and integration of campus-, program-, and course-level outcomes
-Engagement of student affairs, academic services, and external stakeholders in the process
Once we have completed all of the components of the Framework, we will publish the story of student learning.
PM session will address how we publish the information we already collect about program learning outcomes to help tell our story.
We currently have only the outcomes by program posted

21. UNC Charlotte’s Proposed Framework Model
Student Learning Outcomes Statements
Assessment Plans
Evidence of Student Learning
Use of Student Learning Evidence
Assessment Resources
Accreditation
Using the NILOA Transparency Framework as a model, we have developed a UNC Charlotte model. These components must be published.
We have components from the the SLO plans and reports that match the Framework.
Review Respiratory Therapy as an example of what is available which includes student expectations, curriculum impact, and program change and improvement. Can publish these as they are.
Review BYU example of how the information might be published differently.
Differences:
BYU report includes courses that support each SLO and the areas of improvement are broader than one SLO.
UNC Charlotte report includes specific results using course name and number and specific performance targets and achievement of those targets.
All of the components from the Framework must be included but we can determine how to present the information and what level of detail we provide
Using the two models, discuss what details should be included on the website and what details cause you concern if published on the website.

22. Report Outs
Compare and contrast the UNC Charlotte plan and report with the BYU example.

23. Next Steps: Submission of SLO Plans and Reports
Some programs lack documentation on:
The annual faculty review of student learning outcomes assessment data
Changes made based on evidence of student learning
Impact of the prior year’s changes on student learning

24. Follow-Up to Assessment Retreat
Encourage departments to devote at least one spring department meeting to the review of assessment data
Appoint a scribe, could be the College Assessment Coordinator to document the department review of assessment data
Encourage departments to document evidence-based changes in SLOs Assessment Plans and Reports

25. Examples of Evidence-Based Changes
Curricular Changes: Added or revised a course(s), changed course sequence, developed a curriculum map
Course Revisions: Added or revised course content, changed the textbook
Pedagogy Changes: Revised methodology of delivering course materials, integrated technology
Revised Assessments: Changed or modified SLO(s), changed or modified assessment tool(s)

26. Examples of Evidence-Based Changes (cont.)
Changed Criteria: Modified rubric criteria, increased Performance Outcome
Revised Policies: Changed entrance requirements
Professional Development: Provided faculty (or TA) with training
Budget Decisions: Requested additional resources

27. Wrap Up All Plans and Reports are due May 31st
Student Learning Outcome information from Plans and Reports to be posted on the website in Fall 2016
Complete all Transparency Framework components by March 1, 2017

28. Questions

29. Sources of Campus Level Outcomes
Adapted from Susan Hatfield , Ph.D., Presentation to 2010
Written Communication
Campus-Level Learning Outcomes
Oral Communication
Critical Thinking
Quantitative Reasoning
Co-curricula Outcome
Information Literacy
Co-curricular Outcome
Co-curricular Outcome
General Education
Program Learning Outcome
Selection of some appropriate (not all) College Wide Learning outcomes combined with discipline specific outcomes
Program Learning Outcome
Program Learning
Program Learning Outcome
Co-curricular Outcome
Integrative Learning
Program Learning Outcome
Program Learning Outcome

30. Publication Concerns
Program specific data such as course name and number (anonymity)
Impact of results on annual evaluations and course quality (punitive consequences)
Performance target and % of students achieving the target (perceived low expectations of students)
Others?

31. Voluntary System of Accountability Origins: Spellings Commission Report
“To meet the challenges of the 21st century, higher education must change from a system primarily based on reputation to one based on performance.”
“We urge the creation of a robust culture of accountability and transparency throughout higher education. Every one of our goals, from improving access and affordability to enhancing quality and innovation, will be more easily achieved if higher education institutions embrace and implement serious accountability measures.”
Source: A Test of Leadership: Charting the Future of U.S. Higher Education A Report of the Commission Appointed by Secretary of Education Margaret Spellings
This is why this story was told the way it was.