1. WASC Evaluator Workshop Spring Visits 2010

2. Workshop Outcomes
Know how to prepare for and conduct an effective visit and produce a useful, high-quality team report
Be prepared to make sound judgments about institutions under the Standards
Be familiar with resources that support your work on a team

3. Agenda Context for the Visit/Accreditation Preparing for the Visit
Conducting the Visit
Developing Team Recommendations
Writing the Team Report

4. Context for Accreditation and Visits
The Continuing Evolution of the WASC Process and Standards
The Accountability Movement
Retaining Peer Review
The Impact of the Economy
Value Added, Collaboration, and Ongoing Efforts to Refine and Improve

5. Recent Changes in the Institutional Review Process and Standards
Implement 2009 changes to Institutional Review Process re: Student Success, Program Review and EE Sustainability
Implement 2009 changes to CFRs
Clarify the scope of the CPR visit to review the “infrastructure” for assessment of student learning
Examine Program Review and Program-Level Student Learning in a systematic way
Allow teams more time together on visits
Tools: Table A (RB pg. 41); Table B (RB pg. 47)

6. THE THREE-STAGE REVIEW PROCESS

7. THE THREE-STAGE REVIEW PROCESS
Institutional Proposal
Capacity and Preparatory Review
Educational Effectiveness Review
This applies to all institutions, regardless of where they are in the accreditation cycle.
This is intended to be for a maximum of 10 years.

8. TIMELINE FOR THREE-STAGE REVIEW PROCESS

9. INSTITUTIONAL SELF-REVIEW
The heart of accreditation
Built upon an effective internal process of
Evaluation
Reflection
Recommendations
Plans for Action

10. OUTCOMES OF THE ACCREDITATION PROCESS FOR THE INSTITUTION-1
Validation of institution’s presentation of evidence of compliance and improvement
Effective use of indicators of institutional performance and educational effectiveness
Greater clarity about institution’s educational objectives
Improvement of institution’s capacity for self-review and quality assurance

11. OUTCOMES OF THE ACCREDITATION PROCESS FOR THE INSTITUTION-2
Deeper understanding of student learning
Use of assessment results to improve program and institutional practices
Systematic engagement of the faculty on issues of assessing and improving teaching and learning processes

12. THE INSTITUTIONAL PROPOSAL

13. THE ROLE OF THE INSTITUTIONAL PROPOSAL
Guides the entire accreditation review process
Connects institution’s context and priorities with the Standards of Accreditation
Provides primary basis for both institution self-review and team evaluation
Allows alignment of accreditation activities to institutional strategic plan and key areas chosen for improvement

14. THE INSTITUTIONAL PROPOSAL-1
Section 1
Sets the institutional context
Relates the proposal to the Standards
Describes how the proposal was developed
Describes extent to which the process generated broad institutional support

15. THE INSTITUTIONAL PROPOSAL-2
Section 2
Frames the review process to align the two reviews
Describes a coherent vision and specific outcomes for the entire review cycle as a single connected process
Specifies what institution intends to accomplish and how the CPR and EER are aligned to achieve those outcomes, framed as results not as activities to be undertaken

16. THE INSTITUTIONAL PROPOSAL-3
Section 3
Demonstrates a feasible work plan and engagement of key constituencies
Includes:
A work plan and milestones
The effectiveness of data gathering and analysis systems
A commitment of resources to support the review

17. THE STRUCTURE OF THE INSTITUTIONAL PROPOSAL-4
Section 4
A set of Appendices that include:
Data exhibits
Information about off-campus and distance education degree programs
Institutional stipulations signed by the Chief Executive Officer

18. THE INSTITUTIONAL LETTER OF INTENT
Submitted by institutions seeking Candidacy or Initial Accreditation, the LOI serves the same purpose as the proposal
Includes suggestions from Eligibility Approval Letter
Submitted to Assigned WASC Liaison, one year in advance of Capacity Review
Instructions are in “How to Become Accredited” on WASC website

19. THE CAPACITY AND PREPARATORY REVIEW

20. PURPOSE OF CAPACITY AND PREPARATORY REVIEW
Review and verify the information in the institutional presentation (report and data)
Evaluate key institutional resources, structures, processes in light of Standards
Evaluate institution’s infrastructure to support student and program learning
Assess institution’s preparedness to undertake Educational Effectiveness Review

21. THE EDUCATIONAL EFFECTIVENESS REVIEW

22. PURPOSE OF EDUCATIONAL EFFECTIVENESS REVIEW
Invite sustained engagement by the institution on the extent to which it fulfills its educational objectives
Enable the Commission to make a judgment about extent to which institution fills its Core Commitment to Educational Effectiveness

23. THE TWO CORE COMMITMENTS

24. CORE COMMITMENT #1
“The institution functions with clear purposes, high levels of institutional integrity, fiscal stability, and organizational structures to fulfill its purposes.”

25. CORE COMMITMENT #2
“The institution evidences clear and appropriate educational objectives and design at the institutional and program level. The institution employs processes of review, including the collection and use of data, that ensure delivery of program and learner accomplishments at a level of performance appropriate for the degree or certificate awarded.”

26. THE FOUR STANDARDS

27. Institutional Purposes
STANDARD 1: Defining Institutional Purposes and Ensuring Educational Objectives
Institutional Purposes
Integrity

28. STANDARD 2: Achieving Educational Objectives Through Core Functions
Teaching and Learning
Scholarship and Creativity
Support for Student Learning

29. STANDARD 3: Developing and Applying Resources and Organizational Structures to Ensure Sustainability
Faculty and Staff
Fiscal, Physical, & Information Resources
Organizational Structures & Decision Making Processes

30. STANDARD 4: Creating an Organization Committed to Learning and Improvement
Strategic Thinking and Planning
Commitment to Learning and Improvement

31. POLICIES MANUAL AT

32. INTERIM REPORTS AND SPECIAL VISITS

33. INTERIM REPORTS AND SPECIAL VISITS
May or may not be connected to a sanction
Intended to monitor institutional issues identified by Commission or to assess how institution will move into compliance
Both focus on only a few specified areas of concern
Interim Reports are reviewed by a panel of peer reviewers (IRC)
Special Visit length and number of team members are determined by number, depth, and complexity of issues

34. Expectations for Two Reviews
Capacity and Preparatory
Preparatory = readiness for the Educational Effectiveness Review
Capacity = purposes, integrity, stability, resources, structures, policies, processes
Educational Effectiveness
Demonstrating student learning
Demonstrating institutional learning
Demonstrating evidence-based decision-making
Tool: Expectations for Two Reviews (RB pg. 20)

35. The CPR and EER as a Whole
The CPR evaluates what an institution has for infrastructure (staff/faculty, resources, processes, facilities, systems, structures).
The EER evaluates how well that infrastructure works and the results that the institution achieves.

36. Navigating Multiple Purposes-1
Focusing on the institution 
Applying the Standards and CFRs
Focusing on Proposal themes/topics
Evaluating capacity and effectiveness under Standards; addressing team-identified issues
Reviewing the whole institution
Focusing on specifics, e.g., distance education, samples of program reviews
Advancing institutional development
Addressing “compliance” matters

37. Navigating Multiple Purposes-2
Allowing flexibility and experimentation on visit 
Ensuring consistency and fairness among visits and quality control of visits and reports
Using the CPR to evaluate EE readiness
Leaving evaluation of educational effectiveness until EER
Supporting institutional creativity and excitement
Reporting to the Commission and serving the public

38. Covering the Impact of the Financial Recession on Institutions
Questions to ask the institution:
How has the financial recession affected your institution?
How has your institution responded?
What plans are in place in case the current state of affairs becomes permanent?

39. Preparing for the Visit (Visit Guide, Part II, pp. 29-52)

40. Timeline For CPR/EER Reviews
12 weeks months
Institution mails report to team and WASC
Team holds conference call
Site visit held and team report written
Institution responds to errors of fact in team report
Institution responds to final team report
Commission acts at February or June meeting
Tool: CPR or EER Timeline (VG, pg. 29)

41. Roles and Responsibilities of Team Members and Staff
Role of Team Chair (RB pg. 193)
Role of Team Assistant Chair (RB pg. 195)
Role of assigned WASC staff liaison (VG pg. 7)
Team assignments

42. Pre-visit Preparation
Read all the documents from WASC
Standards, CFRs, policies, visit guide, rubrics
Background documents re: institution and purpose of the visit, including Proposal and/or last action letter/team report
Read the institutional report
Review the data portfolio and exhibits
What to look for and highlight?
Tools: Timeline (VG pg. 8, VG pg. 29)

43. Reviewing the Exhibits
Enrollment data
Headcounts and FTE
Graduation data
Faculty data
Key financial indicators
Inventory of Educational Effectiveness Indicators
Inventory of Concurrent Accreditation and Key Performance Indicators
Tool: How to Review WASC Data Exhibits (RB pg. 61)

44. Reading the Report
Has the institution done what it said it would do in its Proposal?
Has it collected and analyzed data effectively?
Are its conclusions supported by evidence?
Are there serious problems or potential areas of noncompliance?
Does the report contain recommendations for further institutional action?

45. Worksheet for Team Conference Call
Organizes team’s responses to institutional materials
Helps team make preliminary evaluation under the Standards
Provides basis for team to work toward consensus
Should be submitted in advance of call
Tool: Team Worksheet (VG pg. 42)

46. Team Conference Call
Evaluates quality of institutional report and alignment with Proposal and previous action letter(s)
Identifies areas of good practice, improvement, and further inquiry
Identifies issues, strategies, evidence needed
Identifies persons and entities to be interviewed
Makes or refines team assignments
Plans visit logistics

47. Off-Campus Sites and Distance Education Programs (special requirement for some visits)
Prior to Visit: Sites will be identified and assignments made
Review substantive change action letters to determine if issues have been identified
Develop plan for the review of the programs and/or sites
During Visit
Interview faculty, administrators and students
Evaluate facilities OR online infrastructure
Observe classes
Document visit and findings in appendix
Discuss important findings with team for inclusion in report, as appropriate
Tools: Protocols (RB pg. 157, RB pg. 162)
Forms (RB pg. 55, RB pg. 58)

48. Compliance Audit (special requirement for some visits)
Required for:
Institutions seeking Candidacy and Initial Accreditation
Some institutions under sanction
Additional report submitted by institution in advance of the visit—with links to documents
Tool: Compliance Audit Checklist (RB, pg. 51)

49. Determining Strategy for CPR Visit
What evidence is provided to show capacity and readiness for EE?
Why was it chosen?
What are the strengths and weaknesses of the evidence?
What other evidence do you want to review to evaluate capacity and preparation for EE?
Do any issues arise with regard to the Standards?
Meetings: format/methodologies

50. Determining Strategy for EER Visit
What evidence is provided to show EE?
Why was it chosen?
What are the strengths and weaknesses of the evidence?
What other evidence do you want to see to evaluate effectiveness?
Do any issues arise with regard to the Standards?
Meetings: format/methodologies

51. Drafting in Advance of the Visit
Assistant Chairs draft outline of team report and Section I
Team members draft outline or text for which they are responsible, using institution’s report and data portfolio, with space for additional evidence, analysis and conclusions
Tool: Team Reports (VG pg. 54)

52. Team Executive Session
Discuss preliminary findings
Identify major issues for exploration
Refine lines of inquiry
Confirm team assignments
Discuss use of tools and rubrics
Review draft team report
Discuss options for confidential team recommendation
Review schedule

53. Conducting the Visit

54. Process of Visit
Team meets at start of visit to confirm roles, assignments, logistics, and agenda
Team meets frequently re: observations, emerging recommendations, and issues
Team members complete draft sections of report and turn them in to Assistant Chair on the last day
Team agrees on report recommendations and confidential recommendation to Commission

55. Visit Schedule Executive sessions and debriefings with team only
Meetings and interviews with key individuals and groups
Open meetings with students, faculty and staff
Document review
Time for drafting report sections
Final exit meeting
Tool: Sample Visit Schedule (VG, pg. 40; SV, pg. 68)

56. Confidential Email Account
Set up by WASC as extension of open meetings
Checked by Assistant Chair during visit
Important s shared with team and investigated
Comments included in team report only if the institution has a chance to address them
Tool: Sample Notification re: Confidential Account (RB, pg. 151)

57. Approaches Used on Visits
Document review
Interviews and meetings
Mini-questionnaires
Techniques for small and large meetings
Fishbowl exercises
Audits
Plan visit methodologies in advance
as part of schedule.

58. Document Review DO as much as possible in advance: Use to:
Check compliance
Evaluate the level of institutional engagement
Examine the evolution of a policy or process
Identify direct and indirect evidence of student and organizational learning
Confirm report claims

59. Interviews Use to: Gather information Explore issues
Build relationships with members of the institution
Validate impressions and observations

60. Tips for Good Interviews
Decide on a protocol for interview
Prepare questions and lines of inquiry in advance
Ask questions that elicit information, stimulate discussion, or require judgment
Avoid interrogation, leading questions, or loaded language
Avoid consultation, giving solutions, or talking about your institution
Let them do the talking

61. Alternative Forms of Interview
Fishbowl
Brainstorm/free discussion on a salient topic
Go-round
Bundling
Audit

62. Evaluating Program Review and Student Learning on EER Visits
Tool: Suggested Approaches for Evaluating Program Review (RB, pg. 156)

67. Rubrics re: Assessment of Student Learning
Academic Program Learning Outcomes
Use of Portfolios in Assessing Program Outcomes
Use of Capstones in Assessing Program Outcomes
Integration of Student Learning Assessment into Program Review
General Education Assessment Process
Tool: Rubrics (VG, pp )

68. Educational Effectiveness Framework
Use with team to evaluate institution’s “place”
Use language of rubric to describe the institution in the report
Ask the institution to evaluate itself and discuss
Confer with team toward end of visit to mark a copy of the EEF
Submit the marked EEF confidentially to WASC
Tool: Educational Effectiveness Framework (RB, pp )

69. The Exit Meeting
Team chair communicates commendations and key recommendations that will be included in report
Chair may ask team members to participate
The meeting is not a dialog, discussion or debate

70. Developing Team Recommendations

71. Two Kinds of Recommendations
Team recommendations at the end of team report, delivered at the exit meeting
Confidential Team Recommendation to the Commission for action
Tools:
Commission Decisions on Institutions (Visit Guide pg. 89; SV Guide pg. Append. F)
Commission and Team Decision Matrix (RB pg. 177)

72. Team Report Recommendations
Should be:
Overarching and important
Supported by evidence
Linked clearly to Standards and CFRs
Supported by text in the report
Distinguish recommendations from suggestions and observations embedded in the report
Tool: Educational Effectiveness Framework (RB pg. 160)

73. Confidential Recommendation to Commission-1
CPR
Proceed to EER or reschedule EER visit
Conduct a Special Visit (not preferred)
Add time to EER visit
Issue a notice of concern or impose a sanction (rare)
Tool: Confidential Team Recommendation Form (VG pg. 71)

74. Confidential Recommendation to Commission-2
EER
Grant Candidacy, Initial Accreditation or Reaffirmation of Accreditation for specified term
Sanction or Notice of Concern
Interim Report or Special Visit
Tool: Confidential Team Recommendation Form
(VG pg.71)

75. Confidential Recommendation to Commission-3
Special Visit
Varies with status of institution
Next steps
Removal or continuation of sanction (note two-year limit on sanctions)
Tool: Confidential Team Recommendation Forms (SV Guide, pg. 48)

76. Producing Effective Team Reports

77. Report Preparation Logistics
Follow report template
Start writing before the visit
Complete your sections on site and give to Assistant Chair for editing together
Tool: Team Report Templates (VG, pg. 55; SV Guide, Appen. K)

78. Using Evidence in Team Reports
Use qualitative and quantitative evidence
Select evidence carefully and purposefully
Connect evidence to an assertion or question
Analyze information; do not just set forth data
Let evidence suggest improvements
Use evidence that speaks to the institution’s themes and the team's questions

79. Team Use of the Standards and CFRs
Team judgments must be linked to specific Standards and CFRs
CFRs must be cited in reports
Standards and CFRs form the basis for Commission decisions
Standards and CFRs provide a context for continuous quality improvement

80. What is an effective team report?
Reflects a thorough assessment of the institution’s capacity, preparation, and/or effectiveness
Is evidence based
Cites the Standards and CFRs
Provides the basis for a sound and supportable Commission decision
Identifies important areas for institution to address

81. New requirements in the Institutional Review Process
Institutions will cover the following in their reports:
Student Success (at CPR and EER)
Program Review (at EER)
Sustainability of EE (at EER)
Teams should address in the team report
Tool: Table B (RB pg. 47)

82. Tips for Writing Team Reports
Consider multiple audiences: institution, Commission, and next team
Know your areas of responsibility, including length and depth of your section
Start writing before you arrive on campus
Address priorities and goals set by the institution
Address Commission’s concerns (last action letter)
Make commendations, but don’t overdo it
Use praise that doesn’t send wrong or mixed signal

83. More Tips on Team Reports….
Be sure to check facts
Support findings and recommendations with evidence --and tie them to CFRs
Ensure evidence is sound and valid
Distinguish recommendations from suggestions or observations
Use formal language and tone (e.g., not “we/they”)
Don’t mention personnel by name
Don’t prescribe solutions

84. After the Visit

85. What happens next?
Assistant Chair prepares draft for Chair, team and staff review; changes as needed
Chair sends to institution for corrections of fact
Chair finalizes draft and submits to WASC
Chair sends Confidential Team Recommendation and completed EEF to WASC
WASC sends report to institution

86. Then…
Staff prepares draft action letter, which is reviewed by team Chair
Commission Panel reads report and documentation including institution’s written response, meets with institutional representatives at Commission meeting
Panel makes recommendation to Commission, and Commission acts
Staff finalizes draft action letter on behalf of Commission

87. Also after the visit….
Team members send reimbursement forms to WASC within 30 days
Hotel arranged and paid directly by institution
Travel / food reimbursed
Rental car must be approved in advance by WASC staff
Spouse or assistant costs not covered
See policy for more details
Team members should not have any contact with the institution
About the visit OR
Consult with the institution for one year

88. The Team’s Impact Peer review is the foundation of accreditation.
The team report forms the basis for the Commission action and its letter.
The team report and action letter inform the work of the institution for years to come.
Why were you chosen for a team?

89. Resources for Teams Appendices of Visit Guide
Team Materials and Institutional Report mailed weeks in advance of visit
WASC Website:
WASC Advisory (sent prior to visit)
WASC Staff

90. for your service to the region
Thank you
for your service to the region

91. Announcements
The materials presented during this webinar and a recording of this session will be posted at: