Implementing Peer Review in Student Affairs Annual Planning at an MSI Doctoral Research University J. Judd Harbin, Ph.D. University of Nevada, Las Vegas Monday, March 5, 2018 Convention Center, 116

Introduction Time zones photo source: https://www.flickr.com/photos/rrosie/3232834348

5:00 a.m. for me Coffee image source: https://pxhere.com/en/photo/555842

Parameters Focusing on review of annual planning at the department-level Peer review refers to internal process within the division

Peer Review Audience Participation What comes to mind? History Strengths Limitations

Overview of Creating Peer Review at UNLV Prior to 2014 2014-15 Planning Cycle Collaboration with Assessment Committee and Leadership Group 2015-16 Implementation 2016-17 Refinement 2017-18 Refinement

Prior to 2014 Planning Definitions Planning Template (PDF) Campus Life Pillars (long-term aspirational pursuits) Aligned departmental goals with pillars Aligned annual “strategies” with departmental goals Supervisor Review

Pillar Dept Goal Unit Goal Strategy Immediate Term (0 – 3 years) Aspirational Dept Goal Long Term (5 to 10 years) Unit Goal Shorter Term (3 to 5 years) Strategy Immediate Term (0 – 3 years)

Components of a Strategy Strategy Statement Leadership (Assignment) Timeline Budget Assessment Plan Metrics Learning Outcomes

Coordinator Supervisor Department Director Associate VP

Director of Assessment Associate VP Director of Assessment

2014-15 Review Enter new Director of Assessment in Campus Life (me) AVP advised me to review all strategies to gain familiarity Bonus: Provided baseline estimate of planning skill

2014-15 Review: Solo Review Using existing template and definitions, For each strategy submitted, Did the staff member provide adequate information for each section?

Ratings Each Component Each Strategy Accept as Written Accept with Minor Edits Revision Needed Incomplete Each Strategy Overall rating matched the lowest Section Rating

Audience “Incomplete” Why do these pose a special problem? Often had good information in many if not all completed sections BUT missed one or more sections

Results (2014-15) Strategies reviewed: 150 Portion Incomplete: 8.7% Count: 13 Portion Accepted: 26.7% Count: 40 Red X source: https://pixabay.com/en/cross-mark-red-sign-icon-mark-304374/ Green check source: https://pixabay.com/en/check-mark-tick-mark-check-correct-1292787/

Director of Assessment Coordinator Supervisor Department Director Associate VP Director of Assessment

Collaboration with Leadership & Staff Summarizing results of and my notes from solo review What kinds of feedback have staff had in the past? What kinds of feedback would be useful going forward? How to improve the quality of plans developed and submitted?

Enter Assessment Committee

Composition of Committee Departmental Representatives Residential Life Student Union Campus Recreational Services Facilities (including technology) Student Engagement & Diversity Most reps had served since 2012 Now entering 3rd year

Role of Assessment Committee Consider the results of my review of the annual plans Discuss with their directors and departmental colleagues What hindered quality of plan in the past? What will help improve quality of plans in the future? Identify and implement staff development plan to improve planning.

Audience What do you think the committee representatives heard from their directors and fellow departmental staff? Lack of feedback about prior plans Wondering if anyone ever read their plans Unsure how much detail to include

Committee’s Staff Development Plan Create guidelines for self-evaluation Use director’s notes Identify the most common mistakes or omissions Develop questions to help staff gauge level of detail needed Annual reminder training for all staff during July Provide feedback to staff every year

Coordinator Supervisor Department Director Assessment Review Associate VP

Enter Peer Review

Director Initiated Participated in peer review of academic program assessment plans Collegial / Collaborative peer review process Eager even enthusiastic committee response at the prospect of peer review! AND… they had questions

Committee Collaboration What would the criteria be for peer review? How many strategies do we review each year? Does everyone on committee review every strategy? Do we review every Campus Life pillar every year? Do we review every department every year? When do we do this? How do we do this?

Decisions Use the guidelines for self-evaluation as frame peer review criteria Review all departments each year Review all pillars each year Two day retreat Divide time proportionately among departments Depending on number of strategies submitted each year Not enough time to peer review every strategy Use feedback on the reviewed strategies to revise/update the other ones

Timeline: July Staff members begin drafting their plans for the year.

Timeline: August Staff members review draft plans with their supervisors and directors. Committee Orientation Welcome new members to committee Review self-evaluation guide Review peer review form Discuss process Schedule departmental feedback sessions during late September By 31st – Draft plans are due

Timeline: September Peer Review Retreat ASAP after Labor Day Two full days Prepare one-page summary for each department Complete ten (10) departmental feedback sessions Negotiate timeline for final plans from director

Timeline: October Most final plans are due by September 30 but some will be due in early October Review final plans to verify needed changes were made Follow up as necessary

Process Collective review Track changes and comments directly in planning documents (Word) Track section and strategy ratings in Google Form Departmental feedback meetings Director of Assessment plus the Department Representative to Committee Director of Department plus any staff they invite Negotiated timeline for final plans

Reminder of Results (2014-15) Strategies reviewed: 150 Portion Incomplete: 8.7% Count: 13 Portion Accepted: 26.7% Count: 40 Red X source: https://pixabay.com/en/cross-mark-red-sign-icon-mark-304374/ Green check source: https://pixabay.com/en/check-mark-tick-mark-check-correct-1292787/ 35.4%

Subsequent Results (2015-16) Strategies reviewed: 82 Portion Incomplete: 7.3% Count: 6 Portion Accepted: 80.5% Count: 66 Red X source: https://pixabay.com/en/cross-mark-red-sign-icon-mark-304374/ Green check source: https://pixabay.com/en/check-mark-tick-mark-check-correct-1292787/ 87.8%

Image: https://netcommunity.unlv.edu/SrAVPDevelopment Feeling AWESOME! https://netcommunity.unlv.edu/SrAVPDevelopment Image: https://netcommunity.unlv.edu/SrAVPDevelopment

Subsequent Results (2016-17) Strategies reviewed: 56 Portion Incomplete: 32.1% Count: 18 Portion Accepted: 41.1% Count: 23 Red X source: https://pixabay.com/en/cross-mark-red-sign-icon-mark-304374/ Green check source: https://pixabay.com/en/check-mark-tick-mark-check-correct-1292787/ 73.2%

WHAT HAPPENED?! New hires Large department restructured its plans starting in July but didn’t finish

New Hires Revise onboarding Workshop strategy writing Emphasize resources available for strategy writing Consultations Guidelines for self-evaluating strategies

Subsequent Results (2017-18) Strategies reviewed: 58 Portion Incomplete: 32.8% Count: 19 Portion Accepted: 39.6% Count: 23 Red X source: https://pixabay.com/en/cross-mark-red-sign-icon-mark-304374/ Green check source: https://pixabay.com/en/check-mark-tick-mark-check-correct-1292787/ 73.2%

WHAT HAPPENED?! Large department (same as year before) restructured into 3 departments New Hires (added positions) Two new directors Four new assistant directors Additional new hires

RESPONSE for AY2019 Cycle Reiterate with the executives overseeing each department the need to forward complete strategies for review Provide additional training for new supervisors and directors so they know what to look for during their reviews Focus new employee training even more to workshop strategies

Costs Peer Review Retreat Follow-Up Meetings Final Review = ~2 hours 2 full days (19 clock hours) 9.5 staff members (180.5 staff hours) Large conference room (SmartRoom) [No charge] Beverages, Snacks, Lunches (~$600) Follow-Up Meetings 10 summary reports @ 0.5 hour each = 5 staff hours 10 departments @ 1 hour each @ varying staff in each = ~72 staff hours Final Review = ~2 hours

Questions & Discussion J. Judd Harbin, Ph.D. judd.harbin@unlv.edu (702) 895-2973 Twitter: @DrJuddHarbin LinkedIn: juddharbin

Thank you for joining us today! Please remember to complete your online evaluation following the conference. See you in Los Angeles in 2019!