1. Pilot College Expectations and Implementation Steps Multiple Measures Assessment Project Pilot College Convening http://w.rpgroup.org/projects/multiple-measures-assessment-project

2. General Expectations Identification of local point person/persons/committee Participation/communication with MMAP Local retrospective analysis (optional) Webinar: http://bit.ly/MMAPCARTWebinarhttp://bit.ly/MMAPCARTWebinar DED: http://bit.ly/MMAPRetrospectiveDEDhttp://bit.ly/MMAPRetrospectiveDED For files, contact: Dan Lamoree: dlamoree@edresults.orgdlamoree@edresults.org

3. General Expectations - 2 Implement cohort for Spring 2016 Upload cohort of applicants (supplemented by CCC Apply) Receive recommended placements Help pilot measurement of non-cognitive variables(NCVs) NCV Scales: http://bit.ly/MMAPNCVScaleshttp://bit.ly/MMAPNCVScales NCV Information Sheet: http://bit.ly/MMAPNCVInfohttp://bit.ly/MMAPNCVInfo Contact: Craig Hayward: chayward@rpgroup.orgchayward@rpgroup.org Help test reliability of self-reported HS data Unweighted cumulative HS GPA, last math course and grade, last English course and grade Contact: John Hetts: jhetts@edresults.orgjhetts@edresults.org Upload student level data to CalPass Plus Placements First semester course enrollments and outcomes

4. Key aspects of implementation Initiation Early Implementation –––––––– Program development Resource planning Professional Development Communication plan Outreach/K-12 Finalizing implementation – Local research & customization of rule sets – Placement execution and communication – Evaluation plan – Finalizing timeline & tracking execution

5. Some caveats: Context matters While all colleges have capacity to do this, colleges are at very different places in the process Not all steps are necessary or even always desired at every college Process isn’t necessarily linear Implementation matters but implementation matters

6. Step 1: Initiation Develop decision-making group(s) appropriate to local context Workgroup, subcommittee, campus-wide committee… Identify & recruit key stakeholders/decision-makers – Discipline faculty in English and Math (as well as ESL and Reading) Assessment committees/subcommittees Counseling/Matriculation/Student Support Services Staff and Faculty – – Student Success Staff and Faculty (Tutoring, Success Courses) – Academic Senate representation – Academic and Executive Administration – Students

7. Step 1: Initiation Develop regular meeting schedule Establish common ground – Introduction to & review of background research – Use of discussion activities/questions – Identification of local barriers/challenges and development of plans/strategies to address Set stretch goals for desired change at college Attend/review webinars Review/discuss statewide MMAP research

8. Step 1: Initiation: Develop Comprehensive Plans Engagement Plan Faculty involvement Professional development Engaging special programs Assessment Plan Placement process Non Cognitive measures Students w/o transcripts Student Services Plan Counseling Support services (tutoring) Outreach – HS Assessment Enrollment Plan Class offerings Implementation process Communication Plan Informing stakeholders Informational materials Informing feeder HS Research Plan Quarterly analysis Longitudinal analysis Equity impacts

9. Step 2: Program Development Eligibility guidelines (who/what conditions) Multiple measures rule sets/protocol* – Statewide default and/or local refinement – Integration with SIS/Assessment Other program components – Piloting of noncognitive variables* – Collecting self-reported transcript data College responsibilities/commitments Contingency planning – student support services, scheduling buffers, late-starting classes, early alert, just-in-time intervention

10. Step 2: Resource Planning and Development Enrollment management – Cohort baseline attendance – New enrollment expectations based on program (changes in demand by level) Staffing needs for program elements and/or changes in course level Exploration of external sources of funding Calendar development – Backwards mapping from significant dates Student-centered scheduling

11. Step 3: Communication Planning Internal: students, faculty, staff, departments, committees – Overall summary – Additional/supporting information – Resources – Program elements – Schedules/Timelines – Key contacts – Ongoing information/updates – Progress summaries – Methods Web home/portal Campus listserv/mailing lists Outreach events/activities Community awareness campaign

12. Step 3: Communication Planning External/Outreach: K-12 students, parents, counselors, teachers, community, businesses/funders – Similar categories/methods but also – Intersegmental relationship/trust development (esp. faculty) – Introduce evidence based placement and what it means for students/districts – Communication of needs to district – Work with Cal-PASS PLUS to verify and update MOUs and data uploads – https://www.calpassplus.org/calpass/join/members https://www.calpas plus.org/calpas /join/members – Identify primary outreach contact from college to work with CalPASS. – Name, college, role, contact info to Victoria Pluim, vpluim@edresults.orgvpluim@edresults.org

13. Step 3: Professional Development Planning Unlearning & replacing previous narratives Discipline faculty: – Transfer level courses (e.g., first-year students in first year English again) – Developmental courses (e.g., best students at appropriate higher level) – Training support for faculty transitioning between levels Student support services faculty and staff – How to interpret and incorporate additional evidence of student capacity into education planning and other student contact Institutional research and IT skills and capacity

14. Step 4: Finalizing Implementation Local research & customization of rule sets Placement execution and communication Evaluation plan Completing timeline with backward mapping Tracking execution

15. Step 4: Local research and rule set customization Complete local version of statewide retrospective analysis and rule set development (if desired) College and District specific versions of the statewide dataset are now available for local/district institutional research department including – DED DED – SPSS Variables SPSS Variables – R R Contact dlamoree@edresults.org dlamoree@edresults.org Provide name, institution, role, and contact information

16. Step 4: Placement execution & communication Method of execution of placement – CalPASS Plus using statewide model – CalPASS Plus using local customization – Local data-sharing & local customization – Other alternatives (also useful for backup plan for helping students with missing data) self-report transcript review Method of communication to students – Additional point of contact may be needed Process for setting/updating placements and clearing prerequisites Clear documentation/record of multiple measures rule sets & students to whom applied

17. Step 4: Placement recommendations using CalPASS Plus College staff creates a file of incoming student/applicants being assessed for placement in fall 2015 Upload file to calpassplus.org Receive back a file or files with: indicators showing recommended placements (CB21 levels) based on students’ high school data additional recommendations for eligibility for different transfer-level courses in math variables in dataset (parallel to retrospective data) to allow for local customization (if needed)* Translate to local courses/placements* Import/integration into: assessment database/SIS* student support services protocols * Indicates places where there _may_ be opportunities for colleges to work closely with CalPASS Plus staff on local customization within reasonable limits

18. (If you haven’t already, please complete the pilot college status survey: http://goo.gl/31UJnw)ht p://goo.gl/31UJnw

19. (If you haven’t already, please complete the pilot college status survey: http://goo.gl/31UJnw)ht p://goo.gl/31UJnw

20. MMAP Student Cohort Upload File VariableTypeSourceDescription district_college_identifierX(03)GI01COMIS DED student_identifierX(09)SB00COMIS DED student_identifier_statusX(01)SB01COMIS DED student_birth_date9(08)SB03COMIS DED student_genderX(01)SB04COMIS DED student_high_school_lastX(06)SB12COMIS DED student_first_nameX(30)SB31COMIS DED student_last_nameX(40)SB32COMIS DED

21. Student course output file VariableTypeValuesDescription student_idX(09)SB00COMIS DED student_id_statusX(01)SB01COMIS DED engl_cb21X(01)Y;A;B;C;DCOMIS DEDCOMIS DED. Evidence-based multiple measures assessment course level recommendation in English math_cb21X(01)Y;A;B;C;DCOMIS DEDCOMIS DED. Evidence-based MM assessment course level recommendation in Math. If Y consult below. math_college_algebra9(01)0;1Indicator recommending the student for college algebra; math_cb21 = Y math_libart9(01)0;1Indicator recommending the student for non-STEM algebra; math_cb21 = Y math_stats9(01)0;1Indicator recommending the student for statistics; math_cb21 = Y math_precalc9(01)0;1Indicator recommending the student for precalculus; math_cb21 = Y math_calc_i9(01)0;1Indicator recommending the student for calculus I; math_cb21 = Y math_calc_ii9(01)0;1Indicator recommending the student for calculus II; math_cb21 = Y

22. Step 4: Completing Evaluation Plan Evaluate implementation Every semester and annually Explore disproportionate impacts Changes in success rates at all levels Tracks students over multiple terms when possible Data Transparency Share the data with the workgroup and college Make adjustments based on findings

23. Step 4: Completing Timeline and Tracking Execution Develop/review institutional timeline working backward from key institutional dates & deadlines – Target completion dates – Must meet completion dates – Build in flexibility, ability to respond on the ground, contingency planning Track execution and watch for opportunities – What worked Celebrate milestones/successes! – What didn’t – What could have worked/will work next year – Document changes to plan