NALP Graduate Employment Outcomes Review NALP Conference April 27, 2018 William E. Adams, Jr. Deputy Managing Director Kirsten Winek Manager, Law School Analytics American Bar Association

NALP Employment Outcomes Review Purposes To promote confidence in employment outcome reports Demands for transparency  Criticisms of past practices by some schools To ensure consistency in reporting various categories and types of employment outcomes Guidance on categorization and assumptions To ensure compliance with ABA Standards

NALP Employment Outcomes Review Standard 104 A law school shall furnish a completed annual questionnaire, self-study, site evaluation questionnaire, and such other information as the Accreditation Committee or Council may require. This information must be complete, accurate, and not misleading, and must be submitted in the form, manner, and time frame specified by the Council.

NALP Employment Outcomes Review Standard 509 (a) All information that a law school reports, publicizes, or distributes shall be complete, accurate and not misleading to a reasonable law school student or applicant. A law school shall use due diligence in obtaining and verifying such information. Violations of these obligations may result in sanctions under Rule 16 of the Rules of Procedure for Approval of Law Schools. (c) A law school shall publicly disclose on its website, in the form and manner and for the frame designated by the Council, the following information: (7) employment outcomes; and

NALP Employment Outcomes Review Who Makes and Enforces Rules?— Current Practices Standards Review Committee and Council Data Policy and Collection Committee (DPCC) merged into Standards Review Committee Council sets Protocol and Process Requirements Standards Review Committee will continue to review and make recommendations for changes Standards Review Committee still provides oversight Accreditation Committee For schools found to have misreported data, will review for any action to be taken

NALP Employment Outcomes Review Who Makes and Enforces Rules?— Proposed Governance Change Proposal to merge Standards Review Committee, Accreditation Committee and Council into one body Could be become effective August, 2018

NALP Employment Outcomes Review Revisions to Process this year 2015 Class Reviewed by Berkeley Research Group 2016 Class Review brought inhouse Permitted office to see files Enabled office to provide individualized feedback Utilized someone with Career Services experience to review files

NALP Employment Outcomes Review Website Review Check for posting of reports and accuracy of reports If incomplete, inaccurate or misleading, subject to Red Flag Review Random School Review 10 schools selected randomly for review of all graduate files Random Graduate Review Selected from list of all graduates 2015 – 382 files from 157 schools 2016 – 371 files from 160 schools

NALP Employment Outcomes Review Results from Classes of  2015 and 2016 Employment Outcomes Review 2015 Random Graduate Review 6 schools given elevated Level 1 reviews 2 schools given elevated Level 2 reviews 3 schools given elevated Level 3 reviews Random School Review 2 Schools given elevated Level 2 reviews 2016 2 schools given Level 2 reviews (from Random School)

NALP Employment Outcomes Review Results of SRC Student Survey SRC sent survey to first year students at 10 law schools High Levels of student use of employment data 86% reviewed employment data 83% reported  information about bar pass jobs important in deciding on law school; 68% considered JD advantage jobs important; 82% number employed; 78% number unemployed; 64% number working in Full-time or Part-time positions; 66%, number working in Long-term or Short-term positions

NALP Employment Outcomes Review Protocol for Reviewing Law Graduate Employment Data, and Statement of Procedures for Collecting, Maintaining, and Reporting Law Graduate Employment Data (June 9, 2014) Passed by the Council of the Section of Legal Education and Admissions to the Bar Recommended by the DPCC Developed by Scott Norberg (prior Deputy Managing Director), Peggy Daley of Berkeley Research Group (BRG) and working group of Career Services Directors Beta tested by Work group before implemented EQ Definitions and Instructions FAQ’s

NALP Employment Outcomes Review NALP Advisory Group Selected by NALP Heather DiFranco, Cleveland Marshall College of Law (Chair) Angelica Evans, Texas Courtney Fitzgibbons, New York Law School Arturo Thompson, Kansas Betsy Armour, USC Gould School of Law Melissa Berry, Washington Fiona Hornblower, Boston University Eric Bono, Denver Sturm College of Law Brian Lewis, North Carolina

NALP Employment Outcomes Review Next Steps with the Class of 2017 Website Reviews Making Changes to Your Employment Questionnaire Data Selections for Random School and Random Graduate Reviews Uploading Documents for the Random School and Random Graduate Reviews What happens during the Reviews Levels of Reviews

NALP Employment Outcomes Review Website Reviews Office is in the process of checking to see that schools have posted their Class of 2017 Employment Summary Report Forms to their websites Deadline to post was April 16; URLs were submitted with Employment Questionnaire Schools will be notified if there are any issues

NALP Employment Outcomes Review Making Changes to Your Employment Questionnaire Data The system is now open for changes to your Employment Questionnaire Data If you have any changes to make, contact Ken Williams or Andrew Crane kenneth.williams@americanbar.org or andrew.crane@americanbar.org No deadline for submitting changes; however, please contact Ken or Andrew as soon as you learn of these changes For changes you currently have, contact Ken or Andrew at earliest convenience Continuing obligation to report changes in employment as of March 15 if you learn of them until due date for next year’s Employment Questionnaire

NALP Employment Outcomes Review Selections for Random School and Random Graduate Reviews Random School Review – 10 Schools Selected Schools notified on May 7 (Email to Dean and Head of Career Services) Submit all Class of 2017 Graduate Employment Files for Review Random Graduate Review – usually around 150-160 schools selected Schools notified May 8-11 (Email to Dean and Head of Career Services) Submit selected Graduate Employment Files for Review (usually 1-7)

NALP Employment Outcomes Review Uploading Documents for the Random School and Random Graduate Reviews Where to Upload ABAQuest System (wwww.abaquestionnaire.org) under EQ Protocol Upload How to Upload Instructions will be sent to schools selected for Random School and Random Graduate Reviews Also posted on Questionnaires webpage: https://www.americanbar.org/groups/legal_education/resources/questionnaire.html Deadline to Upload (Early Submissions Appreciated!) Random School Review: June 4 Random Graduate Review: May 25

NALP Employment Outcomes Review What happens during the Reviews? One Reviewer – Kirsten Winek Use Checklist of Key Items Contact Schools with Questions Goals of the Reviews Give Schools Feedback Look for Intentional Misreporting

NALP Employment Outcomes Review Levels of Review Level 1 Reviews For all files of Randomly Selected Schools For all files of Schools with Randomly Selected Graduates with deficient files With permission of SRC and Council, office has used discretion to not elevate schools whose deficiencies appeared more technical than an indication of inaccurate reporting An area for potential amendments to the Protocol If 5% of files are deficient, proceed to Level 2 Review Another area for potential amendments to the Protocol

NALP Employment Outcomes Review Level 2 Reviews Verify reported data of at least 20% of class through contact with graduates, employers or public records If 5% or three files incomplete, inaccurate or misleading, then Level 3 Review Level 3 Reviews Third party review Confirm at least 25% of graduate outcomes by verifying through contact with graduates, employers or public records Undertaken at expense of school

NALP Employment Outcomes Review Prior to Elevation to Level 2 or 3 Review, discussion with school about ambiguities or other questions Missing documentation Possible misclassification of position Internal inconsistency in file Documentation doesn’t support reported employment category or term

NALP Employment Outcomes Review Red Flag Review Schools under sanction for Standard 509 violations Schools identified as having significant inconsistencies or anomalies in their data reporting Schools subject to credible reports of incomplete, inaccurate or misleading reporting ABA determines whether to conduct Level 1, 2 or 3 Review

NALP Outcomes Review Future directions Continue to review with NALP Advisory Group and Council

NALP Employment Outcomes Review ABA Staff Contacts Kirsten Winek, kirsten.winek@americanbar.org (main contact) Ken Williams, kenneth.williams@americanbar.org (data entry) Andrew Crane, andrew.crane@americanbar.org (listserv and data entry) Bill Adams, william.adams@americanbar.org To Join the ABA Career Services Listserv: http://www.americanbar.org/groups/legal_education/resources/legal_educati on_listservs.html Or contact andrew.crane@americanbar.org