1. NALP Graduate Employment Outcomes Review
NALP Conference
April 27, 2018
William E. Adams, Jr.
Deputy Managing Director
Kirsten Winek
Manager, Law School Analytics
American Bar Association

2. NALP Employment Outcomes Review
Purposes
To promote confidence in employment outcome reports
Demands for transparency 
Criticisms of past practices by some schools
To ensure consistency in reporting various categories and types of employment outcomes
Guidance on categorization and assumptions
To ensure compliance with ABA Standards

3. NALP Employment Outcomes Review
Standard 104
A law school shall furnish a completed annual questionnaire, self-study, site evaluation questionnaire, and such other information as the Accreditation Committee or Council may require. This information must be complete, accurate, and not misleading, and must be submitted in the form, manner, and time frame specified by the Council.

4. NALP Employment Outcomes Review
Standard 509
(a) All information that a law school reports, publicizes, or distributes shall be complete, accurate and not misleading to a reasonable law school student or applicant. A law school shall use due diligence in obtaining and verifying such information. Violations of these obligations may result in sanctions under Rule 16 of the Rules of Procedure for Approval of Law Schools.
(c) A law school shall publicly disclose on its website, in the form and manner and for the frame designated by the Council, the following information:
(7) employment outcomes; and

5. NALP Employment Outcomes Review
Who Makes and Enforces Rules?— Current Practices
Standards Review Committee and Council
Data Policy and Collection Committee (DPCC) merged into Standards Review Committee
Council sets Protocol and Process Requirements
Standards Review Committee will continue to review and make recommendations for changes
Standards Review Committee still provides oversight
Accreditation Committee
For schools found to have misreported data, will review for any action to be taken

6. NALP Employment Outcomes Review
Who Makes and Enforces Rules?— Proposed Governance Change
Proposal to merge Standards Review Committee, Accreditation Committee and Council into one body
Could be become effective August, 2018

7. NALP Employment Outcomes Review
Revisions to Process this year
2015 Class Reviewed by Berkeley Research Group
2016 Class Review brought inhouse
Permitted office to see files
Enabled office to provide individualized feedback
Utilized someone with Career Services experience to review files

8. NALP Employment Outcomes Review
Website Review
Check for posting of reports and accuracy of reports
If incomplete, inaccurate or misleading, subject to Red Flag Review
Random School Review
10 schools selected randomly for review of all graduate files
Random Graduate Review
Selected from list of all graduates
2015 – 382 files from 157 schools
2016 – 371 files from 160 schools

9. NALP Employment Outcomes Review
Results from Classes of  2015 and 2016 Employment Outcomes Review
2015
Random Graduate Review
6 schools given elevated Level 1 reviews
2 schools given elevated Level 2 reviews
3 schools given elevated Level 3 reviews
Random School Review
2 Schools given elevated Level 2 reviews
2016
2 schools given Level 2 reviews (from Random School)

10. NALP Employment Outcomes Review
Results of SRC Student Survey
SRC sent survey to first year students at 10 law schools
High Levels of student use of employment data
86% reviewed employment data
83% reported  information about bar pass jobs important in deciding on law school; 68% considered JD advantage jobs important; 82% number employed; 78% number unemployed; 64% number working in Full-time or Part-time positions; 66%, number working in Long-term or Short-term positions

11. NALP Employment Outcomes Review
Protocol for Reviewing Law Graduate Employment Data, and Statement of Procedures for Collecting, Maintaining, and Reporting Law Graduate Employment Data (June 9, 2014)
Passed by the Council of the Section of Legal Education and Admissions to the Bar
Recommended by the DPCC
Developed by Scott Norberg (prior Deputy Managing Director), Peggy Daley of Berkeley Research Group (BRG) and working group of Career Services Directors
Beta tested by Work group before implemented
EQ Definitions and Instructions
FAQ’s

12. NALP Employment Outcomes Review
NALP Advisory Group
Selected by NALP
Heather DiFranco, Cleveland Marshall College of Law (Chair)
Angelica Evans, Texas
Courtney Fitzgibbons, New York Law School
Arturo Thompson, Kansas
Betsy Armour, USC Gould School of Law
Melissa Berry, Washington
Fiona Hornblower, Boston University
Eric Bono, Denver Sturm College of Law
Brian Lewis, North Carolina

13. NALP Employment Outcomes Review
Next Steps with the Class of 2017
Website Reviews
Making Changes to Your Employment Questionnaire Data
Selections for Random School and Random Graduate Reviews
Uploading Documents for the Random School and Random Graduate Reviews
What happens during the Reviews
Levels of Reviews

14. NALP Employment Outcomes Review
Website Reviews
Office is in the process of checking to see that schools have posted their Class of Employment Summary Report Forms to their websites
Deadline to post was April 16; URLs were submitted with Employment Questionnaire
Schools will be notified if there are any issues

15. NALP Employment Outcomes Review
Making Changes to Your Employment Questionnaire Data
The system is now open for changes to your Employment Questionnaire Data
If you have any changes to make, contact Ken Williams or Andrew Crane or
No deadline for submitting changes; however, please contact Ken or Andrew as soon as you learn of these changes
For changes you currently have, contact Ken or Andrew at earliest convenience
Continuing obligation to report changes in employment as of March 15 if you learn of them until due date for next year’s Employment Questionnaire

16. NALP Employment Outcomes Review
Selections for Random School and Random Graduate Reviews
Random School Review – 10 Schools Selected
Schools notified on May 7 ( to Dean and Head of Career Services)
Submit all Class of 2017 Graduate Employment Files for Review
Random Graduate Review – usually around schools selected
Schools notified May 8-11 ( to Dean and Head of Career Services)
Submit selected Graduate Employment Files for Review (usually 1-7)

17. NALP Employment Outcomes Review
Uploading Documents for the Random School and Random Graduate Reviews
Where to Upload
ABAQuest System (wwww.abaquestionnaire.org) under EQ Protocol Upload
How to Upload
Instructions will be sent to schools selected for Random School and Random Graduate Reviews
Also posted on Questionnaires webpage:
Deadline to Upload (Early Submissions Appreciated!)
Random School Review: June 4
Random Graduate Review: May 25

18. NALP Employment Outcomes Review
What happens during the Reviews?
One Reviewer – Kirsten Winek
Use Checklist of Key Items
Contact Schools with Questions
Goals of the Reviews
Give Schools Feedback
Look for Intentional Misreporting

19. NALP Employment Outcomes Review
Levels of Review
Level 1 Reviews
For all files of Randomly Selected Schools
For all files of Schools with Randomly Selected Graduates with deficient files
With permission of SRC and Council, office has used discretion to not elevate schools whose deficiencies appeared more technical than an indication of inaccurate reporting
An area for potential amendments to the Protocol
If 5% of files are deficient, proceed to Level 2 Review
Another area for potential amendments to the Protocol

20. NALP Employment Outcomes Review
Level 2 Reviews
Verify reported data of at least 20% of class through contact with graduates, employers or public records
If 5% or three files incomplete, inaccurate or misleading, then Level 3 Review
Level 3 Reviews
Third party review
Confirm at least 25% of graduate outcomes by verifying through contact with graduates, employers or public records
Undertaken at expense of school

21. NALP Employment Outcomes Review
Prior to Elevation to Level 2 or 3 Review, discussion with school about ambiguities or other questions
Missing documentation
Possible misclassification of position
Internal inconsistency in file
Documentation doesn’t support reported employment category or term

22. NALP Employment Outcomes Review
Red Flag Review
Schools under sanction for Standard 509 violations
Schools identified as having significant inconsistencies or anomalies in their data reporting
Schools subject to credible reports of incomplete, inaccurate or misleading reporting
ABA determines whether to conduct Level 1, 2 or 3 Review

23. NALP Outcomes Review Future directions
Continue to review with NALP Advisory Group and Council

24. NALP Employment Outcomes Review
ABA Staff Contacts
Kirsten Winek, (main contact)
Ken Williams, (data entry)
Andrew Crane, (listserv and data entry)
Bill Adams,
To Join the ABA Career Services Listserv:
on_listservs.html
Or contact