1. Virginia Department for Aging and Rehabilitative Services 1 Barbara J. Burkett, Ph.D., M.S.P.H. Division of Policy and Planning 804-726-1911

2.  Report Requested by the Commissioner and Field Services Director on the following:  Utilization of vocational evaluation services  Program’s effectiveness  Outcomes  Counselors’ satisfaction  Best practices  Other state’s use of vocational evaluation 2

3. 3  Review of Internal DARS Data  Number of Clients Served  Timing of Vocational Evaluations Within The Rehabilitation Process  Demographics of Clients Receiving Evaluations  Outcomes for Clients Receiving Vocational Evaluations Compared to Those Who Do Not Receive Vocational Evaluations  Cost Metrics of Vocational Evaluations  DARS Case Audit Reviews  Literature Search  Roles and Functions of Vocational Evaluators  Review of Various State Vocational Rehabilitation Programs Use of Vocational Evaluators  Population Specific Instruments/Best Practices  Survey of DARS Counselors Regarding Current Program

4.  Review of Internal DARS Data  Data was extracted from AWARE, the DARS case management system, and imported in to SPSS for statistical analysis.  Study Years: Federal Fiscal Years 2011, 2012 and 2013 ▪ Vocational Evaluation Cases Identified ▪ 2011 case notes only ▪ 2012 case notes and vendor file codes ▪ 2013 case notes and vendor file codes 4

5.  Case Notes  String search for every iteration of “vocational evaluation” (Example: vocational evaluation, voc. eval., v.e., evaluation, evaluation, eval,….inclusive of v* and e* with and without punctuation and capitalization)  Verified string search results by opening each case in AWARE. AND  Vendor File  Career Services  Comprehensive Evaluation  Exploratory Evaluation  Referral/Staffing/Consultation  Report Writing 5

6.  17 Vocational Evaluation Positions  15 Currently Filled  2 Vacancies (Eastern, Southwest) 6

7.  Six percent of DARS clients received a vocational evaluation during the study period. 7 FFY # Clients (Application –Post Employment Status) # Vocational Evaluations % of All Clients In Application through Post Employment Status Receiving A Vocational Evaluation 201130,7951,8436.0% 201228,8771,5275.3% 201328,1121,9446.9%

8.  There was no regional variation in requests for vocational evaluation.  There was great variation in the use of vocational evaluation by counselors.  The range in the number of vocational evaluations requested by DARS counselors during the three year time period was 1-127.  Average number of vocational evaluations requested was 8.8 per counselor per year (s.d of 8.7).  The mode was 6 requests per counselor per year. 8

9. 9

10.  Overall for the 3 year time period, transition students were more likely than non transition students to receive a vocational evaluation. 10

11.  More males than females received vocational evaluations. 11

12.  43% cognitive impairment  28.4% psychosocial impairment  7.9% other mental impairment 12

13. 13  The rehabilitation rate was lower for clients who received a vocational evaluation over the 3 year period.

14. 14  Average hourly earnings were lower for clients receiving a vocational evaluation.

15. 15  Clients who received a vocational evaluation were less likely to obtain a job at closure that was the same as indicated on the IPE.

16.  Clients who received vocational evaluations were less likely to be referred for supported employment services. 16

17. 17  The VR cost of the clients’ who received a vocational evaluation were lower than for those clients who did not receive a vocational evaluation.

18.  Conducted by Scott Fraley and Tim Olive Report Written by Liz Smith  Selection Criteria for Cases  Cases closed in FFY2013 that had a vocational evaluation as a planned service  Stratified random sample by region and caseload type  Total sample n=40 18

19.  Five (12%) of the forty cases had a vocational evaluation scheduled, but never conducted. Two of these were caused by the consumer failing to appear for the vocational evaluation. The other three reasons were not evident from the case file.  35 Reports  Twenty (57%) had no job recommendation in the vocational evaluation report. Eight of the 20 cases with no recommendation (23%) were transition cases.  Of the fifteen cases that had a job recommendation in the vocational evaluation report:  For twelve (80%) of the cases: the vocational evaluation recommendation, the vocational goal in the IPE, and the job at case closure matched. 19

20.  If it is the expectation that a job recommendation occur when a vocational evaluation is conducted, then that is not occurring on a consistent basis.  When a job recommendation is made in the vocational evaluation, there is a high level of consistency between that recommendation, the IPE goal, and the consumer’s job at case closure. 20

21.  Response Rate For Survey : 85.7% n/N = 12/14  Number of Vocational Evaluations FFY13  x = 96.5  range = 76-123  Number of Vocational Evaluations April 2014  x = 10.6  Range 6-20  No show rate is a problem with 2 evaluators reporting a no show rate of 46.4% 21

22. 22  Referral Source  DARS evaluators  1 report of parent/guardian  1 client request  1 supported employment vendor  Time From Referral to Evaluation  X =41.7 days  Range 20 – 65  Time to Complete Written Report  X= 6.1 days  Range = 2-16

23.  Template Use  50% use now  25% would like to use  25% would not find it helpful  Job Recommendations Made  82% make job recommendation  18% did not (=2) 23

24.  #1: Incomplete information on client at time of evaluation  #2: Client willingness to participate  #3: Scheduling problems  #4: Poor referrals  #5: Client ability to participate 24

25.  Discussion of vocational evaluation reports with DARS counselors  55% always discuss (n=6)  46% discuss sometimes (n=5)  1 person did not answer this question 25

26.  Instruments  Work Behavior Inventory used by 2 evaluators  Threshold Monthly Work Evaluation: 100% not familiar with instrument  Vocational Cognitive Rating Scale: 100% not familiar with instrument  Work Capacity Evaluation used by 2 evaluators  Maryland Assessment of Social Competence: 100% not familiar  Change Assessment for Vocational Counseling (URI): 100% not familiar  IPS Fidelity Scale: 1 evaluator familiar with instrument  Supported Employment Scale: 1 evaluator familiar with this instrument 26

27.  Population Based Instruments 27 UseDo not useWould Like to Use Ethnic Based712 Minority Based 434 Gender Based524 Hearing Impaired 515 Autism Based803 Mental Illness524 College Bound 902

28.  Carolyn Graham, Ph.D. (VCU) contracted for literature review  Barbara Burkett, Ph.D. (DARS)  Best Practices: Guide to Vocational Assessment and Evaluation-4 th Edition Author: Paul Power  “The State of Vocational Evaluators: A National Study” Sligar and Betters Journal of Rehabilitation Vol. 74, no. 4 28

29.  Sligar and Betters interviewed 63 agencies o 97% (n= 60) of states purchased vocational evaluation services from evaluators in private practice, psychologists, and community rehabilitation programs. o Average number of vocational evaluators by agency was 15.1 with a range of 1-72. o 65% of the respondents did not require a certification o 81% of the programs had guidelines for the vocational evaluation and the final report. 29

30.  The vocational evaluator positions currently unfilled in the Central Region will be filled.  The vocational evaluator position in the Southwest region will not be filled.  A new position has been created. The position will supervise the vocational evaluators, the business development managers and the self employment enterprise staff.  Duties of this position will include: ▪ Updating the DARS Best Practices Manual ▪ Developing improved labor market data for use by DARS ▪ Developing requirements for report writing ▪ Procuring new instruments identified in the literature review and conducting training 30

31. 31

32.  Clearly communicate the purpose of the program evaluation to staff involved.  Describe the role of the program evaluator.  Provide a program evaluation plan as soon as possible to the stake holders.  Include the stakeholders in the planning meetings if possible  Present the findings to those involved and allow for input before final evaluation report.  Don’t take comments personally. This is the nature of th e job. 32

33.  www.decisionswithdata.com www.decisionswithdata.com  Blog 33

34.  Null Hypothesis (Ho) and Alternative Hypothesis (Ha) Stated  Population Distribution Determined For Variables  Probability plot ▪ Normal DistributionNon-normal Distribution or small sample size  Parametric or Non-Parametric Test Chosen  Test of Difference Between Proportions ▪ Z-test: parametric ▪ Chi-Square Test: non-parametric  Test of Difference Between Means ▪ T-test: parametric ▪ Kolomogorov-Smirnov Test: non-parametric  See Appendices of Program Evaluation for statistical formulas 34