1. Customer Satisfaction Research 2018 Q1 Results May 15, 2017
Florida Rehabilitation Council
Division of Vocational Rehabilitation Florida Department of Education
Customer Satisfaction Research 2018 Q1 Results May 15, 2017
Brian Robertson
Mark Noyes
Nat Mildner

2. Contents Survey Methodology Customer Experience Core Metrics
Problems Experienced by VR Customers and Feedback
Understanding the Process if You Disagree with VR About Your Services
Satisfaction with Outside Vendors
Current Employment Status
In-depth: Data Sharing
Potential Survey Modifications
Early Results: April 2018 Focus Groups 2

3. Current Quarter Survey Methodology
Data collection took place from January 3, 2018 through April 9, 2018.
Over the course of data collection, 750 customers completed surveys:
383 customers with open cases
203 customers - closed successfully
165 customers - closed unsuccessfully. 3

4. Current Quarter Response Rates%
APPOR Response Rate (RR3)
28.3%
AAPOR Respondent Cooperation Rate (COOP3)
83.4%
AAPOR Respondent Refusal Rate (REF3)
7.4%
AAPOR – The American Association for Public Opinion Research.
Response Rate – The response rate is the number of completed interviews divided by the number of eligible reporting customers in the sample.
Cooperation Rate – This represents the proportion of all cases interviewed out of all eligible customers ever contacted. That is, the percent of identified respondents that ended up completing the interview.
Refusal Rate – The refusal rate represents the proportion of all cases in which the respondent refused to be interviewed, or broke off an interview, out of all respondents that were contacted and spoken with. 4

5. Customer Experience Core Metrics5

6. Florida VR Customer Experience Dashboard
Measure
2017 Q3
2017 Q4
2018 Q1
Overall Satisfaction and Expectations 80
Experience with Services Provided by VR
Experience with VR Staff and Counselors 90 91
Communications with VR Staff
Customer Control and Involvement 81
Outcomes and Meeting Goals 83 82
Satisfaction with Current Employment 75
Satisfaction with Outside Vendors 79 77 76
All things considered, would you tell your friends with disabilities to go to the VR program for help? (% Yes) 95 94 93
Have you experienced any problems with VR or the services they have provided to you? (% No) 6

7. Overall Satisfaction and Expectations:
Q01: Overall, how satisfied are you with the Florida VR’s program? Q33: How satisfied would you say you are with the services provided by VR? Q35: Please rate how well VR services met your expectations. 7

8. Experience with Services Provided by VR:
Q09: How satisfied are you with the choice of services?
Q11: How satisfied are you with the choice of service providers? 8

9. Experience with VR Staff and Counselors:
Q09: How helpful is the staff of VR in helping you achieve your VR goal?
Q29: The VR staff treats me with dignity and respect. 9

10. Communications with VR Staff:
Q13: How satisfied are you with the information you were given about the choices you had?
Q15: How satisfied are you with how long it took your counselor to answer your questions and address your concerns?
Q21: How easy is it for you to contact your VR counselor? 10

11. Customer Control and Involvement:
Q05: How satisfied are you with your involvement in your VR experience?
Q07: How satisfied are you with your choice of a vocational goal? 11

12. Outcomes and Meeting Goals:
Q25: The VR services I am receiving will help me become more financially independent.
Q27: The VR services I am receiving will help me become more independent, in general.
Q31: The VR staff will help me reach my job goal. 12

13. Satisfaction with Current Employment:
Q51: If working, thinking about your current job, how satisfied are you with what you are doing? Would you say you are...? 13

14. Satisfaction with Outside Vendors:
Q46: How satisfied were you with the services provided by OUTSIDE VENDOR? 14

15. Recommend VR:
All things considered, would you tell your friends with disabilities to go to the VR program for help?
(% reporting yes) 15

16. Experienced Problems:
Have you experienced any problems with VR or the services they have provided to you?
(% reporting no) 16

17. Problems Experienced by VR Customers and Feedback
Customers were asked if they experienced problems specifically with VR or the services they received from VR and reasons for dissatisfaction. 17

18. 18% of customers experienced problems with VR or the services they provided.

19. The Most Commonly Mentioned Problems (2018 Q1):%
Customers
Issues communicating with staff or counselors, difficulty reaching staff or counselors
38% 58
VR Counselors or staff did not did not provide the help I needed, broke promises, customer had to do all the work 48
Difficulties in finding employment, few job options
25% 42
VR counselors or staff was unprofessional, rude, did not care
23% 28
Needed more help finding work, need to help customers find work, customer had to find a job on their own
20% 35
Waiting to receive services, have not received services
17% 27 19

20. 36% of customers reporting problems indicated that VR worked to resolve their problem(s).20

21. Customer Feedback
Customers were asked to provide additional feedback when they were not positive in their assessment.
Feedback was collected throughout the survey in response to 17 questions.
Customers could provide more than one response.
The following table summarizes the most common responses from 497 customers that provided additional feedback. 21

22. Customer Feedback 22 % Customers%
Customers
Counselor did not return calls, s or follow up
29%
130
Better communication needed
132
Had to leave multiple messages before getting a call back
21%
104
Counselor was not helpful or supportive
17% 89
Didn't receive job search help
16% 79
Did not receive employment, VR could not find me a job
15% 78
Changing counselors, switching too much, causes problems 84 22

23. Understanding the process if you disagree with VR about your services
Customers were asked if they were aware of the options available if they disagreed with VR’s decisions about the services they did or did not receive. 23

24. 75% of customers were aware they could contact their VR counselor in cases of disagreement.24

25. 66% of customers were aware they could contact the VR Unit Supervisor.25

26. 49% of customers were aware they could contact the VR Ombudsman Office.26

27. 50% of customers were aware they could contact Disability Rights Florida, Client Assistance Program.27

28. 58% of customers were aware they could appeal the decision(s) they disagreed with, if necessary.28

29. Satisfaction with Outside Vendors
Customer were asked to provide feedback on the services they received from outside vendors. 29

30. 62% of customers with an outside vendor were satisfied with the services provided.30

31. 18% of customers using an outside center reported problems with their vendor.31

32. The Most Commonly Mentioned Vendor Problems (2018 Q1):%
Customers
Did not receive or received little help from Vendor
69% 12
Vendor no help or help not effective
54% 11
Negative experience or dissatisfied
52% 10
Received no help in reaching plan or goals
47% 8
Need more guidance, support, explanation
25% 4 32

33. Current Employment Status33

34. 35% of current and former customers are working full or part time.
Q50: Finally, these last few questions ask about what you are currently doing. Are you currently...? %
Working full time, more than 35 hours per week
11%
Working part time
25%
Currently looking for a job
29%
In school or receiving job training
34%
Keeping house 2%
Currently unable to work 7%
Volunteering my time 34

35. 83% of working customers were satisfied with their current job.35

36. Most Commonly Mentioned Reasons for Being Dissatisfied with their Current Job:
Q52: If Dissatisfied with Job: Why do you say that?
(Multiple responses accepted) (n=27) %
Dissatisfied with aspects or parts of job
28%
Need a job aligned with skills, training
24%
Unpleasant work environment
15%
Received employment but not enough hours or pay
14%
Job is physically demanding
12% 36

37. *A separate presentation of these results has also been prepared.
In Depth: Data Sharing
*A separate presentation of these results has also been prepared.

38. Five organizations have agrees to share results.
FL Department of Vocational Rehabilitation
ME Division of Vocational Rehabilitation
ND Division of Vocational Rehabilitation
VT Division of Vocational Rehabilitation
VT Division for the Blind and Visually Impaired

39. Consumer Experience Core Metrics
FL VR
ME DVR
ND DVR
VT DVR
VT DBVI
Overall Satisfaction and Expectations 80 74 82 79
Experience with Services Provided by VR/DBVI 76 84 83
Experience with VR/DBVI Staff and Counselors 91 88 93
Communications with VR/DBVI Staff 78 87 86
Consumer Control and Involvement 81
Outcomes and Meeting Goals 85

40. Consumer Experience Core Metrics
FL VR
ME DVR
ND DVR
VT DVR
VT DBVI
Ease of the application process for VR/DBVI services. 87 81 82 86 88
Accessibility of the VR/DBVI office for someone with your type of disability. 93 92 95
Would you tell your friends with disabilities to go to the VR/DBVI program for help? (% yes) 98
Did you experience any problems with VR/DBVI or the services they have provided to you? (% no) 75 84 89
Satisfaction with Current Employment 80 73 85

41. Problems Faced by VR/DBVI Consumers

42. The types problems faced by VR/DBVI consumers
FL VR
ME DVR
ND DVR
VT DVR
VT DBVI
Issues communicating with staff or counselors, difficulty reaching staff or counselors
38%
35%
13%
48%
24%
VR Counselors or staff did not did not provide the help I needed, broke promises, customer had to do all the work
40%
17% 9%
Difficulties in finding employment, few job options
25%
37%
VR counselors or staff was unprofessional, rude, did not care
23%
42%
21% 6%
14%
VR program was not effective in helping customer or helping customer meet goals 7%
11%
34%
22%
Needed more support or services, had to fight to obtain services, needed more guidance 5%
12%
10%
18%

43. The types problems faced by VR/DBVI consumers.
FL VR
ME DVR
ND DVR
VT DVR
VT DBVI
Needed more help finding work, need to help customers find work, customer had to find a job on their own
20% 0% 5%
22%
Waiting to receive services, have not received services
17% 6%
29% 4%
Not assigned a counselor, switched counselors, counselors need to spend more time with customers 8%
13% 9%
Difficulties with VR counselor or staff, dissatisfied 2%
10% 1%
Customer involvement in and/or control over his or her VR experience
14%
Needed more information about programs and services, did not provide enough information
11%

44. Percent by Organization Offering Feedback
Consumer Feedback
Customers were asked to provide additional feedback when they were not positive in their assessment (Why do you say that?)
Feedback was collected throughout the survey in response to most questions.
Customers could provide more than one response.
The following table summarizes the most common responses from consumers that provided additional feedback.
Percent by Organization Offering Feedback
FL VR
61%
ME DVR
64%
ND DVR
58%
VT DVR
57%
VT DBVI
50%

45. Customer Feedback
FL VR
ME DVR
ND DVR
VT DVR
VT DBVI
Issues communicating with staff or counselors, difficulty reaching staff
55%
48%
38%
47%
Difficulties with paperwork and forms, need help completing forms, need alternate formats
34%
33%
35%
31%
36%
Difficulties in finding employment, few job options
16%
21%
10%
24%
25%
Needed more support or services, had to fight to obtain services, needed more guidance
11%
18%
22%
20%
VR program was not effective in helping customer or helping customer meet goals
15%
17%
19%
VR Counselors or staff did not provide the help I needed, broke promises, customer had to do all the work
26% 4%
13%

46. Customer Feedback
FL VR
ME DVR
ND DVR
VT DVR
VT DBVI
Needed more help finding work, need to help customers find work, customer had to find a job on their own
16%
17% 9%
10%
Needed more information about programs and services, did not provide enough information
18%
VR counselors or staff was unprofessional, rude, did not care
11%
19%
13% 8%
Waiting to receive services, have not received services
12% 7%
Not assigned a counselor, switched counselors, counselors need to spend more time with customers
15%
Location of VR facilities, need additional offices, expanded operating hours 4%

47. Awareness of the Client Assistance Program
Were you informed that if you had a problem with VR/DBVI, you could address it with the Client Assistance Program?
FL VR
ME DVR
ND DVR
VT DVR
VT DBVI
Yes
50%
71%
84%
68%
64% No
34%
16%
11%
23%
27%
Unsure
17%
13% 5% 9%
Total
100%

48. Improving the VR Consumer Experience Survey48

49. Assessing the experience of students in transition
With the implementation of WIOA, there is a greater emphasis on providing services for students in transition.
We are exploring changes to our VR consumer experience survey to include items focused on this population.
The questions on the next slide are a series we developed for Nevada VR.
We are also looking at other areas to assess experience:
Educational assistance
Educational attainment
Coordination of services (schools, VR, other agencies)
More detail on employment

50. While a client of Vocational Rehabilitation, did you receive any of the following pre-employment transition services? 
Job exploration counseling or career counseling
Work based learning experiences including a job, internship, or volunteering.
Counseling on opportunities to attend college, learn a trade, or get a professional certification after high school.
Work place readiness to help get ready for the challenges of work.
Instruction in self advocacy to help you live the life you want including peer mentoring.
FOR EACH SERVICE THEY RECEIVED ASK:
How valuable did you find this service?

51. 2018 Focus Groups 51

52. 2018 Focus Groups Two groups: 4/17 in Miami 4/18 in Jacksonville
Focus: Communicating with those going through the transition from high school to further education or the workforce
13 participants, 12 aged and 1 aged 25-34 52

53. Communication
Communication was seen as key to having a positive experience with VR.
Those who felt their counselor communicated well most often had a positive experience. Consumers valued both “hard” communication skills like returning calls promptly and “soft” communication skills like making consumers feel valued and motivated.
Most communication was through traditional methods such as phone, letters, and in-person meetings.

54. Opportunities
Reaction to a proposed “dashboard” to gather information in a central location was broadly positive.
Most felt that there currently was not a single source for written information about where they were in the VR process and what the next step was.
“Everything seems to be on paper and sometimes when I meet, I have to drive 30 minutes out just to meet for 10 minutes where I feel like some things I can know about if we had somewhere like an online site to where I know my status about where all my classes are because if ... I'll say this. If we have verbal conversations over the phone and Voc Rehab is making it mandatory for their employees to put in the status of their clients every time they talk then it keeps you updated. You know what I mean?”

55. Suggestions for Improving Communication
Easier opportunities for improving communication include:
Sending appointment confirmations via All 13 participants received on their phones and checked it frequently.
“She just gives me ... by mail the appointment. She'll be like, "This will come a day before the appointment. That's no time.”

56. Online Forms and Electronic Signatures
Allowing electronic signatures: Several mentioned traveling to VR offices to sign paperwork in person was difficult and time consuming.
“Cause coming for two minutes to sign a piece of paper and leave? That is pointless”
“I don't agree with coming and signing the paper. Meeting for 10 minutes and to sign a piece of paper? I gotta schedule a ride to come for 45 minutes?”
Several also mentioned that they would like to be able to fill out forms and applications online.
"Even just the application for the VR. If you can fill that out online because I have two friends that I've referred to, and they have yet to fill it out, and they have yet to fill it out. It's like, "Why can't they just fill it out online?" Everything is virtual now. Nothing is like we're doing right now."

57. Contacting Consumers Directly
Contact consumers directly rather than their parent or guardian when possible: They perceived VR contacting their parents as being treated like children and also adding an another layer to communication where messages could fall through the cracks.
"Don't call my mama. Because my mama don't ... Sometimes she would leave stuff but sometimes she don't. And so then I don't know what you're talking about but you all done had a whole conversation about what I need to do. Once you hit 18, baby, call me"
“Stay more in contact with the client instead of the clients' parents. Like it's something wrong. Is there something wrong with me? I'm old enough for you to call me.”
“When I can't pick up the phone at the moment, she calls my uncle, which I don't like that... I'm of age to do my own thing”

58. I’d be happy to answer any questions you might have.
Thank You
I’d be happy to answer any questions you might have.