1. Consumer Experience Research 2019 Q3 Results October 10, 2019
Florida Rehabilitation Council
Division of Vocational Rehabilitation Florida Department of Education
Consumer Experience Research 2019 Q3 Results October 10, 2019
Brian Robertson
Jotham Illumanati

2. Contents Survey Methodology Trends to Watch
Consumer Experience Core Metrics
Problems Experienced by VR Consumers and Feedback
Understanding the Process if You Disagree with VR About Your Services
Satisfaction with Outside Vendors
Current Employment Status
Those Whose Cases Were Unsuccessfully Closed
Youth in Transition Analysis
PEQATAC Summit Summary
Upcoming Research 2

3. Current Quarter Survey Methodology
Data collection took place from July 14, 2019 through October 7, 2019.
Over the course of data collection, 750 consumers completed surveys.
332 general VR consumers with open cases
157 general VR consumers - closed successfully
152 general VR consumers - closed unsuccessfully
109 transition youth consumers 3

4. Current Quarter Response Rates%
APPOR Response Rate (RR3)
16.9%
AAPOR Respondent Cooperation Rate (COOP3)
85.7%
AAPOR Respondent Refusal Rate (REF3)
4.3%
AAPOR – The American Association for Public Opinion Research.
Response Rate – The response rate is the number of completed interviews divided by the number of eligible reporting consumers in the sample.
Cooperation Rate – This represents the proportion of all cases interviewed out of all eligible consumers ever contacted. That is, the percent of identified respondents that ended up completing the interview.
Refusal Rate – The refusal rate represents the proportion of all cases in which the respondent refused to be interviewed, or broke off an interview, out of all respondents that were contacted and spoken with. 4

5. Trends to watch 5 Trending Positive in 2019 Q3
Overall Satisfaction and Expectations
Outcomes and Meeting Goals
Satisfaction with Current Employment
Satisfaction with Outside Vendors
The percentage of consumers reporting problems decreased.
The percentage those reporting problems with outside vendors decreased.
The percentage reporting problem resolution increased significantly.
Changing counselors, switching counselors too often is leading to problems NOT a top area mentioned by consumers providing feedback. 5

6. Trends to watch
Some metrics measuring understanding of the dispute resolution process continue to trend downward.
Less than half are aware they can contact the VR ombudsman.
Just two thirds know they can appeal a decision if necessary.
A little more than half are aware they can contact the Disability Rights Florida, Client Assistance Program. 6

7. 2019 Q3 Survey Results 7

8. Consumer Experience Core Metrics8

9. Florida VR Consumer Experience Dashboard
Measure
2018 Q4
2019 Q1
2019 Q2
2019 Q3
Overall Satisfaction and Expectations 78 79 81
Experience with Services Provided by VR 80
Experience with VR Staff and Counselors 90
Communications with VR Staff 77
Consumer Control and Involvement
Outcomes and Meeting Goals 84 82 85
Satisfaction with Current Employment 73 76
Satisfaction with Outside Vendors 74
All things considered, would you tell your friends with disabilities to go to the VR program for help? (% Yes) 91 94 93
Have you experienced any problems with VR or the services they have provided to you? (% No) 83 9

10. Overall Satisfaction and Expectations:
Q01: Overall, how satisfied are you with the Florida VR’s program? Q33: How satisfied would you say you are with the services provided by VR? Q35: Please rate how well VR services met your expectations. 10

11. Experience with Services Provided by VR:
Q09: How satisfied are you with the choice of services?
Q11: How satisfied are you with the choice of service providers? 11

12. Experience with VR Staff and Counselors:
Q19: How helpful is the staff of VR in helping you achieve your VR goal?
Q29: The VR staff treats me with dignity and respect. 12

13. Communications with VR Staff:
Q13: How satisfied are you with the information you were given about the choices you had?
Q15: How satisfied are you with how long it took your counselor to answer your questions and address your concerns?
Q21: How easy is it for you to contact your VR counselor? 13

14. Consumer Control and Involvement:
Q05: How satisfied are you with your involvement in your VR experience?
Q07: How satisfied are you with your choice of a vocational goal? 14

15. Outcomes and Meeting Goals:
Q25: The VR services I am receiving will help me become more financially independent.
Q27: The VR services I am receiving will help me become more independent, in general.
Q31: The VR staff will help me reach my job goal. 15

16. Satisfaction with Current Employment:
Q51: If working, thinking about your current job, how satisfied are you with what you are doing? Would you say you are...? 16

17. Satisfaction with Outside Vendors:
Q46: How satisfied were you with the services provided by OUTSIDE VENDOR? 17

18. Recommend VR:
All things considered, would you tell your friends with disabilities to go to the VR program for help?
(% reporting yes) 18

19. Experienced Problems:
Have you experienced any problems with VR or the services they have provided to you?
(% reporting no) 19

20. Problems Experienced by VR Consumers and Feedback
Consumers were asked if they experienced problems specifically with VR or the services they received from VR and reasons for dissatisfaction. 20

21. 19% of consumers experienced problems with VR or the services provided.21

22. The Most Commonly Mentioned Problems (2019 Q3):%
Counselor did not return calls, s or follow up
33%
Counselor was not helpful or supportive
28%
Counselor would not listen, dismissed concerns
22%
Received no help in reaching plan or goal
17%
Nothing achieved, VR not effective
Did not receive employment, VR could not find me a job
16%
% among those experiencing problems 22

23. % among those experiencing problems
48% of consumers reporting problems indicated that VR worked to resolve their problem(s).
% among those experiencing problems 23

24. Consumer Feedback
Consumers were asked to provide additional feedback when they were not positive in their assessment.
Feedback was collected throughout the survey in response to 25 questions.
Consumers could provide more than one response.
The following table summarizes the most common responses from 527 consumers that provided additional feedback. 24

25. % among those providing feedback
Consumer Feedback %
Counselor did not return calls, s or follow up
28%
Better communication needed
23%
Had to leave multiple messages before getting a call back
22%
Counselor was not helpful or supportive
14%
Get voic , never answers the phone
Did not receive employment, VR could not find me a job
13%
Staff did not return calls, s or follow up
% among those providing feedback 25

26. Understanding the process if you disagree with VR about your services
Consumers were asked if they were aware of the options available if they disagreed with VR’s decisions about the services they did or did not receive. 26

27. 79% of consumers were aware they could contact their VR counselor in cases of disagreement.27

28. 63% of consumers were aware they could contact the VR Unit Supervisor.28

29. 47% of consumers were aware they could contact the VR Ombudsman Office.29

30. 55% of consumers were aware they could contact Disability Rights Florida, Client Assistance Program.30

31. 67% of consumers were aware they could appeal the decision(s) they disagreed with, if necessary.31

32. Satisfaction with Outside Vendors
Consumer were asked to provide feedback on the services they received from outside vendors. 32

33. % among those receiving services from an outside vendor
85% of consumers with an outside vendor were satisfied with the services provided.
% among those receiving services from an outside vendor 33

34. % among those receiving services from an outside vendor
9% of consumers using an outside center reported problems with their vendor.
% among those receiving services from an outside vendor 34

35. Feedback about Vendors (2019 Q3):%
Vendor did not return calls, s or follow up
33%
Vendor no help or help not effective
28%
Did not receive or received little help from Vendor
Negative experience or dissatisfied
13%
Vendor would not listen, dismissed concerns
10%
% among those not satisfied or that experienced problems with an outside vendor 35

36. % among those receiving services from an outside vendor
70% of consumers felt that both the VR and their vendor(s) kept them informed and up-to-date.
% among those receiving services from an outside vendor 36

37. What worked well during experiences with outside vendors:
VEND05: In thinking about your experience with the outside vendor what worked well for you?
(Multiple responses accepted) %
All, everything
10%
Communication and accessibility of counselor 8%
Having a support system with a counselor to determine goals and achieve that goal 6%
Getting a job
Talking with a counselor, having a relationship
% among those receiving services from an outside vendor 37

38. Challenges consumers faced while working with outside vendors:
VEND06: What has been the most challenging part of your experience working with the vendor?
(Multiple responses accepted) %
Process of seeking employment
12%
Transportation
Application, acceptance, getting started, waiting
11%
Personal barriers, being independent, coping with disability 6%
Job or school is challenging
Deciding on a career goal
% among those receiving services from an outside vendor 38

39. Current Employment Status39

40. 40% of current and former consumers are working full or part time.
Q50: Finally, these last few questions ask about what you are currently doing. Are you currently...? %
Working full time, that is, more than 35 hours per week
14%
Working part time
26%
In school or receiving job training
42%
Currently looking for a job
31%
Keeping house
Volunteering my time
Currently unable to work
10% 40

41. 79% of working consumers were satisfied with their current job.
% among those with currently employed 41

42. Feedback on Employment%
Not a career move, only temporary job
48%
Dissatisfied with aspects or parts of job
24%
Bored with job, been at job too long, burnt out, need change
21%
Too few hours
16%
Low pay, does not meet financial need
Varying hours, unreliable hours, scheduling problem
15%
% among those with not satisfied with current employment 42

43. 82% of working consumers find their job meaningful.
% among those with currently employed 43

44. Those Whose Cases Were Unsuccessfully Closed
Consumers were asked if they requested their case be closed and reasons for doing so. 44

45. % among those with cases closed unsuccessfully
45% of consumers with unsuccessful cases asked for their case to be closed.
% among those with cases closed unsuccessfully 45

46. (Multiple responses accepted, most common responses)
Consumers most often asked for their case to be closed due to an injury or disability, others said they were dissatisfied or VR services were ineffective.
(Multiple responses accepted, most common responses) %
Unable to work due to injury or disability
27%
Dissatisfied with VR, services not effective
22%
Have other priorities, needed a break
14%
VR provided all promised services
12%
Moved away from VR
11%
% among those with cases closed unsuccessfully asking for case to be closed 46

47. UNSUC01B: Why do you believe your case was closed?
Among those that did not ask for their cases to be closed, the most frequent reason they felt their case was closed was a lack of communication.
UNSUC01B: Why do you believe your case was closed? %
Lack of communication
20%
Transportation issues, customer couldn't keep appointments 5%
Have other priorities, needed a break
Dissatisfied with VR, services not effective
Health issues
Completed program, received all expected services 4%
% among those with cases closed unsuccessfully that did not ask for case to be closed 47

48. % among those with cases closed unsuccessfully
59% of consumers whose cases closed unsuccessfully received a closure letter with the reason for closure and their rights to appeal.
% among those with cases closed unsuccessfully 48

49. Youth in Transition Analysis49

50. Florida VR Consumer Experience Dashboard Comparisons
Measure
General VR
Youth in Transition
Overall Satisfaction and Expectations 80 87
Experience with Services Provided by VR 83
Experience with VR Staff and Counselors 89 96
Communications with VR Staff 78 82
Consumer Control and Involvement 81
Outcomes and Meeting Goals 84
Satisfaction with Current Employment 76
All things considered, would you tell your friends with disabilities to go to the VR program for help? (% Yes) 94 98
Have you experienced any problems with VR or the services they have provided to you? (% No) 95 50

51. % among youth in transition experiencing problems
Youth report problems at a significantly lower rate than consumers overall (5.4% vs 19%). But they experience different problems. %
Vendor would not listen, dismissed concerns
27%
Counselor did not return calls, s or follow up
21%
Had to leave multiple messages before getting a call back
Nothing achieved, VR not effective
Staff provided no cooperation, no help
Listen to customer, understand needs, wants, ability
17%
Received no help in reaching plan or goals
Did not receive employment, VR could not find me a job
Distance, too far away
% among youth in transition experiencing problems

52. % among youth in transition
Most Pre-ETS recognized at least one of the services they were receiving. Just one tenth reported not using any of the services FLVR provides. %
Workplace readiness to help get ready for the challenges of work
66%
Work based learning experiences including a job, internship, or volunteering
55%
Job exploration counseling or career counseling
63%
Instruction in self-advocacy or peer-mentoring to help you live the life you want
64%
Counseling on opportunities to attend college, learn a trade, get professional or industrial certification
43%
Did not use any of these services 9%
% among youth in transition

53. % Finding Service Valuable % among youth in transition using service
Almost all youth using these services programs found the services valuable.
% Finding Service Valuable
Job Exploration and Career Counseling
98%
Work-based Learning Experiences
97%
Opportunities for further education
95%
Workplace Readiness Services
Instruction in Self-Advocacy
99%
% among youth in transition using service

54. PEQATAC Summit Summary54

55. PEQATAC Summit Conference Presentation: Understanding Pre-ETS Populations: What to Ask and What to Know
Our presentation focused on designing research for Pre-ETS populations and the challenges we face in surveying this population.
The main goal of the presentation was to get people thinking about surveying the Pre-ETS population and answer three questions during a roundtable discussion:
What questions do we need to ask of the PRE-ETS population to provide the information you need?
What is the most effective way of communicating with your Pre-ETS population?
How do we get the Pre-ETS population to recognize the entity providing their services?
What did we learn? 55

56. PEQATAC Summit Conference Presentation: Understanding Pre-ETS Populations: What to Ask and What to Know
Unfortunately, there was no clear approach about what to ask, but some topics were suggested.
Coordination with all parties (vendors & school based special Ed)
Exploring all options & facilitating transition
Putting youth’s priorities & goals first
Educational attainment & aspirations
Connecting to resources via media that is accessible/modern
Common performance measures (Credentials)
Pace of services vs. expectations
There was some consensus that the questions should be more youth focused. 56

57. PEQATAC Summit Conference Presentation: Understanding Pre-ETS Populations: What to Ask and What to Know
In terms of how to survey the Pre-ETS population most agreed they are most likely to respond to electronic communications.
Use texts to elicit cooperation to an online survey (but there are potential legal issues).
Use .
If the survey is to be done by telephone, let them know ahead of time by text or .
Letters/postcards could be tried as a means of pre-notification but may not work (youth don’t normally communicate in this fashion).
How do we get the Pre-ETS to identify FLVR as the provider of services?
Increase brand awareness
MDR will be working with VT DVR on a series of focus groups that are designed to explore what Pre-ETS think of as the most important aspects of their experience with VR with the goal of developing an online survey specifically focused on Youth in Transition.
I am sure they will share results once the research is done! 57

58. Upcoming Research 58

59. In 2020, MDR will work with FRC and FLVR to develop and conduct two focus groups.
The topic area will be the Pre-ETS populations
Factors that go into working with that population
How are services being received.
What is the impact?
How can services be improved?
Over the next few months, we will work with FRC and FLVR to develop topics
The FRC will need to decide from whom we should get feedback: students, parents, teachers?
Finally, I am developing a summary of research across all of the VR agencies for which we have conducted research.
I hope that you are willing to share your results with other agencies.
Empty 59

60. I’d be happy to answer any questions you might have.
Thank You
I’d be happy to answer any questions you might have.