1. DEVELOPMENT OF SURVEY FROM AN ITEM BANK For Counselors Motivational Interviewing Performance Management and Vocational Rehabilitation Program Evaluation By Ifeoma Chika Iyioke, PhD.

2. PERFORMANCE MANAGEMENT(PM)  Process by which an agency plans and monitors organizational practices and progress towards goals  PM involves assessment and measurement  Assessment and measurement are activities for collecting, analyzing and/or reporting information about the performance of an individual, group, organization, system or component.

3. PM IN VOCATIONAL REHABILITATION: CONDITION  Emphases on Program Outcome Measures  Standards and Indicators Outcome-Based

4. PM IN VOCATIONAL REHABILITATION: GAPS  Many confounding factors with outcome measures  Difficult to compare performance of state – agencies and counselors  Lack of focus on the program process and the primary function of the vocational rehabilitation

5. PURPOSE OF PRESENTATION  Introduce a strategy for vocational rehabilitation counselor PM by monitoring program process  An Item Bank for development of surveys for management of counselor performance of counseling  Connection with vocational rehabilitation program evaluation

6. FOOD FOR THOUGHT  Why measure?  Who or object to measure?  What to measure?  How to measure?

7. COUNSELOR FUNCTIONS  Consultant  Career/Vocational Counseling  Service Coordinator

8. COUNSELING FRAMEWORKS  Directive Counseling  Trait-Factor (Parson, 1909, 1989)  Person-Environment Fit (Dawis & Lofquist, 1984; Lofquist & Dawis, 1991)  Motivational Interviewing (Miller & Rollnick, 1991)  Non-Directive Counseling -  Client-Centered (Carl Rogers, 1951)

9. WHAT IS AN ITEM BANK?  A repository for survey/test questions  Survey/test questions organized and catalogued in a similar way to books in a library by topics  Contains also measurement characteristics of the survey/test questions (e.g., their reliability and validity)

10. EXAMPLE OF AN ITEM BANK See 

11. WHY AN ITEM BANK  An assessment tool for standardized performance management  Provides program administrators access to a variety of survey/test questions to cover a whole variety of situations  Can be used to develop effective and efficient surveys/tests for meeting the needs of criterion referenced evaluation  Facilitates assessing the performance of individual employees, monitoring changes, and for evaluating program innovations

12. DEVELOPING ITEM BANK FOR MOTIVATIONAL INTERVIEWING PM  Started with Literature Review - of Motivational Interviewing  What, Why, and How  Item Bank Design  Populating Item Bank - Creation of Items

13. MI: DEFINITION  An Assessment Method  Interview on motivation via open-ended questions)  An Intervention (stage-based and tailored)  A counseling method (directive and person- centered )  A method of communication (collaborative conversation and partner-like relationship)

14. MI: WHY Motivation  Readiness or priority  Willing or Importance  Ambivalence - Approach-avoidance conflict  Able or Confidence

15. MI: WHY Group A: Low Importance, low confidence - These people neither see change as sufficiently important nor believe that they could succeed in making such a change if they tried Group B: Low importance, high confidence – these people are confident that they could make the change if they thought it were important to do so but are not persuaded that they want to change Group C : high importance, low confidence – here the problem is not in willingness to change, for these people express desire to do so. The problem is low confidence that they could succeed if they tried Group D: High importance, high confidence – these people see it is important to change and also believe that they could succeed

16. MI: PRINCIPLES  Explore motivation  Express empathy Develop discrepancy  Roll with resistance  Support self-efficacy

17. MI: METHODS Enhance Importance OARS  Open-ended questions  Affirmations  Reflections  Summaries  Eliciting change talk

18. MI: METHODS Enhance Confidence  Review and affirm his or her own past successes in changing behavior  Give a choice of strategies to try, it is more likely that a person may find one that is acceptable as a starting point

19. DESIGN OF AN ITEM BANK  Define the purpose – formative or summative performance management and evaluation  Define content domain – Question classification and organization Scheme of an item bank  Motivational Interviewing: Why, Components, Principles and Methods

20. MI: ITEM BANK DESIGN

21. HOW TO POPULATE THE ITEM BANK  Compile and share previous questions  Create new survey questions

22. HOW TO USE THE ITEM BANK FOR COUNSELOR PM  Build the Plan for performance management into the customer satisfaction survey  The counselor should keep a record of the customers they serve and the level of motivation presented at the time they began to be counseled and at the end of the planning process.

23. HOW TO USE THE ITEM BANK FOR COUNSELOR PM  Question sampling and survey delivery procedures  Administration electronic vs. paper  The surveys are to be administered to the customers served by the counselor at intervals to monitor the counselor delivery of counseling services.

24. HOW TO USE THE ITEM BANK FOR COUNSELOR PM  Grading – Criterion-Referenced vs. Norm-referenced  Temporal Feedback on areas of strengths and weaknesses  Follow-up with training and remediation of areas of weaknesses

25. MI AND VOCATIONAL REHABILITATION PROGRAM EVALUATION: MODEL  Engagement & performance of motivational interviewing  consumer motivation  consumer engagement  consumer outcomes (e.g. employment, satisfaction)

26. MI PROGRAM EVALUATION QUESTIONS  To what extent did the counselor deliver the intended counseling style of motivational interviewing to the consumers?  Does relationship exist between the level of consumer perception of counselor engagement and performance of MI with their motivation, engagement, and outcomes (e.g. employment, satisfaction)?

27. EXERCISE  The counselor asked about the things I consider most important in life - ?  The counselor highlighted inconsistency between my actions and the things I consider most important in life?  The counselor listened to what I had to say -?  The counselor asked about the justifications for my actions-?  The counselor praised my efforts towards accomplishing my goal(s) -?

28. QUESTIONS ? Thank you for listening