1. Quality By Design Statewide Human Services Workforce Evaluation Using an Integrated Framework American Evaluation Association Conference San Antonio, TX November 2010 California Social Work Education Center (CalSWEC) UC Berkeley, School of Social Welfare

2. CalSWEC Created in 1990, the California Social Work Education Center (CalSWEC) is a consortium of California’s –20 accredited social work graduate schools,20 accredited social work graduate schools –the 58 county departments of social service and mental health, the California Departments of Social Services (CDSS) and Mental Health (CDMH),Social Services (CDSS)Mental Health (CDMH) –the California Chapter of the National Association of Social Workers, professional associations, and foundations.California Chapter of the National Association of Social Workers It is the nation's largest coalition of its kind working together to provide professional education, student financial aid, in- service training, and workforce CalSWEC facilitates the integration of education and practice to assure effective, culturally competent service delivery in the human services.

3. Goals The partnership’s goals are to: –re-professionalize public human service through a specialized education program for public human services, –develop a continuum that connects pre service education to in service training, –engage in research and evaluation to develop evidence based practices –advocate for responsive policies and resources to support practice improvement and client outcomes.

4. Today’s Discussion Review of Evaluations thus far: –Title IV-E Stipend Program—MSW/BASW program that provides specialized preparation for work in Public Child Welfare Agencies –In-Service Training Program for Child Welfare—develops common and specialized in-service curriculum for Public Child Welfare Agencies –Mental Health Program—MSW program that provides specialist preparation for work in Mental Health Agencies Discussion of where we are headed: –A conceptual model for integrating the evaluations using theoretical constructs and longitudinal design as guiding principles –Evaluation goal to improve our ability to better assess the impact of these programs on practice and client outcomes.

5. Title IV-E Stipend Program

6. Overall Program Goal To recruit and prepare a diverse group of social workers for professional careers in public human services, with child welfare emphasis.

7. Title IV-E Research Questions 1.Is the curriculum being delivered as intended? 2.Are the students learning the curriculum? 3.To what extent are the graduates able to practice what they learned in the program within the child welfare agencies? 4.Do CalSWEC IV-E graduates remain in public child welfare? 5.What effects, if any, has the project had on the public child welfare agency and workforce? 6.Does the IV-E program have effects on child and family outcomes?

8. Are we meeting our overall goal? Entry/Graduation Survey 10 cohorts panel study of all MSWs in participating CalSWEC Universities Designed to compare IV-E MSWs with Non-MSWs. Measured diversity, attitudes toward poverty, motivations and career goals, preferences for field of practice and client population.

9. Curriculum Snapshot Every two years, the Snapshot captures classroom & field program elements at the school level for 20 member programs Curriculum competency infusion & partnership activities with county and nonprofit agencies are included Schools also report on curriculum delivery methods & program structure Is the curriculum being delivered as intended?

10. New Graduate Survey Annual survey of recent graduates, completed six months into their employment Asked how well their MSW programs prepared them for public CW social work Graduates rank the quality of a variety classroom & field experiences Also asked what content they believe is most important for beginning SWs to know Are the Students Learning the Curriculum?

11. Findings at 6 months to 1.5 years Post Graduation The New Graduate Survey is done each year as the MSW classes graduate and begin their first jobs. It is delivered as a Web survey. Currently this component of the evaluation is an extension of the Curriculum Snapshot (In School Evaluation). –The classroom experiences that best prepared graduates for their work in public child welfare were practical techniques (91%), followed closely by clinical education (85%); then vignettes/role- playing (79%) and diversity education (69%). –The field experiences that best prepared them were hands-on direct practice experience (97%), then shadowing/observing (93%), followed by mentoring/guidance (84%) and supervision and prior experience (tied at 83%). –Topics graduates most often noted their programs covered especially well were diversity/sensitivity education, child welfare policy and advocacy/social justice, followed by making assessments and relationship building and gaining client/family participation. –Respondents noted that some MSW program components could have been done differently or were handled less than adequately by their programs: Field Placement Quality at nearly 85%, an increase of 15% over last year’s survey. Next noted was Working with the court system (82%), similar to the prior year’s responses. Also similar were the frequently cited Case management (76%), and Substance abuse education (nearly 79%). Clinical skills for assessment (73%) and selecting appropriate intervention models (71%).

12. To what extent are the graduates able to practice what they learned in school within the public child welfare agencies? and Do CalSWEC IV-E graduates remain in public child welfare?

13. 3-Year Post Graduate Survey & Interview Ongoing study of CalSWEC IV-E grads 3 years post graduation. Measures –Retention Status –Work conditions –Work experiences –Work attitudes –Demographics

14. Findings at 3 years Post Work Obligation Those who stay in child welfare differ significantly from those who leave after their work obligation is completed. –Worker characteristics (being of mixed ethnicity and cohort) and previous county employment (except in the last model when county differences were controlled for) predicted retention. – In addition to worker characteristics, at least one variable from each of these categories predicts retention: extrinsic job factors (salary, hours and supervisor support), intrinsic job factors (level of success), and response-to- job factors (client-related stress). –One worker characteristic (cohort) and two response-to-job factors (burnout- emotional exhaustion and visit-related stress) were significantly associated with leavers. Supervisor support significantly increases the odds of several types of job satisfaction, except client-related job satisfaction. Peer support increased the odds of client-, growth-, office-, and salary-related job satisfaction.

15. 5-Year Post Graduate Survey & Interview Ongoing study of CalSWEC IV-E grads 5-6 years post graduation Examines the graduates’ career paths with measures of: –Retention Status –Jobs, roles, and responsibilities –Work experiences –Demographics

16. Findings from the Retrospective Study on the 5 – 6-Year-Post Grads 154 of the Retrospective respondents fell into this category –98 were county employees while IV-E –132 were women –51 were Caucasian –70 were bilingual Due to missing data we could not isolate these respondents and perform a separate analysis. However, we are now conducting a separate survey just for those who graduated 5-6 years ago, based on findings from the Retrospective study.

17. Findings: Retrospective Study—Training & Licensure Having access to training more than twice a year is associated with longer retention. Getting agency support for licensure is associated with longer retention.

18. Findings: 8 to 10+ years (The Retrospective Study) The data showed distinct drop-offs in retention of Title IV-E graduates at 3 & 10 years. Not surprisingly, length of service is longer in the field of child welfare than in the first position.

19. Findings: Retrospective Study— Supervisory Position Having at least one supervisory position is associated with longer retention. (Note how the distance between the curves gets larger at about 5-6 years.) However, holding an administrative or managerial position is not. What about those who are still in direct service positions?

20. Workforce study Census survey of the California child welfare workforce conducted every 3-5 years How do agencies organize workers and assignments? What are the education levels of child welfare workers? Where are unmet needs in workforce diversity and language capabilities? What agencies are underserved by master’s level social workers? How many current workers are participated in and are interested in IV-E? What effect has the Title IV-E program had on public child welfare agencies?

21. Findings—Education Level

22. Findings-Language Of the survey respondents, more than one in four speaks a language other than English, primarily Spanish Los Angeles County has the highest proportion of Spanish-speaking staff By comparison of the 510 Title IV-E participants, 68.4% (349) state they speak another language besides English on the job

23. What effects does the IV-E Program have? The unanswered question. Does the IV-E Program have effects on child and family outcomes? We sure hope so……

24. Child Welfare In Service Training Evaluation in California: Where We’ve Been and Where We’re Going

25. Why evaluate child welfare training? Almost every practice improvement initiative involves training Very few public child welfare agencies systematically evaluate their training “The tail wags the dog” – The evaluation process informs the curriculum, which in turn informs practice.

26. Key Partners in California’s Child Welfare In-Service Training Evaluation

27. Key Steps 2001 - Macro Evaluation Team convenes, begins planning. 2002 - PIP mandates development & Implementation of Framework for Training Evaluation [1][1] 2004 – Framework Completed and Adopted 2004 to 2009 – Partners implement Framework Late 2008/Early 2009 – Begin Strategic Planning process for next 3 year period Fall 2009 – Implementation of next strategic plan commences [1][1] Parry, C. & Berdie, J. (2004). Training evaluation framework report. Berkeley, CA: California Social Work Education Center.

28. Timeline of Activities

29. How did we design evaluation efforts to determine if training had an effect? Framework for Evaluation: Levels of Training Eval in CA –Level 1: Tracking training (Demographics) –Level 2: Formative evaluation of training courses (course level) –Level 3: Satisfaction –Level 4: Trainee knowledge acquisition –Level 5: Skills acquisition (as demonstrated in class) –Level 6: Transfer of learning (TOL: use of knowledge and skill on the job) –Level 7: Agency/client outcomes

30. Why Evaluate at these Levels? Chain of Evidence: –Establishes a linkage between training and desired outcomes for the participant, the agency, and the client such that a reasonable person would agree that training played a part in producing the desired outcome.

31. Let’s see how far we’ve come…

32. What do we know now? (Summary of Progress/Results by Level) Level 1: Demographic data – 4,608 new line worker hires from mid-2005 thru 6/30/10 540 new supervisors Level 2: Formative evaluation data – Comprehensive observation-based data used to revise 7 standardized statewide curricula Level 3: Satisfaction data – collected at regional level only

33. What do we know now? (Summary of Progress/Results by Level) Level 4: Knowledge test data – –Pre & Post (3 multi-day curricula) –Post-only (1 single-day curriculum) –Significant learning from pre- to post-test (See sample report) –IV-E Effects

34. Examples of Findings: Training Evaluation of IV-E vs. non IV-E Family Engagement & Case Planning Test Versions: 1.25, 1.26. Date Range: Classes conducted between January 2008 and December 2008. N=743 complete pairs of pre‐ and post‐tests. Gains from pre‐ to post‐test are statistically significant for both Title IV‐E and non‐IV‐E participants. Title IV‐E participants scored higher at pre‐test and post‐test than non‐Title IV‐E participants. These differences are statistically significant.

35. Summary of Progress/Results, cont’d Level 5: Skill assessment in classroom (2 Curricula with embedded skills assessment) – Majority of new CWWs made 3 out of 4 correct decisions when asked to indicate whether or not child maltreatment occurred. Level 6: Transfer of Learning – Completed regional studies on Transfer and Field Training. (See 2009 White Paper, noted at end of this PPT.) Level 7: Outcomes level – Focused on building blocks/chain of evidence

36. Where are We Going?

37. Where Are We Going (by Level)? Level 1: Demographic data – continue collection Analyze lineworker, supervisor, IV-E trainee test data, added section on expectations for the work Level 2: Formative evaluation data – continue collection & integration. Revise forms for observers/trainers; develop forms for e-learning platform. Level 3: Satisfaction data – continue regional/county collection

38. Where Are We Going? (cont’d…) Level 4: Knowledge tests – continue –Move toward more diagnostic testing (focused on key knowledge) –Continue to analyze differential functioning of items based on demographic characteristics –Pilot study to assess possible effect of stereotype threat in trainee test performance (SDSU site) –Explore trainer-level differences in test item performance –Compare/monitor Title IV-E vs. non-IV-E performance

39. Where Are We Going? (cont’d…) Level 5: Skill assessment in classroom - continue –Continue to analyze differential performance: Physical Abuse and Sexual Abuse Identification –Pilot SDM TM version of the Critical Thinking in Child Welfare Assessment: Safety, Risk & Protective Capacity curriculum (assessment curriculum specific to CA Assessment tool) –Revise and continue Casework Supervision module –Pilot neglect scenario as part of an embedded eval (SDSU site)

40. Where Are We Going? (cont’d…) Level 6: Transfer of Learning – Feasibility study Level 7: Outcomes – Feasibility study, and Continue building Chain of Evidence to link training to outcomes. Other training evaluation projects, outside of levels 1-7: –Attitudes/Values Evaluation re CMI 1 (physical abuse): pilot by SDSU site. –Attitudes/Values Evaluation re CMI 2 (Sexual Abuse): Collaboration with UNC School of Medicine. –Trainer Evaluation –Quality Assurance

41. MENTAL HEALTH PROGRAM

42. MENTAL HEALTH WORKFORCE SHORTAGES 2001: Approx. 4 million adults in California needed MH services 1 2004-05: 600,000 clients received county MH services 2 2008: Vacancy rate for MH providers in California = 20- 25%; higher in rural areas. 3

43. CALIFORNIA MENTAL HEALTH SOCIAL WORK WORKFORCE TRENDS 2000 – 2010: # of social work graduate programs increased from 13 to 20 Council on Social Work accredited programs or accreditation candidates. 2007: 15,560 LCSWs represent 21.5% of all licensed MH professionals in California. This is the second largest group after Marriage and Family Therapists (37%), followed by psychologists (19%), psych techs (13%), psychiatrists (9%), and psychiatric MH nurses (.5%). – 66% of LCSW’s reside in urban communities.

44. CalSWEC MENTAL HEALTH INITIATIVE 1993 – Mental health directors, social work educators, and practitioners began to explore creating a program modeled on the Title IV-E program. 2003 - CalSWEC Board members re-ignited efforts to: –Develop a curriculum to train graduate students for careers in public mental health services –Create partnerships between education and provider communities to recruit and deploy more social workers into the mental health system –Search for resources to support stipends and program development.

45. MENTAL HEALTH SERVICES ACT (MHSA or PROP. 63) - 2004 1% tax on income >$1 million to transform the public mental health system 2005 revenues = $900 million Expands mental health services in five program areas: (1) Children’s systems of care; (2) Adult and Older Adult systems of care (3) Prevention and early intervention (4) Innovative programs (5) Education and Training

46. MHSA WORKFORCE, EDUCATION, AND TRAINING FUNDS (WET) GOALS Address critical MH workforce shortages Retool the existing workforce to create and sustain system transformation Create/strengthen career pathways for consumers and family members.

47. MENTAL HEALTH PROGRAM CalSWEC receives MHSA WET funding of $5.8 million annually for stipends and program activities that is distributed to Schools of Social Work. –Stipends ($18,500) for up to 196 final-year MSW students –Related program and indirect costs at each school. Schools develop and implement curricula that teach a set of core competencies identified by CalSWEC stakeholders. Stipend students have a one-year payback obligation to work in a county or contract mental health agency or to pay the loan back in cash with interest.

48. MENTAL HEALTH PROGRAM CURRICULUM COMPETENCIES Culturally and Linguistically Competent Generalist and Advanced CLC MH Practice Social Work Practice / MH Practice Human Behavior and the Social Environment / MH Practice General Social Work / Mental Health Management, Policy, Planning and Administration

49. MHP EVALUATION UC Berkeley School of Social Work Outcome Study: Cohort characteristics Payback obligation performance Post-payback employment Methods: - Analysis of administrative data - Paper and interview surveys of project coordinators Loma Linda University, Dept. of Social Work & Social Ecology: Implementation of the curriculum competencies Preparation of MHESP graduates to work in recovery-oriented mental health systems Methods: - Self-report surveys of schools - Content analysis of syllabi - Survey of graduates (IRB approved) - Survey of employment supervisors

50. FINDINGS The program contributes to the diversity of the mental health workforce. –56% of the 2005 – 2010 cohorts are ethnically and culturally diverse; Latino is the largest population –57% of the cohorts speak at least one language in addition to English; Spanish is spoken most often. Graduates are meeting their payback obligations –92% of the graduates of the 2005-08 cohorts completed their employment obligations; 51% worked in county-operated mental health agencies, and 41% worked in contract agencies. Graduates are continuing their careers in public mental health. –69% of the 2005-08 cohorts were still at their payback agency in 2010. –54% were in county agencies and 46% were employed in contract agencies.

51. FINDINGS Continued… Curricula in schools of social work are changing to address mental health competencies –A survey of the MHP schools of social work identified a number of strategies being used to implement the mental health competencies with a particular focus on content about recovery, and inclusion of consumers and family members on advisory boards and as participants in classroom presentations. Schools are creating pathways for greater collaboration among faculty and with agencies –Schools have developed specialized seminar courses for MH stipend students, increased involvement of MH Stipend Project Coordinators in their school/department curriculum committee meetings and increased interaction with local county mental health agencies. Analysis of data from the graduate surveys and supervisor interviews is in process.

52. EVALUATION AUDIENCES or Who cares/ should care? Key stakeholders Funders – current and potential Policymakers –Elected –Appointed Advocates Researchers Taxpayers

53. NEXT STEPS Build consensus about central role of program evaluation Strengthen capacity for more sophisticated and rigorous evaluation strategies within the CalSWEC framework. Broaden dissemination of our story and evaluation results. Find resources to support expanded evaluation efforts.

54. Where we are headed: Integrated Evaluation Framework

55. Main Assumptions and Points of Evaluation The curriculum, delivered as intended, will prepare social work students for work in public human services. Graduates are better prepared for public human services practice than others. The number of MSWs is increased in the state’s public human services staff population. Graduates are staying in public human services agencies after their work obligation is completed. Graduates who stay in public human services have been able to affect practice, program, & policy through experience and promotion. Graduates are involved in the educational process through becoming field instructors, sitting on admissions panels etc.

56. Psychological Theories: Locus of Control (Spector and Michaels (1986) Learning Theory (Miller, 1996) Organizational Turnover Theory (Hom et al., 1992). Sociological Theories: Social Comparison Theory (Geurts et al., 1998) Social Exchange Theory Social Ecological Theory (Moos, 1979). Economic theoretical: Human Capital, Dual Labor Market Models, (Miller, 1996). Professionalism Bureaucracy Organizational citizenship behavior Goodness of fit Social Exchange Theory Perceived Organizational Commitment (Rhoades & Eisenberger, 2002) Organizational Social Context (Glisson) Integrated Turnover Model (Mowday, Porter, & Steers, 1982) Intrinsic/Extrinsic Job Factor Theory (Benton, in progress) Locus of Control Theory (P. Spector) CalSWEC mission & program objectives Control theory (Buckley, Fedor, Veres, Wiese, & Carraher, 1998) Occupational sociology (Landsman, 2001) Role theory Expectancy theory Locus of Control Theory (P. Spector) Reactivity in research design Effects of feedback Adult learning theories Education Models of the schools. 8 years to 10 + years5 to 6 years3 years6 months to 1.5 year post graduation (Core Training) In School Theoretical Frameworks & Literature

57. Is graduate still working in public agency? Stem: To what extent do public sector workers perceive an influence on policy, organization and/or program development/evaluati on? Q2. To what extent have graduates career goals been realized in public sector employment? Q3. What are the graduate’s future goals? A. Do CalSWEC graduates stay longer than non CalSWEC? Why or why not? Is graduate still working in public agency? Stem: What are the career paths of public sector workers? Q2. Are they able to practice social work skills, knowledge & values? Q3. Are job expectations being met? Q4. Does agency/organization support graduates career path/goals? A. Do CalSWEC graduates stay longer than non CalSWEC? Why or why not? Is graduate still working in public agency? Stem: What factors contribute to retention of public sector workers? Q2. Are they able to practice social work skills, knowledge & values? Q3. Are job expectations being met? Q4. Does agency/organization support graduates career path/goals? A. Do CalSWEC graduates stay longer than non CalSWEC? Why or why not? Is graduate still working in public agency? Stem: To what extent are the graduates prepared for work in public agency? Q2. Are they able to practice what they learned? (core) Q3. What are the graduates’ expectations for the job? Q4. What are the graduates’ career goals? A. Do CalSWEC trainees’ knowledge levels skill acquisition differ from non- CalSWEC trainees? (core) Stem: Are the competencies infused into the MSW curriculum? Q2. Does use of the student competency assessment raise degree of infusion of competencies in school curriculum? Q3. To what extent do the schools models reflect adherence to CalSWEC’s program requirements? RESEARCH QUESTIONS 8 years to 10 + years5 to 6 years3 years6 months to 1.5 years (Core Training) In SchoolStudy Interval Research Questions—The Model

58. Exchange Relationship Organizational culture and climate Organizational politics Work schedules Mentorship Organizational responsiveness Supervisor/Manageme nt support Professional development Exchange Relationship Promotion/Professional development Organizational culture and climate Organizational politics Available resources Resources Mentorship Organizational responsiveness Supervisor/Management support Exchange Relationship Job stressors & job satisfiers Org responsiveness Role conflict Available resources Organizational culture & climate Supervisor support/ quality supervision Professional development Salary Attitudes and preparation for training and professional development Demographics: size, age, model CalSWEC instituted curricula (workshops, field class) vs. pre- existing curricula (core classes, field placement) Extent to which the students are exposed to the curriculum competencies 8 years to 10 + years 5 to 6 years3 years6 months to 1.5 year (Core Training) In School Organizational Variables

59. Survival time to leave Survival time to promotion Compensation Demographic Self-efficacy Work locus of control Autonomy Demographics Self-efficacy Work locus of control Autonomy Commitment to service profession Career Goals Individual factors: Demographic characteristics Commitment to a service profession Career Goals Work Locus of Control Response to job factors: Job satisfaction Met expectations Attempts to change conditions/office Knowledge level /acquisition Skill mastery Demographic characteristics Job expectations Commitment to a service profession Career Goals Demographic: previous experience in CW, race, gender, etc 8 years to 10 + years5 years3 years6 months to 1.5 year post graduation (Core Training ) In School Individual Variables

60. Integration: IVE Stipend Program and In-Service Program Evaluation All newly hired public child welfare workers are required to complete Core Training within two years of hire. (Legislation) Demographic information is collected at the time newly hired child welfare workers take Core Training as part of In-service Evaluation. As of 2010, we have added questions about career goals, job expectations and social commitment to the Demographic Questionnaire which will allow us to compare IV-E with non IV-E child welfare workers and follow them over time.

61. Integration: IVE Stipend Program and In-Service Program Evaluation The inservice evaluators are already able to compare IV-E with non IV-E on certain knowledge tests. Pre- Post-Evaluations are done with Child & Youth Development, Placement & Permanency, Family Engagement in Case Planning & Case Management. Embedded Evaluations are done for Child maltreatment Identification & Neglect and Sexual Abuse.

62. Integration: Mental Health Program and Child Welfare Programs Ongoing development of evaluation capacity to use the framework. –Database development –Identifying other applicable theories –Resource development –Curriculum evaluation

63. Key points Leverage resources of multiple evaluation efforts Utilize common evaluation framework to tie evaluation efforts together Dissemination of findings –Contractual obligation –Stakeholders –Policy

64. Contact Information Title IV-E Stipend Program Chris Mathias, Director cmathias@berkeley.edu Sherrill Clark, Research Specialist sjclark@berkeley.edu Susan Jacquet, Research Specialist sjacquet@berkeley.edu Elizabeth Gilman, Curriculum Specialist egilman@berkeley.edu

65. Contact Information In-Service Training Program Barrett Johnson, Director barrettj@berkeley.edu Leslie Zeitler, Training & Evaluation Specialist lzeitler@berkeley.edu

66. Contact Information Mental Health Program Gwen Foster, Director gwen77f@berkeley.edu Beverly Buckles, Primary Investigator, Dean, Social Work & Social Ecology, Loma Linda University bbuckles@llu.edu James Midgley, Harry & Riva Specht Professor, UC Berkeley School of Social Welfare jmidg@berkeley.edu

67. Title IV-E Stipend Program References Reference for CalSWEC Goal – http://calswec.berkeley.edu/CalSWEC/CalSWEC_FactSheet.pdf>http://calswec.berkeley.edu/CalSWEC/CalSWEC_FactSheet.pdf –http://calswec.berkeley.edu/CalSWEC/CalSWEC_FactSheet.pdfhttp://calswec.berkeley.edu/CalSWEC/CalSWEC_FactSheet.pdf Reference for CalSWEC Research Questions –Grossman, B., Laughlin, S., & Specht., H. (1992). Building the commitment of social work education to publicly supported social services: The California model. In K.H. Briar, V.H. Hansen, & N. Harris (Eds.), New Partnerships: –Proceedings from the National Public Child Welfare Symposium, 1991. Florida International University, North Miami Institute on Children and Families at Risk. For more CalSWEC publications based on CalSWEC studies see: –http://calswec.berkeley.edu/CalSWEC/ResearchPubs.htmlhttp://calswec.berkeley.edu/CalSWEC/ResearchPubs.html

68. In-Service Training References Multiple reports and other information on CA’s In-Service Training Evaluation efforts are on the CalSWEC website, including: –Evaluation of the California Common Core for Child Welfare Training: Implementation Status, Results and Future Directions (December 2009), –Original Framework for Child Welfare Evaluation report (2004) –12 Years of Proceedings from the National Human Services Training Evaluation Symposia –Many other concept papers and reports http://calswec.berkeley.edu/CalSWEC/CWTraining.html CalSWEC also posts child welfare Common Core curriculum online, and can share associated training evaluation materials (validated test items, demographic forms, code books, etc.) http://calswec.berkeley.edu/CalSWEC/RTA.html

69. CalSWEC Mental Health Program References All references are from Lok, V. and Chapman, S. 1.Lund, L. (2005). Mental Health Care in California Counties: Perceived Need and Barriers to Access, 2001. Center for Health Statistics 2.Jew-Lochman (2008). Health Data Summaries for California Counties, 2006. Department of Health Services 3.California Department of Mental Health. (2008). Mental Health Services Act Five-Year Workforce Education and Training Development Plan. Retrieved April 23, 2008 from http://www.dmh.ca.gov/Prop63/MHSA/Workforce_Education_and_Training/ FiveYearPlan_5-06-08.pdf http://www.dmh.ca.gov/Prop63/MHSA/Workforce_Education_and_Training/ FiveYearPlan_5-06-08.pdf 4.NASW Center for Workforce Studies. (2006). NASW National Study of Licensed Social Workers. Washington, DC: National Association of Social Workers. 5.Lok, V., and Chapman, S. (2009). The Mental Health Workforce in California: Trends in Employment, Education, and Diversity. Center for the Health Professions