“You Can’t Always Get What You Want: Accountability and Limits in Score Reports to Boards” Troy Elliott Association of Social Work Boards 2006 Annual ConferenceAlexandria, Virginia Council on Licensure, Enforcement and Regulation Expect the Unexpected: Are We Clearly Prepared? Censored image of psychometrician in Mick Jagger style tight leather pants

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia Who We Are Association of Social Work Boards (ASWB) 49 states, DC, USVI, 7 provinces Public protection mission Social work licensure examinations—about 25,000 administrations annually Other services: Disciplinary databank, continuing education provider approval, credentials registry, communication, etc.

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia The ASWB Examinations Five categories (Associate, Bachelors, Masters, Advanced Generalist, Clinical) 4 option multiple choice 150 scored, 20 pretest CBT 4 hour time limit Psychometric and administration support through ACT, Inc.

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia The Score IRT process – no single raw score applicable to all forms Slight variations to maintain consistent overall difficulty – anchor exam

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia The score, continued, unfortunately So what everyone gets is the equated score, right? If only it were that easy…

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia The score, continued, unfortunately The Scaled Score Zone, or Ketchup vs. Catsup— the debate rages on “70” versus “75” Where does it exist? Why does it exist? How many English majors does it take to develop an apt simile? 7075

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia Now that the basics are covered… Who gets what right away: Passing Candidates

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia Now that the basics are covered… Who gets what right away: Failing Candidates

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia Now that the basics are covered… Who gets what right away: Regulatory Boards

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia Summary reports Every year, to boards

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia Choose your cliché The greatest thing since sliced bread….or something else entirely? Our great idea: reports to schools

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia School data What the schools get

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia How could anything go wrong? Misuse – program outcome, marketing Requests for more data Fingerpointing

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia What does this have to do with my topic? Reporting service highlighted relationship between regulatory boards and schools Some boards interested in school data

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia Data salad, hold the context Requests and questions from ASWB member boards Subscores? Jurisdictional pass rates? School data shared with everyone?

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia Data salad, hold the context ASWB’s initial answers: No

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia What’s our problem anyway? The three Ps: philosophy, psychometrics, potential litigation

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia What’s our problem anyway? Philosophy Whose data is it?

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia What’s our problem anyway? Psychometrics 150-item test with content areas ranging from 3 – 22 percent of the examination. Schools—and sometimes, entire jurisdictions—with small testing populations, even smaller first-timer numbers.

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia What’s our problem anyway? Potential litigation Misuse/misrepresentation of data Hiring/firing/program changes based on poor data Would we prevail? Probably. But at what cost?

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia Coming soon to an association near you Protection of the exam meets service to members

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia Under consideration… Compilation of jurisdictional pass rates, with identification where allowed All school data reported to boards (in addition to existing program)

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia What did we learn? A victim of our own success? Licensing (and the exam)more important than ever Fewer exemptions More importance among employers Recognition by educational programs

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia What did we learn? Looking for outcomes in all the wrong places Test data can get caught up in political and economic pressures faced by schools

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia What did we learn? Member education never stops Turnover not just limited to board members Multidisciplinary boards increase the need for education (“If they can do it why can’t you?”) Culture of profession regulated impacts expectations, available options

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia What do we still need to learn? Information culture – has the data train left the station? Expectations “Good” data vs. “transparency” Whose data is it anyway?

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia What do we still need to learn? Association culture – when is service a disservice? Long range planning, goal identification: don’t forget overall profile Tough decisions: sometimes you have to make them

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia And you thought I’d never shut up Troy Elliott, Communications Director Association of Social Work Boards 400 South Ridge Parkway, Suite B Culpeper, VA