CORE Evidence-/based Regulation: A Regulatory Performance Measurement System

Goal To help State Boards of Nursing improve their management and service delivery

Need for Study Multiple stakeholders were demanding accountability Lack of clarity among stakeholders about Board roles and responsibilities Trend toward outcome measurement at state level

Purpose of the Study Incorporating data from internal and external sources Using benchmarking strategies Identifying best practices Establishment of a Performance Measurement System

History In 1998, the NCSBN Board of Directors appointed a project advisory group to provide oversight and guide development of a performance measurement system that incorporates data collection from internal and external sources, identification of best practices, and the use of benchmarking strategies.

History Twelve dedicated member boards of nursing (Kentucky, Louisiana-RN, Maryland, Missouri, Nebraska, New Mexico, North Carolina, North Dakota, Ohio, Tennessee, Texas-RN and West Virginia-PN) contributed time and resources to the development of 10 instruments used to gather data in five areas: discipline, licensure, practice, education and governance.

Phase 1 of the Project 1.Validation of Board Roles Study conducted by The Urban Institute

Methods to Gather Information Interview –Focus Groups –Individuals Review of Secondary Documents

Results of Phase 1: Roles Identified –Establish scope of practice for nurses –Issue licenses to qualified nurses –Assure continued competence –Investigate complaints and impose disciplinary sanctions as appropriate

Phase 2 of the Project 1.Validation of Board Roles 2.Identification of Performance Indicators

Results of Phase 2: Technical Work Group developed –Performance indicators –Outcome Indicators –Output Indicators –Efficiency Indicators

Examples Performance Indicator –Timeliness of complaint handling Outcome Indicator –Average time for complaint resolution

Examples continued Output Indicator –Number of complaints resolved in FY Efficiency Indicator –Average cost per completed complaint

Phase 3 of the Project 1.Validation of Board Roles 2.Identification of Performance Indicators 3.Tool Development

Tool Development Process Original tools developed and piloted –Because the science of regulation was not formed enough to guide their efforts, they did not know which data would prove to be the crucial evidence and so they included all data. Original tools revised and further tools developed –6 data collection tools for boards of nursing –Surveys to collect data from 6 stakeholder groups

Board Surveys 1.Discipline 2.Licensure 3.Education Program Approval 4.Practice 5.Governance (Executive Staff) 6.Governance (Board President)

Board Surveys Included Processes used, e.g.: –Investigator caseloads –Use of site visits or self-reports for education programs Timeliness issues, e.g.: –Days needed to processes license request

Board Surveys Included Outcomes achieved, e.g.: –Number of discipline cases closed Opinions, e.g.: –From Executive Staff and Board President

Stakeholder Groups 1.Licensed nurses 2.Health care employers 3.Nurses who had been the subjects of complaints 4.Persons who had lodged complaints 5.Nursing associations 6.Nursing education programs

Stakeholder Surveys Included Perceptions of boards –Timeliness, –Fairness, –Adequacy of regulation, etc. Satisfaction with boards –Communication with stakeholder group, –Nursing program approval process, etc.

Phase 4 of the Project 1.Validation of Board Roles 2.Identification of Performance Indicators 3.Tool Development 4.Data Collection

Data Collection Stakeholder contact information submitted by boards –Random samples selected from those submitted 6 data collection tools sent to boards of nursing

Data Collection Wisdom Not all boards routinely collected the data asked for –Many boards used this as an opportunity to improve/modify amount and types of data collected Language/definitions (i.e., financial data, board processes) differed among boards

Phase 5 of the Project 1.Validation of Board Roles 2.Identification of Performance Indicators 3.Tool development 4.Data Collections 5.Reports of Findings

Report Format I.Aggregate findings A.Data results 1.From board surveys 2.From stakeholder surveys B.Relationships among variables II.State-specific findings A.Comparison of state with all states B.Comparison of state with like boards

Example of Comparison with Aggregate Data, i.e., Ed. Program Perceptions Approval ProcessState Rating Aggregate Rating Interval between board visits Preparation time for board visits Feedback/evaluation provided by board

Comparisons with Similar Boards Boards evidenced a wide variety of resources, structures & processes Boards were compared to other boards similar in a number a variables

Comparison Variables Size of staff Staff assigned to specific functions Numbers of investigators Whether or not state mandated reporting of errors Processes related to complaint review

Comparison Variables continued Board structure Standard of proof Staff autonomy Number of board meeting per year Timeliness of discipline processes Timeliness of licensure processes

Comparisons of Sample Board with Other Boards with Similar Numbers of Staff

Phase 6 of the Project 1.Validation of Board Roles 2.Identification of Performance Indicators 3.Tool development 4.Data Collections 5.Reports of Findings 6.Search for best practices

The Search for Best Practices Data were used to identify boards with consistently high ratings in –Outputs –Effectiveness Ratings were explored in 5 functional areas –Discipline, licensure, education program approval, practice and governance

The Search for Best Practices Selected boards were interviewed to discover –Practices common among boards with consistently high ratings –Differences from boards with lower ratings

Discipline Best Practices Boards with the highest ratings on discipline outcomes –Delegated authority to board staff –Communicated well with stakeholders –Hired investigators and attorneys & actively managed discipline process –Trained and mentored investigative staff –Applied discipline sanctions consistently

Licensure Best Practices Boards with the highest ratings on licensure outcomes –Secured essential human and other resources –Made an aggressive commitment to customer service

Education Best Practices Boards with the highest ratings on education outcomes –Provided consultative, as well as evaluative services to education programs –Took a leadership role in establishing congruence between education and regulation

Practice Best Practices Boards with the highest ratings on practice outcomes –Facilitated understanding of legal scope of practice –Made an aggressive commitment to customer service –Established a high level of involvement with the statewide nursing community –Delegated authority to board staff

Governance Best Practices Boards with the highest ratings on governance outcomes –Promoted an understanding of the respective roles of staff and board members –Built an effective working relationship and a high level of trust between board and staff –Facilitated an effective working relationship among board members –Demonstrated a commitment to board member development

Phase 7 of the Project 1.Validation of Board Roles 2.Identification of Performance Indicators 3.Tool development 4.Data Collections 5.Reports of Findings 6.Search for best practices 7.Development of Ongoing System of Performance Measurement

CORE Nursing Boards educated on CORE –Manuals prepared, distributed and explained –Ongoing presentations and publications Best Practice Tool Kit –Submissions by boards of systems and processes that have facilitated best practice

CORE Ongoing improvement of data collection system –All survey items linked to outcome and/or best practice –Data collection streamlined –Additional tools created and piloted Information Technology Finance Board Member

CORE Timeline 2006 Advisory Committee Formed 2000 Stakeholder Surveys Board Surveys Urban Institute Study Stakeholder Surveys Revised Questionnaires

Schedule JanFebMarchAprilMayJuneJulyAugSeptOctNovDecJanFebMarchAprilMayJuneJulyAugSeptOctNovDec Conduct Stakeholder Surveys Analyze Data Report Results in Aggregate & by State Conduct Board Surveys Analyze Data Report Results in Aggregate & by State X X X X X XX XX X

Board Participation 2006 Stakeholder Surveys Nurses Nurses Subject of a Complaint Persons Who Made a Complaint About a Nurse AssociationsEmployers Education Programs Total = 33 different boards participated in 2006 Stakeholder Surveys

Utilization of Data by Boards of Nursing Data has been used to: Support decision-making Develop mandated reports Provide information to legislators Change data management processes Improve stakeholder satisfaction Streamline processes Determine priorities

Benchmarking "How are we doing?" "Are we tracking the right measures?" "How do we compare with others?" "Are we making progress fast enough?" "Are we using the best practices?"

CORE Performance Benchmarking Process Benchmarking Identifying best practices Adopting best practices

Performance benchmarking The collection of (generally numerical) performance information and making comparisons with other compatible organizations.

Benchmarking of Board Performance Quality Efficiency (cost effectiveness & timeliness) Stakeholder survey Board survey (?)Board survey

Criteria of Item Selection Measurable Comparable between states Objective, if possible

Licensure –Cost per licensee –Cost per license processed –Timeliness to process licensure by initial examination by endorsement by renewal by reinstatement Benchmarking Board Efficiency

Practice –Cost per licensee –Number of practice questions handled per FTE Benchmarking Board Efficiency

Education Program Approval –Rate of education site visits per staff Benchmarking Board Efficiency

Discipline –Average number of investigations completed by each investigator –Rate of investigations completed –Rate of complaints resolved –Rate of nurses disciplined –Timeliness of complaint handling –Cost per resolved complaint –Cost per investigation –Cost per licensee –Cost per discipline case Benchmarking Board Efficiency

Mock Benchmarking Report

Process Benchmarking The comparison of practices, procedures and performance answering the question of What the best practice is in this topic and how to adopt the best practice in my board

Identification of Best Practice How to define best practice How to identify best practice

Level I – Properly conducted RCT, systematic review or meta-analysis Level II –Quasi-experimental, correlational, descriptive, survey, evaluation, and qualitative Level III –Expert opinion or consensus statements Levels of Evidence From Nancy Spector, who adapted from Polit and Hungler (2004) & Gallagher (2003)

Identifying best practice Choose a process to study: One process at a time If desire to study more than one process, it is best to set up separate studies for each process.

Identification of best practice –With clear research questions –With clearly defined conceptual model

For example What is the best practice for a discipline process?

A Conceptual Model Board Practice Authority of staff Communication with PMCs & CNAs Actively Managing disciplinary process Mentoring investigative staff Consistent discipline sanctions Desired outcomes Cost effectiveness Perceived fairness and timeliness of the discipline process Perceived quality of the boards disciplinary process Perceived effectiveness of boards disciplinary process

Operationalize Variables Authority of staff Staff autonomy in resolving selected discipline cases Staff/Board relationship Communication with PMCs and CNAs Board communication to PMCs and CNAs before initiation of investigative process and after the completion of the disciplinary process Board communication to PMCs and CNAs on rationale for the board action Availability of real person for questions Attitude of board toward CNAs

Operationalize Variables Actively Managing disciplinary process a)Authority to hire or directly contract with the investigators and attorneys involved with the disciplinary process. b)Investigators being an RN c) Consistent process (including case management and resolution) d)Investigators involved in the entire disciplinary process

Operationalize Variables Mentoring investigative staff Training for investigators that may or may not include CLEAR training. Training materials available to novice investigators. Mechanism of mentoring investigators

Operationalize Variables Consistent discipline sanctions Enforcement processes are reviewed on a regular basis to ensure consistency of discipline sanctions. The board develops policies and procedures to ensure consistent handling of complaints. Formal mechanics are in place to review proposed sanctions for consistency with previous action taken for similar cases.

Possible Areas of Interest For example: What is the best time to renew a license? Are there any difference in outcomes between using a RN as investigator vs. an non-RN investigator?

Performance Benchmarking

Process Benchmarking

Asked A Lot of Questions ? ? ? ? ? ? ? ? ?? ?? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?

Relationships Timeliness of handling complaints # of FTEs or staff workload Example:

More Precision Q: Were you treated fairly? OR Consider the matters under review without any existing prejudice towards the people or organizations involved (the decision-maker should act in a manner that is unbiased, fair and open-minded) Communicating to the parties involved what is at issue, what decisions might be made adversely to the parties involved and what evidence and information the board may use in reaching its decision (the person should be informed of the matter) Offering a procedure by which the parties involved may test, challenge or contradict the information and evidence used by the board in reaching its decision Offering a procedure by which the evidence, information and submissions from the parties involved can be submitted to the board (opportunity to make oral or written submissions to the board) The reasons for the boards decisions must be based on the facts of the case under consideration The person should be told the reasons for the decision There is consistency between previous actions and proposed sanctions for similar cases

Next Steps I d e n t i f y a n d r e m o v e b a r r i e r s t o p a r t i c i p a t i o n Performance Measures Support member boards adaptations of best practices Best Practices Identified by Focus Groups OR Committee Identified Best Practice