1. Long-Term Care Ombudsman Program Evaluation Activities
Presented by Louise Ryan, MPA
Ombudsman Program Specialist
Administration for Community Living
November 6, 2017

2. Overview Design Process Evaluation Outcome Evaluation
Why evaluation matters
What evaluation can do for Ombudsmen and the Long Term Care Ombudsman Program (LTCOP)
LTCOP Evaluation History – Institute of Medicine Study-1995
ACL’s approach to the LTCOP Evaluation
Design
Process Evaluation
Outcome Evaluation

3. Why Evaluation Matters
We do not have enough resources to do everything
We need to use the resources we have wisely
We want to know that what we are doing is improving lives
Following our gut or doing what we have always done is not enough. In some cases we need to be able to answer the question of why we should invest resources in this versus that; in others we need to determine if the use of resources is justified. Even when we have evidence that something should work or has worked in other circumstances, evaluation can confirm whether it is still working within our current context.

4. Why Evaluation Matters
(1) understanding how the program or policy addresses a problem of interest
(2) Inform improvements to program or policy design or management,
(3) support/change resource allocations,
(4) Identify promising practices or lessons learned
(5) improve quality of program or policy assessment.
-GAO
Does the LTCOP actually improve and protect the lives of residents, the services provided by facilities, and give voice to residents and their families
Do local staff and volunteers receive enough training to be effective and provide high quality services? Is the service provided across staff and volunteers consistent?
Is current program management in terms of staffing, use of volunteers, frequency of facility visits effective and efficient?
How have State and local programs addressed a range of challenges and what can be shared with other programs facing similar challenges?
As we look to the forward, what kinds of policies and support would be helpful from ACL

5. What evaluation can do for Ombudsmen and the Long Term Care Ombudsman Program
The LTCOP has three primary legislative mandates.
Advocate for residents of long-term care facilities
Advocate for systemic change by representing residents’ interests
Ensure access to LTCOP services by conducting outreach and education by providing visits, information and consultation to residents and their families
The LTCOP has three primary legislative mandates.
First, the LTCOP serves as an advocate for
residents of long-term care facilities, including nursing homes, board and care homes, assisted living facilities,
and similar adult residential care facilities. In this capacity, ombudsmen identify, investigate and resolve
complaints about the care residents receive with respect to their health, safety, welfare, and rights.
Second, the LTCOP advocates for systemic change by representing residents’ interests before government agencies and
analyzing, commenting on, and monitoring federal, state and local regulations, policies and actions that
potentially affect residential long-care facilities.
Third, the program is charged with outreach and education by
providing information and consultation to residents and their families and collaborating with other agencies.
Evaluation can tell us how and whether the program is meeting these mandates, which program elements or approaches are best under what circumstances and what improvements can be instituted

6. What evaluation can do for Ombudsmen and the Long Term Care Ombudsman Program
Some outcomes of advocating for residents of long-term care facilities
Increased residents’ awareness of their rights
Increased access to needed services
Improved prevention of problems experienced by residents
Increased resident confidence in raising issues related to their rights and quality of life
Improved quality of care received
LTCOP serves as an advocate for residents of long-term care facilities

7. What evaluation can do for Ombudsmen and the Long Term Care Ombudsman Program
Some outcomes of advocating for systemic change – when representing residents’ interests before government agencies and analyzing, commenting on, and monitoring federal, state and local regulations, policies and actions -
Increased provision of resources to support person-centered care in facilities
Increased coalition/stakeholder involvement
Reductions in repeat violations in facilities
Increased quality of care provided to residents
Second, the LTCOP advocates for systemic change by representing residents’ interests before government agencies and analyzing, commenting on, and monitoring federal, state and local regulations, policies and actions that potentially affect residential long-care facilities.

8. What evaluation can do for Ombudsmen and the Long Term Care Ombudsman Program
Some outcomes related to outreach and education
Increased knowledge of residents’ rights among residents, facility staff, and other stakeholders
Increased perception of Ombudsman professionalism
Increased stakeholder advocacy on behalf of Ombudsman issues
Increased implementation of best practices
Third, the program is charged with outreach and education by providing information and consultation to residents and their families and collaborating with other agencies.

9. First Evaluation – 1993-1995 Real People Real Problems:
An Evaluation of the Long-Term Care Ombudsman Programs of the Older Americans Act (1995)
The IOM appointed a 16-member expert committee
Conducted site visits, seven commissioned papers, numerous contacts with a wide array of ombudsmen and individuals with whom they interact, a one-day invitational symposium, and two meetings of a technical panel.

10. Institute of Medicine (IoM)
The committee’s report examines four key issues:
the extent of compliance with the program’s federal mandates, including conflict of interest issues;
the availability of, unmet need for, and effectiveness of the ombudsman program for residents of LTC facilities;
the adequacy of federal and other resources available to operate the programs; and
the need for and feasibility of providing ombudsman services to older individuals who are not residing in LTC facilities.
To conduct the study, the IOM appointed a 16-member expert committee comprising individuals recognized for their expertise in LTC, medicine, medical sociology, health care policy and research, clinical research, health law, health care administration, state government policy and program administration, consumer advocacy, public health, voluntarism, and the LTC ombudsman program. To inform itself on issues pertaining to this charge, the committee engaged in a variety of fact finding activities. These included site visits, seven commissioned papers, numerous contacts with a wide array of ombudsmen and individuals with whom they interact, a one-day invitational symposium, and two meetings of a technical panel.
"Summary." Institute of Medicine Real People Real Problems: An Evaluation of the Long-Term Care Ombudsman Programs of the Older Americans Act. Washington, DC: The National Academies Press. doi: /9059. ×

11. Institute of Medicine
“identified considerable barriers to effective performance that the ombudsman programs encounter. Significant among these are:
Inadequate funding,
Resulting staff shortages,
Low salary levels for paid staff,
Structural conflicts of interest that limit the ability to act,
Uneven implementation within and across states.”
(Institute of Medicine, 1995, at p. 161)

12. Institute of Medicine recommendations
The Committee made recommendations on all areas of study
3.4. The committee recommends that the Assistant Secretary for Aging issue clearly stated policy and program guidance that sets forth the federal government’s expectations of state long-term care ombudsman programs. Such guidance should articulate operational principles in terms of basic elements of the program,..
5.4. The committee recommends that the Assistant Secretary for Aging continue the efforts of the Administration on Aging to develop, refine, and implement a uniform data collection and reporting system…
These are a couple of examples of where the recommendations were implemented: NORS and LTCOP federal regulation

13. IoM continued
The committee concluded that the ombudsman program serves a vital public purpose and merits continuation with its present mandate. Through advocacy efforts at both the individual resident and the system levels, paid and volunteer ombudsmen uniquely contribute to the well-being of LTC residents—complementing, but not duplicating, the contributions of regulatory agencies, families, community-based organizations, and providers.
Page vi

14. The LTCOP Evaluation: Design
Develop recommendations for a rigorous and comprehensive study design that investigates program efficiency and program effectiveness at multiple levels, including the resident/family, facility, local/state/program, and federal levels

15. The LTCOP Evaluation: Design Challenges
Complexity of the Program and its Relationship to Measuring Outcomes
Decentralization and Diversity of Program Implementation
Full-coverage and the Problem of Comparison Groups
Hybrid Nature of the Program
Reliance on Trained Volunteers
In multifaceted and complex programs such as the LTCOP, it is challenging to establish a clear and causal relationship between inputs, outputs, and outcomes while also taking into account contextual factors that may influence these relationships
Although states are tasked with ensuring that state-level programs meet specific requirements that flow down from the OAA, they also exercise considerable discretion in fulfilling program functions in a manner that best serves their elderly populations. This means that great variations exist in the structure, operation, and effectiveness of ombudsman programs across states. Although the resulting diversity of program characteristics tailored to local contexts is a strength of the program, the same heterogeneity poses challenges for developing sound research and evaluation designs. There is also variation in data collected and in who is served
In addition to the hybrid role1 that ombudsmen play (resembling the classical ombudsman model but without the neutral stance the position has historically assumed), the LTCOP itself can be understood as a hybrid program when we view its advocacy activities across different levels. At the individual/consumer level, advocacy is provided in the form of direct services that are typical of human service delivery programs. These services are offered through consultations, referrals, complaint handling, and via education and outreach activities to build the capacity for advocacy among long-term services and supports (LTSS) residents and their caregivers. At the systems level, the LTCOP engages in more traditional advocacy activities common to advocacy organizations such as monitoring and commenting on laws, regulations, government policies and action, and mobilizing stakeholders toward collective action to support person-centered care practices and policies.

16. Blue Print of the LTCOP
The definition of a blue print is ‘a detailed plan of how to do something’, and that is what a logic model is as well.

17. Two Types of Evaluation
Process Evaluation: focuses on the program's operations, implementation, and service delivery
Currently underway for LTCO
Outcome Evaluation: will build on the process evaluation
focuses on the effectiveness of the program and its outcomes
Will include resident interviews, facility staff interviews or surveys, more outreach to Ombudsman programs

18. Process Evaluation Questions
How is the LTCOP structured and how does it operate at the local, State, and Federal levels including who the program serves, how it is staffed, and what data are collected about activities and outcomes?
How do LTCOPs use existing resources to resolve problems of individual residents and to bring about changes at the facility and governmental (local, State, and Federal) levels that will improve the quality of services available/provided?
With whom do LTCOPs partner, and how do LTCOPs work with partner programs?
How does the LTCOP provide feedback on successful practices and areas for improvement?
Drawing on data collected at the Federal, State, and local levels of the program, the process evaluation will assess the following areas:
The data collection capacity and practice of State and local LTCOP programs
How the program operates at the Federal, State, and local levels independently, and in conjunction with each other
3. The range of ways that ombudsmen (a) identify, investigate and resolve complaints about the care residents receive with respect to their health, safety, welfare and rights; (b) represent residents’ interests before government agencies and analyze, comment on, and monitor Federal, State and local regulations, policies and actions that potentially affect residents of long-term care facilities; and (c) provide information and consultation to facilities, residents, and their families as well as collaboration with other agencies
4. LTCOP use of volunteers and paid staffing
5. Program reach, including inclusion of residential care communities
6. Variability in program characteristics and the contextual differences that contribute to this variability, and
7. Promising innovations or adaptations that may strengthen program effectiveness and/or efficiency.

19. The LTCOP Evaluation: Special Study
Operates under a very different landscape now than it operated during its last comprehensive evaluation 20 years ago (Institute of Medicine, 1995)
Faces more intense challenges of:
an aging population in greater need of care,
competition for resources, and
a changing long-term care system with:
nursing homes focusing on post-acute services,
greater consumer options in residential care settings,
more older adults to “aging in place,”
greater use of home and community-based services.
What are important, defining changes of the LTSS landscape currently and in the foreseeable future? How is the LTCOP preparing for, addressing, or struggling with these features? For those addressing the changes, what is working and why are they using the identified approaches? For those not addressing the changes, what barriers are they facing? What are the implications of these changes and the ways that the LTCOP is adapting or not adapting? Policy implications, Advocacy implications, Legal implications
The special study will help ACL better understand the circumstances or environment in which programs thrive or struggle.

20. Questions & Resources Institute of Medicine Report
NORC Process Evaluation
Louise Ryan