Collecting and Using Data ROMA Data for Management and Accountability

Why collect data?  The ultimate goal is to help local CAAs and states to use their own data to manage and improve outcomes.  While we want to see improved quality of reports to an outside third party, this is not the prime objective.

Quality of the data In order to assess the quality of the data that is being collected at the national level, we must identify the way data is being collected by local CAAs and by states. What is our level of confidence in the data that is available for the NPI reporting system?

Types of data collection One question always asked is: who are using electronic systems to collect and organize data, and who are using paper reports? The next level question is: if they are using electronic systems, are they data base systems (MIS) or reporting systems? Do they capture CSBG only or the entire agency? The entire state?

Local or State?  Data entered at local CAAs into a data base or spreadsheet (paper or elect) is client level data that includes demographics, services, and outcomes OR is agency level data that is an aggregate of information from clients served in the agency  Data entered at the state level is from a state wide client level data base is from local CAAs (agency level info) sent electronically or in a paper report

How can it be used?  Local CAAs collect data to have information to report to the state to use in management of the agency’s programs and services to evaluate the quality of outcomes at the F, A and C level.  States collect the data to have information to report to OCS/NASCSP to use in monitoring and management of the various agencies receiving funding to evaluate the quality of outcomes at the F, A and C level.

Raw data is raw data Data must be combined, summarized and analyzed, typically involving comparison over time or with projected targets.

Outcomes may be grouped for analysis, by such things as client characteristics or services received. Does the data collection system permit such grouping?

The distinction between data and information: Information is organized data that tells a story. Remember: data itself has limited or no value unless it is organized and analyzed. Turn data into information! And use it!

As you analyze the data:  Check accuracy of the data. Does it make sense?  Consider the number and characteristics of the recipients of service and those who achieved an outcome.  Seek explanations for unusual outcomes or unusual numbers reported.  Was your performance better than you projected? Why? (error in projection? Change in situation?)  Was it less than you projected? Why?

Are you using the right measurement?  Check that the right measurement is being used to document the outcome.  Consider what is not being covered in the measurement.  Don’t jump to conclusions! Ask: how do I know that the program/service is responsible for the outcome?

USING THE CARTER QUESTIONS  1. How many clients are you serving? Does the agency know the answer to this question: When does a client become a client? Can you produce an unduplicated count for the agency?  2. Who are they?  3. What services do you give them? There can be multiple services within a single program or process. Can you establish the number of services delivered to each client?

 4. What does it cost?  5. What does it cost per service delivered?  6. What happens to the clients as a result of the service?  There can be multiple outcomes for each service delivered.  And there can be multiple services required to meet a single outcome.  You need to establish a number of successful outcomes and be able to identify the number of services required to produce the outcome (not only CAA services, but also partner/collaborator services).

And the Richmond Questions?  7. What does it cost per outcome?  8. What is the value of a successful outcome?  9. What is the return on the investment? What data would be needed to answer any of these questions? These questions cannot be viewed as separate and distinct sets of data. Rather the data needs to be interrelated -- each question to another.

USING IM 49 “OCS received a number of comments from the Network questioning whether ROMA should involve programs beyond the Community Services Block Grant. After careful examination of the CSBG authorizing legislation, which speaks to program coordination requirements both within and beyond eligible entities, consultation with the MATF, and review of ROMA implementation activities that have occurred to date, OCS has concluded that it is both necessary and appropriate to apply ROMA concepts to the work of community action, not CSBG alone.”

Implications for local CAAs  Focus on CSBG? Integrate CSBG?  Track individual clients – correlate services with outcomes; and these with demographics  Track outcomes at Family, Agency and Community levels – what is the relationship among these efforts?

Aggregating Data at the State Level Things to look for: Reporting services and outcomes for the entire agency, not just CSBG Reporting outcomes for clients who receive services from collaborators/partners Reporting all resources Reporting volunteer service at all levels

Assessing Agencies  Do local CAAs: > have a way to report unduplicated clients served? > distinguish between service units and outcomes? > understand the difference between goal 1 and goal 6? > understand the difference between progress toward a goal and achievement of an outcome at a benchmark? > document and collect data for all six goals? > USE THE INFORMATION FROM THEIR DATA TO MAKE DECISIONS?