Homeless Management Information Systems (HMIS) Data Quality: Practical Strategies and Tips for Improving Data Quality

Prepared by Center for Social Policy, UMass Boston and Abt Associates for the U.S. Department of Housing and Urban Development 2 OVERVIEW What Is Data Quality? Why Is Data Quality Important? How Do You Know if Your Data are Good? Playing Your Part to Enhance Data Quality: Tips for Every Level Common Data Quality Issues for Universal Data Elements

Prepared by Center for Social Policy, UMass Boston and Abt Associates for the U.S. Department of Housing and Urban Development 3 What Is Data Quality? Data quality refers to the accuracy and completeness of information collected and reported in HMIS. Quality data allows programs, agencies, and CoCs to make accurate statements and findings about persons served or about program outcomes.

Prepared by Center for Social Policy, UMass Boston and Abt Associates for the U.S. Department of Housing and Urban Development 4 Why Is Data Quality Important? Missing identifiers make unduplicated counts unreliable – Inaccurate counts may under or over represent the population and may impact funding Misrepresenting client characteristics can lead to: – Misdirected resources – Priority given to certain types of programming. Incomplete entry and exit data cannot reveal: – How people move in and out of the homeless system; or – What combinations of services are most effective in moving persons out of homelessness.

Prepared by Center for Social Policy, UMass Boston and Abt Associates for the U.S. Department of Housing and Urban Development 5 How Do You Know If Your Data Are Good? Run Reports to See if Data Are: – Complete – Accurate Reports can be: – Aggregate level, to spot overall issues with CoC, Program or User – Client-level, to find records with problems You can use built-in reports designed to check data quality, or you can also use other everyday standard reports or custom queries. – Timely – Consistent

Prepared by Center for Social Policy, UMass Boston and Abt Associates for the U.S. Department of Housing and Urban Development 6 Are Your Data Complete? Are All Clients Entered? – Compare client counts to bed capacity or on-site check – Compare records in recent period vs. previous periods – Are there family shelter guests with only one client? – Check against number of paper records, if applicable Are All the Required Fields Filled In? – Check the completion rates of each required field. e.g., “Gender: 96% Complete” – Client level, e.g., “List All Clients Where Race Is Null” See Handouts 1 and 2 for sample reports

Prepared by Center for Social Policy, UMass Boston and Abt Associates for the U.S. Department of Housing and Urban Development 7 Special Note on Exit Dates Exit dates are critical for reporting: – How many people are in the system; – Demographics of people still in the system; – Length of stay; and – Program Outcomes and Effectiveness. Without exit dates, very few other data elements are useful or reliable. All programs should have a clear process for recording exit dates and for monitoring this field. Consider exiting out today any clients that are not in program.

Prepared by Center for Social Policy, UMass Boston and Abt Associates for the U.S. Department of Housing and Urban Development 8 What if Client Counts Don’t Match Expectations? If client count is too low: – Not all clients entered? – Many clients entered with the name “John Doe”? If client count is too high? – Clients not exited? – Records not properly de-duplicated? – IDs not entered consistently? Or, maybe, expectations are wrong and data are right! HMIS sometimes reveals unexpected information.

Prepared by Center for Social Policy, UMass Boston and Abt Associates for the U.S. Department of Housing and Urban Development 9 Are Your Data Accurate? Are staff collecting true information and properly recording responses? – Check for valid data (e.g., no veterans are minors) – Check for aggregate data in expected range for target population: <5% of clients older than 75 years, <2% Native Hawaiian/Pacific Islander – Compare random sample of paper files See Handout 3 for sample data validity checks

Prepared by Center for Social Policy, UMass Boston and Abt Associates for the U.S. Department of Housing and Urban Development 10 Are Your Data Timely? Are data entered soon after collected? – Compare dates of program entry to the date the record was created – Run reports on data from yesterday or two days ago. Are changing data kept up to date? – Check for clients still in shelter with “Last Updated” dates more than a month ago.

Prepared by Center for Social Policy, UMass Boston and Abt Associates for the U.S. Department of Housing and Urban Development 11 Are Your Data Consistent? Does Everyone Understand the Questions and Answers in the Same Way? – Compare data from different users/programs serving the same general population. e.g., 95% of Bob’s clients have a disability; 60% of Mary’s clients have a disability. – Take advantage of duplicate records. Spot check to see whether you’re getting the same data (e.g. first name, race) for the same client in different programs or service episodes. – Survey users and see if they record the same responses with sample clients.

Prepared by Center for Social Policy, UMass Boston and Abt Associates for the U.S. Department of Housing and Urban Development 12 Does Everyone Know What “Good” Is? Set Common Standards for your Community Your CoC and Program should have common expectations for – What gets collected and entered – Who enters data – Who and when the data are checked and how errors get fixed – When data get entered – What happens to data after they are entered

Prepared by Center for Social Policy, UMass Boston and Abt Associates for the U.S. Department of Housing and Urban Development 13 Does Everyone Know What “Good” Is? Formalize Expectations in a Data Quality Plan – A set of written policies that set common standards and procedures for ensuring data quality A plan should include: – Principles – Benchmarks – Monitoring Procedures – Incentives – Contractual Agreements/Buy-In Completeness Accuracy Timeliness Consistency See Handouts 4 and 5 for a monitoring report and associated DQ plan and 6 for a worksheet on developing a plan

Prepared by Center for Social Policy, UMass Boston and Abt Associates for the U.S. Department of Housing and Urban Development 14 Playing Your Part to Enhance Data Quality: Tips for Every Level Everyone Plays a Role in Enhancing Data Quality – CoC-Level Technology Processes – Program-Level – User-Level Collection Entry

Prepared by Center for Social Policy, UMass Boston and Abt Associates for the U.S. Department of Housing and Urban Development 15 CoC Level: HMIS Project Staff Roles Create and implement data quality plan Check data quality and provide feedback Provide training, support and documentation Hold regular user groups Convene data quality sub-committee – This committee can spearhead plan implementation Release only good data and clarify limitations with every aggregate data release

Prepared by Center for Social Policy, UMass Boston and Abt Associates for the U.S. Department of Housing and Urban Development 16 CoC Level: Prevent User Errors with Your Technology Customize the software to minimize errors – Screen Design: All required fields in logical flow Required fields marked, labels written clearly Use drop downs, not free text – Validation: Rather than reporting on completeness and accuracy issues after-the-fact, validations can occur during data entry. Tip: Allow “Don’t Know” options or use validate with warnings. Systems that force data entry end up with “Joe Guest” and “99999.” Revisit Handout 3

Prepared by Center for Social Policy, UMass Boston and Abt Associates for the U.S. Department of Housing and Urban Development 17 CoC-Level: Training and Communication “Take Advantage of These Features!” To clear up confusion about question, click… Module for viewing incomplete record, click… To view data quality report: go under reports… The following information is validated…. “Watch Out!” This “intake date” is not the same as this “service begin date”. Last name is listed first here, but last here. In order to answer all the required fields… This report shows missing fields, but not fields that are outdated. Highlight strengths and pitfalls of software in trainings, user groups and other communications. For example:

Prepared by Center for Social Policy, UMass Boston and Abt Associates for the U.S. Department of Housing and Urban Development 18 Program Level: Data Flow Process Create Program-Level Processes for: – Flow of data collection and data entry Whose job is HMIS anyway? – Is there one “HMIS person” or is everyone responsible for entering their own clients? – Must all data be on paper? Is paper allowed? Many providers lack clear processes for entering exit data or updating information If data are updated on paper form, how will the data entry staff spot the new information? If an issue is found by data entry staff, what is the protocol for getting resolution to their question?

Prepared by Center for Social Policy, UMass Boston and Abt Associates for the U.S. Department of Housing and Urban Development 19 Program Level: Monitoring & Using Data Regularly Monitor Data Quality – Review quality assurance and standard reports – Provide feedback to staff and facilitate conversation between those collecting and entering data – Tip: Tie data collection/entry to job duties and performance and allow necessary time for training Integrate Use of HMIS Into Daily Operations, Including Use of the Data – Tip: When staff knows directors are relying on HMIS data to report to the board or to funders, quality is bound to be high. See Handout 7 for sample program communication

Prepared by Center for Social Policy, UMass Boston and Abt Associates for the U.S. Department of Housing and Urban Development 20 User Level: Data Collection Intake or front line staff are often the first point of data collection for clients in need of service Intake or front line staff need to understand and be able to communicate to every client served why information is being captured and how the information will be used including: – Purpose of data collection; – Importance of HMIS at the local level; and – Privacy policies and consent protocols.

Prepared by Center for Social Policy, UMass Boston and Abt Associates for the U.S. Department of Housing and Urban Development 21 User Level: Data Collection Paper – Feels more natural; may be less intimidating; may be similar to “old” process – Tip: Use a data collection form that resembles the computer screen; use block lettering; have response values, not free text on form. Computer – Timely; Avoids extra time/persons required for data entry; can use scan cards for large shelter registration – Tip: Allow clients to sit and view the screen while data is entered; print out a report for the client to have. Vs. See Handout 8 for sample paper form

Prepared by Center for Social Policy, UMass Boston and Abt Associates for the U.S. Department of Housing and Urban Development 22 User Level: Data Entry Data Entry Staff, including volunteers must be trained to: – Search the HMIS for an existing client record by all methods, if applicable – Enter all the information provided – Enter accurate information Proofread for common errors: – Accidentally picking the wrong response category – Typing Data in the wrong field – Misspellings – Tip: False data are usually worse than no data; e.g. “Baby Boy” instead of first name.

Prepared by Center for Social Policy, UMass Boston and Abt Associates for the U.S. Department of Housing and Urban Development 23 User Level: Data Entry Data entry staff should also: – Communicate regularly with front-line staff Clarify shorthand (What does “SA” mean?) – Not independently change or ignore suspicious data Record issues in a data quality log See Handout 9 for sample data quality log

Prepared by Center for Social Policy, UMass Boston and Abt Associates for the U.S. Department of Housing and Urban Development 24 User Level: Understand Required Data Elements Front-line and data entry staff should understand: – What needs to be collected and definitions of each – When to collect each element – Where to enter each element – Tips to ensure each element is entered properly. E.g., If you only have the last 4 digits of SSN, enter 5 spaces first and mark quality code as “Partial SSN Recorded” Use “01/01” for Month/Day if only year of Birth is known “Last Permanent Address” refers to the last place the client lived for 90 days or more See Handout 10 for notes and tips on each universal data element

Prepared by Center for Social Policy, UMass Boston and Abt Associates for the U.S. Department of Housing and Urban Development 25 Feeling Overwhelmed? There are many ways to check data quality. – Start with data completeness checks Key issues to address first: – All Clients Entered – Exit Dates! Exit Dates! Exit Dates! Use this training, the handouts and the Data Quality white paper to implement a more comprehensive data quality plan as you feel comfortable with these first steps. For more information, see “Enhancing HMIS Data Quality” available at MANTRA: Enter and Exit

Prepared by Center for Social Policy, UMass Boston and Abt Associates for the U.S. Department of Housing and Urban Development 26 Summary of Key Points on Data Quality Data should be accurate, complete, consistent and timely All members of the CoC play a role in promoting data quality A Data Quality plan sets standards and procedures to ensure data quality Built-in data quality reports operationalize the goals in the data quality plan

Prepared by Center for Social Policy, UMass Boston and Abt Associates for the U.S. Department of Housing and Urban Development 27 Additional Resources “Enhancing HMIS Data Quality”: Additional resources are available at: Click on “Resource Library” to search for documents