1. NATIONAL ASSOCIATION FOR STATE COMMUNITY SERVICES PROGRAMS 1 NATIONAL ASSOCIATION FOR STATE COMMUNITY SERVICES PROGRAMS (NASCSP) 2012 WAP Orientation Training State Monitoring February 28, 2012 Bob Scott Director, Energy Services

2. NATIONAL ASSOCIATION FOR STATE COMMUNITY SERVICES PROGRAMS 2 Grant Guidance 4.0 GRANTEE PROGRAM OVERSIGHT (Program Monitoring) In-depth guidance on State monitoring of subgrantees In-depth guidance on developing State Monitoring Plans

3. NATIONAL ASSOCIATION FOR STATE COMMUNITY SERVICES PROGRAMS 3 Monitoring Approach Part of State Plan Master File Overview of procedures and guidelines for Grantee monitoring of local agencies

4. NATIONAL ASSOCIATION FOR STATE COMMUNITY SERVICES PROGRAMS 4 Monitoring Procedures Overview Goals of Monitoring Guiding Principles Desktop Reviews Subgrantee Monitoring Visits Monitoring Procedures Local Agency Monitoring Responsibilities Agency Discipline/At-Risk Policy

5. NATIONAL ASSOCIATION FOR STATE COMMUNITY SERVICES PROGRAMS 5 Goals of Monitoring Analyze whether best possible program services are being delivered to low-income population Determine program compliance and accountability Analyze program performance Analyze quality of work

6. NATIONAL ASSOCIATION FOR STATE COMMUNITY SERVICES PROGRAMS 6 Goals of Monitoring Identify problems, deficiencies, and areas that need improvement and advise on how to correct Assist agencies in their program operations and compliance with DOE and State regulations Assess T&TA needs Be a major tool for program improvement

7. NATIONAL ASSOCIATION FOR STATE COMMUNITY SERVICES PROGRAMS 7 Guiding Principles Intended to be constructive process, conducted with consistency, fairness, respect, and timeliness Foster positive, open, and constructive working relationships Two-way process to promote interaction, feedback, and improvements to Grantee and Subgrantee

8. NATIONAL ASSOCIATION FOR STATE COMMUNITY SERVICES PROGRAMS 8 Guiding Principles Understanding that Grantee and Subgrantee share the same goals of program effectiveness and improvement Promote improvement by providing technical assistance, reinforcing strengths, and sharing successes, innovations, and practices seen at other agencies Reports will be based on established policies, procedures and standards

9. NATIONAL ASSOCIATION FOR STATE COMMUNITY SERVICES PROGRAMS 9 Types of Monitoring Desktop Monitoring Subgrantee Visits

10. NATIONAL ASSOCIATION FOR STATE COMMUNITY SERVICES PROGRAMS 10 Desktop Monitoring Not an agency visit Based on reported information to the State by subgrantees Can save time of actual subgrantee visits by preparing and organizing information in advance Performance analysis Review of expenditures Certain compliance issues Identify specific focus areas for agency visit, including problematic jobs

11. NATIONAL ASSOCIATION FOR STATE COMMUNITY SERVICES PROGRAMS 11 Subgrantee Visits Fiscal Monitoring Program Specific Administrative and Management Review Field Procedures Monitoring and Job Inspections

12. NATIONAL ASSOCIATION FOR STATE COMMUNITY SERVICES PROGRAMS 12 Fiscal Monitoring Often part of overall agency fiscal monitoring conducted by specialized Grantee fiscal monitor Sometimes a distinction between fiscal/procurement monitoring and specific WAP issues like job costing and reporting

13. NATIONAL ASSOCIATION FOR STATE COMMUNITY SERVICES PROGRAMS 13 Fiscal Monitoring Key issues identified during ARRA Procurement Invoicing techniques Accurate job costing Proper designation of expenses between Admin. and Program Operations Property records and inventory control

14. NATIONAL ASSOCIATION FOR STATE COMMUNITY SERVICES PROGRAMS 14 Administrative and Management Review Review production, expenditures, and related compliance issues with agency Review Client Files Review client eligibility and income verification documentation Review material, tool, equipment, and vehicle property and inventory records Review client flow charts and scheduling Review safety policies and practices Review insurance policies Inspect warehouse

15. NATIONAL ASSOCIATION FOR STATE COMMUNITY SERVICES PROGRAMS 15 Field Procedures Monitoring and Job Inspections Analysis of quality of field practices, including  Audit procedures and compliance with audit protocols and work orders  Diagnostics  Overall effectiveness  Compliance with State Field Standards  Workmanship  Appearance  Thoroughness  Health and Safety issues  Agency final inspection procedures  Training needs

16. NATIONAL ASSOCIATION FOR STATE COMMUNITY SERVICES PROGRAMS 16 Field Inspections Will probably have several more field inspection visits that fiscal and administrative visits  Frequent visits recommended  Should be a random selection of jobs within reason

17. NATIONAL ASSOCIATION FOR STATE COMMUNITY SERVICES PROGRAMS 17 Field Inspections – Best Practice Focus on agency audits, work orders, work done, and the agency final inspection If any of these are not consistently done well, the agency will likely have problems with call backs Grantee monitoring should analyze effectiveness of subgrantee inspection process - arguably the biggest issue in Quality Control

18. NATIONAL ASSOCIATION FOR STATE COMMUNITY SERVICES PROGRAMS 18 Monitoring Procedures Agency visits are typically scheduled in advance by notification from Grantee Representative Upon arrival, Monitor(s) should conduct entrance interview Local WAP Coordinator should be available at all times during visit Exit conference at conclusion of visit

19. NATIONAL ASSOCIATION FOR STATE COMMUNITY SERVICES PROGRAMS 19 Monitoring Procedures Report should be sent to subgrantee within 30 days (preferably sooner) Agency has to submit corrective actions, if necessary, within specified time frame Grantee reviews agency response Grantee writes agency to note acceptance of response or any unresolved issues

20. NATIONAL ASSOCIATION FOR STATE COMMUNITY SERVICES PROGRAMS 20 Best Practices - Approach Be constructive – conduct yourself professionally, be fair, treat agency personnel with respect, be on time Never monitor with “I gotcha” intentions Always act as a partner and team player – after all, you and the agency should share the same goal of program improvement

21. NATIONAL ASSOCIATION FOR STATE COMMUNITY SERVICES PROGRAMS 21 Best Practices - Approach Help find solutions to problems Understand the agency and local program environment and set realistic expectations Offer constructive criticism based on your knowledge, experience, and the fact that you have observed other agencies that have dealt with similar circumstances

22. NATIONAL ASSOCIATION FOR STATE COMMUNITY SERVICES PROGRAMS 22 Best Practices - Approach States need to tell subgrantee in advance when you are coming, what you will be doing, and what documentation will be needed Don’t be afraid to pat someone on the back, they are in the Weatherization Program and probably need and deserve it

23. NATIONAL ASSOCIATION FOR STATE COMMUNITY SERVICES PROGRAMS 23 Best Practices - Reports The report is the written final word – make sure monitors have time to carefully write report Remember that some of the readers of the monitoring report may not have the same expertise as the monitor Make sure monitoring reports are clear and objective, and major findings are listed first and most prominently Findings should include specific references to Weatherization rules and standards

24. NATIONAL ASSOCIATION FOR STATE COMMUNITY SERVICES PROGRAMS 24 Best Practices - Reports Either report or cover letter must make clear corrective actions needed and possible consequences State Program Managers must insure their monitors are thorough and correct and then must back the reports

25. NATIONAL ASSOCIATION FOR STATE COMMUNITY SERVICES PROGRAMS 25 Best Practices - Reports Remember – monitoring reports are often the only assessment by State an agency receives Should be a thorough assessment that also recognizes good practices, not just deficiencies

26. NATIONAL ASSOCIATION FOR STATE COMMUNITY SERVICES PROGRAMS 26 Best Practices - Reports Monitor(s) should draft reports (or at least an outline of findings) and conduct exit interview so agency is not taken off guard by report

27. NATIONAL ASSOCIATION FOR STATE COMMUNITY SERVICES PROGRAMS 27 Best Practices - Reports Decide on terms and have a clear distinction for each Common terms  Finding – an issue that may be a violation of rules or standards and needs to be addressed  Concern – an issue that may not be in violation of rules or standards but needs to be addressed  Recommendation – notation based on monitor’s experience and expertise on how to improve certain aspect of the program

28. NATIONAL ASSOCIATION FOR STATE COMMUNITY SERVICES PROGRAMS 28 Best Practices - Reports Sometimes a thin line between a finding and a concern Finding – more significant, such as misuse of funds, eligibility issue, major health and safety issue, consistently failing to follow rules or standards Concern – Smaller issue, such as minor file omissions, or field observation that doesn’t significantly impact overall results but needs to be addressed

29. NATIONAL ASSOCIATION FOR STATE COMMUNITY SERVICES PROGRAMS 29 Other Best Practices State needs to have clear, printed up-to- date administrative and field standards to base monitoring Make monitoring flow between visits – do follow-up of last visit Document progress from last visit Confirm whether cited deficiencies were corrected Note areas of improvement

30. NATIONAL ASSOCIATION FOR STATE COMMUNITY SERVICES PROGRAMS 30 Other Best Practices Do frequent monitoring – agency tends to be more responsive to findings and suggestions Consider team monitoring – can cover a lot more ground and can capitalize on individual strengths Focus on issues – not people o Be tough on issues, but (where possible) sensitive to people

31. NATIONAL ASSOCIATION FOR STATE COMMUNITY SERVICES PROGRAMS 31 Other Best Practices Use digital cameras to document and illustrate field work o Make digital pics part of the exit conference and report Talk to clients – assess their satisfaction, ask if they can tell difference in comfort and/or fuel bills, but be careful about discussing findings with them

32. NATIONAL ASSOCIATION FOR STATE COMMUNITY SERVICES PROGRAMS 32 Other Best Practices Make monitoring part of the Training Program and use it as an opportunity to provide T&TA o Include agency field staff in field monitoring o Consider asking agency to select their best jobs and their most problematic ones and visit with them o Have them set up blower door, do diagnostic tests, heating system tests, etc.  Analyze their expertise and practices  See if equipment is calibrated and maintained

33. NATIONAL ASSOCIATION FOR STATE COMMUNITY SERVICES PROGRAMS 33 Other Best Practices Strive to be consistent between monitoring staff o Insure your monitoring forms/tools are as objective as possible o Have monitors do “practice” monitoring together. Inspect same files individually and then compare notes. Go to same house, inspect individually and compare findings and discuss. o Consider developing library of “common findings” so there is similar wording and explanations between monitors in reports

34. NATIONAL ASSOCIATION FOR STATE COMMUNITY SERVICES PROGRAMS 34 Other Best Practices And perhaps most important – due to DOE policy on call backs in WPN 11-3  Formulate specific criteria that spells out criteria for agency call backs  Have open discussions with State Managers, Monitors, and local agency staff

35. NATIONAL ASSOCIATION FOR STATE COMMUNITY SERVICES PROGRAMS Contacts Bob Scott rscott@nascsp.org 202-624-5867 35