1. 1Managed by UT-Battelle for the Department of Energy Jackie Berger APPRISE Quantitative Findings from On-Site Evaluation of Energy Efficiency Program Service Delivery

2. 2Managed by UT-Battelle for the Department of Energy Presentation Overview  Process Field Study design  Sample of findings – Audit observation findings – Measure installation observation findings – Final inspection observation findings  Program recommendations – Training – Equipment – Management

3. 3Managed by UT-Battelle for the Department of Energy PROCESS FIELD STUDY DESIGN

4. 4Managed by UT-Battelle for the Department of Energy Process Field Study Focus  Audit approach and implementation  Energy education delivery  Measure installation/operation of crews  Post-weatherization quality assurance inspection approach and implementation  Client interaction  Interactions among agency staff  Implications for management, procedures, and training

5. 5Managed by UT-Battelle for the Department of Energy Process Field Study Approach  Go beyond anecdotes  Quantify findings across all visits – Develop check lists and rating scales – Train weatherization experts to implement consistently  Enrich data with descriptive information  Make recommendations for program improvement based on significant issues

6. 6Managed by UT-Battelle for the Department of Energy Check List Example Audit Air Leakage and Insulation Diagnostics Applicable Observations Action Taken #% Measured surfaces 1009494% Inspected windows 1009696% Inspected all accessible attics 786988% Measured insulation in all accessible attics 797089% Created access to inaccessible attics 33310% Measured insulation in exterior walls 954749% Measured insulation in basement/crawlspace 745574% Inspected for all typical bypasses 1006262% Visual inspection for air sealing opportunities 1008383% Used blower door while inspecting for leaks 966467%

7. 7Managed by UT-Battelle for the Department of Energy Rating Design Rating1234*5* % of Points needed0%-49%50%-74%75%-89%90%-99%100% Total pointsNumber of Points needed for each rating 40-123--4 50-234--5 60-23-4--56 70-34567 8 4-5678 90-45-6789 100-45-78910 110-56-78-91011 120-56-89-101112 130-67-910-111213 140-67-910-121314 150-78-1011-131415

8. 8Managed by UT-Battelle for the Department of Energy Rating Example Attic Insulation Quality Quality of Attic Insulation 1 All air sealing work completed first 2 Exhaust fans vented to exterior as needed 3 Heat producing devices or systems protected from insulation contact 4 Attic checked for knob and tube wiring 5 Workers wore respirators, safety glasses, gloves, and hard hats while insulating attic 6 Insulation installed in sufficient quantity (bags per ft 2 ) to meet R-value requirement 7 Proper insulation material chosen for attic conditions 8 Open blow insulation is level and of consistent depth 9 Attic ventilation maintained 10 Confined areas blown to dense pack 11 Proper containment used to protect client and belongings Needs Improvement Excellent Mean Rating# Rated Rating12345 Total Points needed 0-56-78-91011 Bold Points needed00066 Attic Insulation Quality4%18%42%19%18%3.357

9. 9Managed by UT-Battelle for the Department of Energy Field Study Flow Chart Draft Forms and Procedures Test Draft Forms and Procedures Revise Forms and Procedures Convene Expert Panel Revise Forms and Procedures Select Agency Sample Conduct Interviews and Observations Wx Expert and Social Scientist Debriefing Analyze Data and Summaries Report Test Forms and Procedures Revise Forms and Procedures Wx Expert Training Social Scientist Training Revise Forms and Procedures

10. 10Managed by UT-Battelle for the Department of Energy Agency Selection  WAP Agencies – Almost 1,000 deliver services across the country – 400 comprised agency billing data sample – Field Process Study – 20 agencies in 20 different states  Selection based on: – Number of states per region based on WAP funding – State selection based on part of region and program size – Agencies selected to represent: Urban/RuralEducation } Based on agency survey PY 2008 planned jobsTraining Measure selectionQuality Control

11. 11Managed by UT-Battelle for the Department of Energy Observation Team  Social Scientists – One week observation at agency  Weatherization manager and staff interviews  Observations – client intake, audit, measure installation, final inspection – Written agency summary report  Weatherization Experts – 3-Week Observations at Agency  Observations – audit, measure installation, final inspection – Post Observation Data Entry  Training  Monthly update calls  Final debriefing

12. 12Managed by UT-Battelle for the Department of Energy Process Field Study Observations Number of Observations Social Scientist Weatherization Expert Total Audit 43112155 Measure Installation 45114159 Final Inspection 3791128 Total 125317442

13. 13Managed by UT-Battelle for the Department of Energy PROCESS FIELD STUDY AUDIT OBSERVATION FINDINGS

14. 14Managed by UT-Battelle for the Department of Energy Audit Observations Check Lists  Audit preparation  Audit introduction  Audit home walk through  Heating assessment  Air conditioning assessment  Ventilation assessment  Water heater assessment  Air leakage and insulation diagnostics  Diagnostic testing*  Combustion safety testing*  Refrigerator assessment  Lighting assessment  Exit interview *Should have been performed, was performed, performed correctly **Described on write-up, recorded thoroughly and accurately Ratings  Audit introduction  Auditor attempt to engage client  Client engagement in audit  Auditor communication skills  Quantity of education  Quality of education  Exit interview  Completeness of inspection  Completeness of testing  Quality of testing  Auditor efficiency  Auditor professionalism  Auditor respect of client’s home  Auditor safety practices  Audit comprehensiveness*  Planned work comprehensiveness*  Audit write up *Excellent, good, poor

15. 15Managed by UT-Battelle for the Department of Energy Audit Introduction IntroLater Intro or Later #%#%#% Explained WAP 7471%66%8077% Explained auditor’s role in WAP 7673%44%8077% Explained client’s role in WAP 3938%99%4846% Asked what client hoped to get from WAP 2726%44%3130% Reviewed energy bill(s) 1212%00%1212% Discussed health and safety 6361%44%6764% Discussed comfort 5957%22%6159% Asked about problems with energy usage 3433%11%3534% Out of 104 observations completed.

16. 16Managed by UT-Battelle for the Department of Energy Audit Introduction Rating Design Quality of Introduction 1Auditor introduced self 2Auditor explained purpose of visit 3Auditor explained WAP process 4Auditor reviewed energy bills with client 5Auditor asked about comfort issues 6Auditor asked about energy issues 7Auditor asked about H&S issues 8Auditor assessed client’s desired outcomes Needs Improvement Excellent Rating12345 Total Points needed0-34-5678 Bold Points needed00033

17. 17Managed by UT-Battelle for the Department of Energy Audit Introduction Rating Needs Improvement Excellent Rating12345 Number of Audits323325112 Percent of Audits31%32%24%11%2% Mean Rating2.2

18. 18Managed by UT-Battelle for the Department of Energy Audit Diagnostic Tests Tests Performed Should Have Been Should Have Been & Was % Performed % Performed Correctly Blower door testing 949197%84% Zonal pressure diagnostics 572442%87% Infrared camera 412049%70% Steady state efficiency 564479%98% Air flow test on forced air furnace 351440%93% Proper coolant charge 500%N/A

19. 19Managed by UT-Battelle for the Department of Energy Audit Write-Up Rating Design Quality of Audit Write-Up 1Did not fail to clearly document need for lead safe weatherization if needed 2Included estimated material quantities 3Specified any needed materials 4Clearly documented client health and safety issues 5Recommended measures prioritized by savings to investment ratio 6Set air leakage reduction targets for crews 7Directed crews to access areas inaccessible during the audit 8Contained clear directions for crews 9Documented client concerns/needs 10Documented client commitments if made 11Included special instructions if needed 12Clear and easy to understand

20. 20Managed by UT-Battelle for the Department of Energy Quality of Audit Write-Up Rating Number Rated Needs Improvement Excellent Mean Rating 12345 Percent with Each Rating Quality of Audit Write-Up9125% 23%31%12%9%2.6

21. 21Managed by UT-Battelle for the Department of Energy Communication of Audit Findings  Audit write-ups do not convey educational opportunities  Work order not always clear or specific enough for installer to understand the desired outcome  Tools that improved information transfer – Pre-work walk through with installer – Attaching photos of critical areas to the work order – In process check-ups with the installers  Air sealing is a particular area for improved communication  Need for flexibility to change work order when problems are found during installations

22. 22Managed by UT-Battelle for the Department of Energy Audit Strengths  Inspection – 99% inspected every accessible room  Heating system – 94% inspected  Filters – 72% inspected  Ventilation – 72% inspected the kitchen – 78% inspected the bathroom ventilation.  Water heater – 93% inspected  Air conditioning – 82% inspected outside unit.  Insulation – 89% measured all accessible attics.  Testing – 97% conducted blower door test (84% correctly)

23. 23Managed by UT-Battelle for the Department of Energy Audit Opportunities for Improvement  Bathroom ventilation flow – 24% assessed  Hot water – 39% checked temperature at the faucet  Shower flow – none of the auditors measured flow  Insulation – 49% measured in exterior walls.  Blower door – 67% used while inspecting for leaks  Zonal pressure diagnostics – 42% done (87% correctly)  IR camera – 49% used (70% correctly)

24. 24Managed by UT-Battelle for the Department of Energy Audit Findings Opportunities  Utilize energy bills  Increase client engagement  Identify client-specific opportunities for reducing energy use  Auditor understanding of pressure boundaries  Auditor understanding of testing purpose and procedures  Conduct worst case draft testing Strengths  Respect and concern for clients  Explanation of WAP program and process  Use of data collection forms  Auditors meet with contractors to explain audit findings and work scope

25. 25Managed by UT-Battelle for the Department of Energy PROCESS FIELD STUDY MEASURE INSTALLATION OBSERVATION FINDINGS

26. 26Managed by UT-Battelle for the Department of Energy Measure Installation Observations Check Lists  Measure installation preparation  Client interaction  Introduction  Air sealing procedures  Attic insulation  Thermostat installation  CFLs Ratings  Windows – sealed, interior storm, exterior storm, sash kit, repair  Doors – weather stripping, treated, replacement, installation  Air sealing (8 different locations rated)  Overall air sealing quality, cleanliness  Attic insulation quality, cleanliness  Attic hatch work quality  Walk-up attic work quality, cleanliness  Wall insulation quality, cleanliness, siding preservation/interior wall finishing  Basement insulation quality, cleanliness  Crawl space - vapor barrier insulation, perimeter wall insulation, vents sealing, ceiling insulation and air barrier installation  Ventilation – existing fans ducted to outside, whole house, attic, kitchen, bathroom

27. 27Managed by UT-Battelle for the Department of Energy Measure Installation Observations Ratings (continued)  Heating system – tune-up, repair, replacement, installation, vent correction  Thermostat installation  Air conditioning – sealing around, tune-up, replacement  Evaporative cooler – repair, replacement  Duct sealing, duct insulation, duct improvement cleanliness  Hot water temperature adjustment, heater wrapped, pipes insulated, heater repaired, replaced, heater venting work  Subcontractor treatment of refrigerator and home  Dryer venting  Low cost – AC cooling coils cleaned, HVAC filters cleaned or replaced, faucet aerators, showerheads, CO detector, smoke detector, fire extinguisher  Crew efficiency, professionalism, cleanliness, safety practices  Communication, attempt to engage client, client engagement, education quantity, education quality

28. 28Managed by UT-Battelle for the Department of Energy Crews and Contractors Observed NumberPercent Crews 3531% Contractors 7465% Both 54% TOTAL 114100%

29. 29Managed by UT-Battelle for the Department of Energy Measure Installation Preparation Information Available NumberPercent Household demographics 5246% Audit report 5649% Work order 11197% Materials list 7364% Out of 114 observations.

30. 30Managed by UT-Battelle for the Department of Energy Air Sealing Rating Design Quality of Air Sealing in Specific Places 1 Fire rated materials used around chimneys, flue pipes, recessed lighting, etc. 2 Materials integrated with existing finishes 3 Used supply-air respirators when using two-part foams 4 Blower door used to target air sealing 5 Blower door and smoke used to verify air sealing effectiveness 6 Caulk not used to seal gaps larger than 1/8” 7 Large holes and gaps covered with rigid materials 8 Area cleaned before caulk or foam applied 9 All material scraps and packaging removed from home 10 Materials installed consistently and are uniform in appearance 11 Gloves and eye protection used when cutting, using caulk or foam 12 Used hard hats when working in confined spaces Needs ImprovementExcellent Rating12345 Total Points needed 0-56-89-101112 Bold Points needed00055

31. 31Managed by UT-Battelle for the Department of Energy Measure Installation Air Sealing Ratings Number Rated Needs Improvement Excellent Mean Rating Rating – Quality of work12345 Percent with Each Rating Attic floor penetrations5810% 41%34%9%5%2.6 Knee walls195% 58%37%0% 2.3 Second floor rim joist30% 33%67%0% 2.7 Basement penetrations to outside248% 46% 0% 2.4 Basement to conditioned space1217% 50%33%0% 2.2 Crawl space to conditioned space140% 64%29%7%0%2.4 Crawl space to outside90% 56%44%0% 2.4 Garage to conditioned space80% 63%38%0% 2.4

32. 32Managed by UT-Battelle for the Department of Energy Measure Installation Strengths  Air sealing prioritized the right areas – 77 percent prioritized sealing at the top and bottom of the envelope.  Attic insulation followed best practices – 77 percent completed attic floor sealing prior to insulation, 88 percent had no gaps or voids, and 95 percent used appropriate materials.

33. 33Managed by UT-Battelle for the Department of Energy Measure Installation Mid-High Ratings AreaRatings Window work 3.5 – 4.0 Door work 2.8 – 3.5 Attic insulation 4.0 Wall insulation 2.9 Basement insulation 3.6 Crawl space work 3.1 – 4.0 Ventilation work 4.1 – 4.8 Thermostat installation 4.1 Air conditioning work 3.0 – 4.0 Water heater work 3.1 – 4.0

34. 34Managed by UT-Battelle for the Department of Energy Measure Installation Opportunities for Improvement  Air sealing diagnostics – 22 percent used the blower door to guide air sealing and 11 percent used the zonal pressure test to affirm appropriate pressure boundaries.  Air sealing – 57 percent sealed all major opportunities. AreaRatings Air sealing 2.4 Heating system work 2.0 – 2.5 Smoke detector installation 2.6 CO detector installation 2.0 Low Ratings

35. 35Managed by UT-Battelle for the Department of Energy Measure Installation Findings Opportunities  Increased use of blower door when air sealing  Respect for clients’ homes (booties, covering furniture)  Crew member safety  Increased assessment of HVAC contractors  Explain CFLs when installing  Client education Strengths  Crew flexibility and dedication (above and beyond work scope)  Crew problem solving  Some very high quality work  Experienced HVAC contractors

36. 36Managed by UT-Battelle for the Department of Energy PROCESS FIELD STUDY FINAL INSPECTION OBSERVATION FINDINGS

37. 37Managed by UT-Battelle for the Department of Energy Final Inspection Observations Check Lists  Home walkthrough  Diagnostic tests*  Combustion safety tests*  Draft tests*  Occupant interaction *Should have been performed, was performed, performed correctly Ratings  Inspection completeness  Testing completeness  Testing quality  Inspector attempt to engage client  Client engagement  Inspector communication skills  Education quantity  Education quality  Inspector efficiency  Inspector professionalism  Inspector cleanliness  Inspector safety practices

38. 38Managed by UT-Battelle for the Department of Energy Final Inspection Occupant Interaction Applicable Number Work Completed #% Visit expected 918897% Adult occupant present 918796% Discussed work performed 896775% Discussed energy/cost savings 892528% Discussed health & safety issues 892629% Discussed job satisfaction 906370%

39. 39Managed by UT-Battelle for the Department of Energy Final Inspection CO Combustion Safety Tests Tests Performed Should Have Been Should Have Been & Was % Performed % Performed Correctly Heating system 543972%97% Water heater 503876%95% Gas stove 26831%100% Space Heater 700%N/A Ambient 451840%100%

40. 40Managed by UT-Battelle for the Department of Energy Number Rated Needs Improvement Excellent Mean Rating 12345 Percent with Each Rating Quality of Testing 8816%17%19%13%35%3.3 Final Inspection Testing Quality Rating Design Quality of Testing 1 Inspector test equipment calibration current 2 Tests done correctly 3 Test results used to grade work quality 4 Inspectors proficient in operating test equipment 5 Combustion appliances disabled during blower door and duct testing

41. 41Managed by UT-Battelle for the Department of Energy Final Inspection Strengths  The inspectors discussed the work that was performed – 75 percent discussed the work that was performed with the client.  Inspections were complete – 90 percent examined all accessible rooms.  Blower door testing was usually conducted – 85 percent conducted this test (85 percent of those performed correctly.)

42. 42Managed by UT-Battelle for the Department of Energy Final Inspection Opportunities for Improvement  Zonal pressure test – 33 percent performed the test (87 percent done correctly.)  IR camera – 44 percent used the camera (95 percent done correctly.)  Combustion safety – 31 percent performed this test on the gas stove (100 percent done correctly.)  Worst case draft test – 60 percent performed this test on the heating system (92 percent correctly.)

43. 43Managed by UT-Battelle for the Department of Energy Final Inspection Findings Opportunities  Increased client education – Explain measures installed – Reinforce client action plan  Improved testing quality  Increased assessment of installation quality  Reduced use of auditor to conduct final inspection – additional perspective Strengths  Good job when following work order closely  Time saved when contractor attends  Information to clients – who to call if they have problems  Referrals to additional assistance programs

44. 44Managed by UT-Battelle for the Department of Energy SUMMARY

45. 45Managed by UT-Battelle for the Department of Energy Average Ratings By Agency and Overall

46. 46Managed by UT-Battelle for the Department of Energy Training Needs  Energy bill utilization – to assess needs and educate clients  Building science fundamentals  Critical thinking – assessing unique situations  Combustion safety testing  Zonal pressure testing  IR camera use  Ventilation assessment  Safe work practices – worker safety and lead safe work  Interviewing skills – how to understand the client’s needs  Client education

47. 47Managed by UT-Battelle for the Department of Energy Equipment Needs  GPS  IR camera  Boroscope and fiber optic scope and video  Personal safety equipment

48. 48Managed by UT-Battelle for the Department of Energy Management Opportunities  Standards and procedures  Policy manuals  Forms and checklists