1Managed by UT-Battelle for the Department of Energy Jackie Berger APPRISE Quantitative Findings from On-Site Evaluation of Energy Efficiency Program Service Delivery

2Managed by UT-Battelle for the Department of Energy Presentation Overview  Process Field Study design  Sample of findings – Audit observation findings – Measure installation observation findings – Final inspection observation findings  Program recommendations – Training – Equipment – Management

3Managed by UT-Battelle for the Department of Energy PROCESS FIELD STUDY DESIGN

4Managed by UT-Battelle for the Department of Energy Process Field Study Focus  Audit approach and implementation  Energy education delivery  Measure installation/operation of crews  Post-weatherization quality assurance inspection approach and implementation  Client interaction  Interactions among agency staff  Implications for management, procedures, and training

5Managed by UT-Battelle for the Department of Energy Process Field Study Approach  Go beyond anecdotes  Quantify findings across all visits – Develop check lists and rating scales – Train weatherization experts to implement consistently  Enrich data with descriptive information  Make recommendations for program improvement based on significant issues

6Managed by UT-Battelle for the Department of Energy Check List Example Audit Air Leakage and Insulation Diagnostics Applicable Observations Action Taken #% Measured surfaces % Inspected windows % Inspected all accessible attics % Measured insulation in all accessible attics % Created access to inaccessible attics 33310% Measured insulation in exterior walls % Measured insulation in basement/crawlspace % Inspected for all typical bypasses % Visual inspection for air sealing opportunities % Used blower door while inspecting for leaks %

7Managed by UT-Battelle for the Department of Energy Rating Design Rating1234*5* % of Points needed0%-49%50%-74%75%-89%90%-99%100% Total pointsNumber of Points needed for each rating

8Managed by UT-Battelle for the Department of Energy Rating Example Attic Insulation Quality Quality of Attic Insulation 1 All air sealing work completed first 2 Exhaust fans vented to exterior as needed 3 Heat producing devices or systems protected from insulation contact 4 Attic checked for knob and tube wiring 5 Workers wore respirators, safety glasses, gloves, and hard hats while insulating attic 6 Insulation installed in sufficient quantity (bags per ft 2 ) to meet R-value requirement 7 Proper insulation material chosen for attic conditions 8 Open blow insulation is level and of consistent depth 9 Attic ventilation maintained 10 Confined areas blown to dense pack 11 Proper containment used to protect client and belongings Needs Improvement Excellent Mean Rating# Rated Rating12345 Total Points needed Bold Points needed00066 Attic Insulation Quality4%18%42%19%18%3.357

9Managed by UT-Battelle for the Department of Energy Field Study Flow Chart Draft Forms and Procedures Test Draft Forms and Procedures Revise Forms and Procedures Convene Expert Panel Revise Forms and Procedures Select Agency Sample Conduct Interviews and Observations Wx Expert and Social Scientist Debriefing Analyze Data and Summaries Report Test Forms and Procedures Revise Forms and Procedures Wx Expert Training Social Scientist Training Revise Forms and Procedures

10Managed by UT-Battelle for the Department of Energy Agency Selection  WAP Agencies – Almost 1,000 deliver services across the country – 400 comprised agency billing data sample – Field Process Study – 20 agencies in 20 different states  Selection based on: – Number of states per region based on WAP funding – State selection based on part of region and program size – Agencies selected to represent: Urban/RuralEducation } Based on agency survey PY 2008 planned jobsTraining Measure selectionQuality Control

11Managed by UT-Battelle for the Department of Energy Observation Team  Social Scientists – One week observation at agency  Weatherization manager and staff interviews  Observations – client intake, audit, measure installation, final inspection – Written agency summary report  Weatherization Experts – 3-Week Observations at Agency  Observations – audit, measure installation, final inspection – Post Observation Data Entry  Training  Monthly update calls  Final debriefing

12Managed by UT-Battelle for the Department of Energy Process Field Study Observations Number of Observations Social Scientist Weatherization Expert Total Audit Measure Installation Final Inspection Total

13Managed by UT-Battelle for the Department of Energy PROCESS FIELD STUDY AUDIT OBSERVATION FINDINGS

14Managed by UT-Battelle for the Department of Energy Audit Observations Check Lists  Audit preparation  Audit introduction  Audit home walk through  Heating assessment  Air conditioning assessment  Ventilation assessment  Water heater assessment  Air leakage and insulation diagnostics  Diagnostic testing*  Combustion safety testing*  Refrigerator assessment  Lighting assessment  Exit interview *Should have been performed, was performed, performed correctly **Described on write-up, recorded thoroughly and accurately Ratings  Audit introduction  Auditor attempt to engage client  Client engagement in audit  Auditor communication skills  Quantity of education  Quality of education  Exit interview  Completeness of inspection  Completeness of testing  Quality of testing  Auditor efficiency  Auditor professionalism  Auditor respect of client’s home  Auditor safety practices  Audit comprehensiveness*  Planned work comprehensiveness*  Audit write up *Excellent, good, poor

15Managed by UT-Battelle for the Department of Energy Audit Introduction IntroLater Intro or Later #%#%#% Explained WAP 7471%66%8077% Explained auditor’s role in WAP 7673%44%8077% Explained client’s role in WAP 3938%99%4846% Asked what client hoped to get from WAP 2726%44%3130% Reviewed energy bill(s) 1212%00%1212% Discussed health and safety 6361%44%6764% Discussed comfort 5957%22%6159% Asked about problems with energy usage 3433%11%3534% Out of 104 observations completed.

16Managed by UT-Battelle for the Department of Energy Audit Introduction Rating Design Quality of Introduction 1Auditor introduced self 2Auditor explained purpose of visit 3Auditor explained WAP process 4Auditor reviewed energy bills with client 5Auditor asked about comfort issues 6Auditor asked about energy issues 7Auditor asked about H&S issues 8Auditor assessed client’s desired outcomes Needs Improvement Excellent Rating12345 Total Points needed Bold Points needed00033

17Managed by UT-Battelle for the Department of Energy Audit Introduction Rating Needs Improvement Excellent Rating12345 Number of Audits Percent of Audits31%32%24%11%2% Mean Rating2.2

18Managed by UT-Battelle for the Department of Energy Audit Diagnostic Tests Tests Performed Should Have Been Should Have Been & Was % Performed % Performed Correctly Blower door testing %84% Zonal pressure diagnostics %87% Infrared camera %70% Steady state efficiency %98% Air flow test on forced air furnace %93% Proper coolant charge 500%N/A

19Managed by UT-Battelle for the Department of Energy Audit Write-Up Rating Design Quality of Audit Write-Up 1Did not fail to clearly document need for lead safe weatherization if needed 2Included estimated material quantities 3Specified any needed materials 4Clearly documented client health and safety issues 5Recommended measures prioritized by savings to investment ratio 6Set air leakage reduction targets for crews 7Directed crews to access areas inaccessible during the audit 8Contained clear directions for crews 9Documented client concerns/needs 10Documented client commitments if made 11Included special instructions if needed 12Clear and easy to understand

20Managed by UT-Battelle for the Department of Energy Quality of Audit Write-Up Rating Number Rated Needs Improvement Excellent Mean Rating Percent with Each Rating Quality of Audit Write-Up9125% 23%31%12%9%2.6

21Managed by UT-Battelle for the Department of Energy Communication of Audit Findings  Audit write-ups do not convey educational opportunities  Work order not always clear or specific enough for installer to understand the desired outcome  Tools that improved information transfer – Pre-work walk through with installer – Attaching photos of critical areas to the work order – In process check-ups with the installers  Air sealing is a particular area for improved communication  Need for flexibility to change work order when problems are found during installations

22Managed by UT-Battelle for the Department of Energy Audit Strengths  Inspection – 99% inspected every accessible room  Heating system – 94% inspected  Filters – 72% inspected  Ventilation – 72% inspected the kitchen – 78% inspected the bathroom ventilation.  Water heater – 93% inspected  Air conditioning – 82% inspected outside unit.  Insulation – 89% measured all accessible attics.  Testing – 97% conducted blower door test (84% correctly)

23Managed by UT-Battelle for the Department of Energy Audit Opportunities for Improvement  Bathroom ventilation flow – 24% assessed  Hot water – 39% checked temperature at the faucet  Shower flow – none of the auditors measured flow  Insulation – 49% measured in exterior walls.  Blower door – 67% used while inspecting for leaks  Zonal pressure diagnostics – 42% done (87% correctly)  IR camera – 49% used (70% correctly)

24Managed by UT-Battelle for the Department of Energy Audit Findings Opportunities  Utilize energy bills  Increase client engagement  Identify client-specific opportunities for reducing energy use  Auditor understanding of pressure boundaries  Auditor understanding of testing purpose and procedures  Conduct worst case draft testing Strengths  Respect and concern for clients  Explanation of WAP program and process  Use of data collection forms  Auditors meet with contractors to explain audit findings and work scope

25Managed by UT-Battelle for the Department of Energy PROCESS FIELD STUDY MEASURE INSTALLATION OBSERVATION FINDINGS

26Managed by UT-Battelle for the Department of Energy Measure Installation Observations Check Lists  Measure installation preparation  Client interaction  Introduction  Air sealing procedures  Attic insulation  Thermostat installation  CFLs Ratings  Windows – sealed, interior storm, exterior storm, sash kit, repair  Doors – weather stripping, treated, replacement, installation  Air sealing (8 different locations rated)  Overall air sealing quality, cleanliness  Attic insulation quality, cleanliness  Attic hatch work quality  Walk-up attic work quality, cleanliness  Wall insulation quality, cleanliness, siding preservation/interior wall finishing  Basement insulation quality, cleanliness  Crawl space - vapor barrier insulation, perimeter wall insulation, vents sealing, ceiling insulation and air barrier installation  Ventilation – existing fans ducted to outside, whole house, attic, kitchen, bathroom

27Managed by UT-Battelle for the Department of Energy Measure Installation Observations Ratings (continued)  Heating system – tune-up, repair, replacement, installation, vent correction  Thermostat installation  Air conditioning – sealing around, tune-up, replacement  Evaporative cooler – repair, replacement  Duct sealing, duct insulation, duct improvement cleanliness  Hot water temperature adjustment, heater wrapped, pipes insulated, heater repaired, replaced, heater venting work  Subcontractor treatment of refrigerator and home  Dryer venting  Low cost – AC cooling coils cleaned, HVAC filters cleaned or replaced, faucet aerators, showerheads, CO detector, smoke detector, fire extinguisher  Crew efficiency, professionalism, cleanliness, safety practices  Communication, attempt to engage client, client engagement, education quantity, education quality

28Managed by UT-Battelle for the Department of Energy Crews and Contractors Observed NumberPercent Crews 3531% Contractors 7465% Both 54% TOTAL %

29Managed by UT-Battelle for the Department of Energy Measure Installation Preparation Information Available NumberPercent Household demographics 5246% Audit report 5649% Work order 11197% Materials list 7364% Out of 114 observations.

30Managed by UT-Battelle for the Department of Energy Air Sealing Rating Design Quality of Air Sealing in Specific Places 1 Fire rated materials used around chimneys, flue pipes, recessed lighting, etc. 2 Materials integrated with existing finishes 3 Used supply-air respirators when using two-part foams 4 Blower door used to target air sealing 5 Blower door and smoke used to verify air sealing effectiveness 6 Caulk not used to seal gaps larger than 1/8” 7 Large holes and gaps covered with rigid materials 8 Area cleaned before caulk or foam applied 9 All material scraps and packaging removed from home 10 Materials installed consistently and are uniform in appearance 11 Gloves and eye protection used when cutting, using caulk or foam 12 Used hard hats when working in confined spaces Needs ImprovementExcellent Rating12345 Total Points needed Bold Points needed00055

31Managed by UT-Battelle for the Department of Energy Measure Installation Air Sealing Ratings Number Rated Needs Improvement Excellent Mean Rating Rating – Quality of work12345 Percent with Each Rating Attic floor penetrations5810% 41%34%9%5%2.6 Knee walls195% 58%37%0% 2.3 Second floor rim joist30% 33%67%0% 2.7 Basement penetrations to outside248% 46% 0% 2.4 Basement to conditioned space1217% 50%33%0% 2.2 Crawl space to conditioned space140% 64%29%7%0%2.4 Crawl space to outside90% 56%44%0% 2.4 Garage to conditioned space80% 63%38%0% 2.4

32Managed by UT-Battelle for the Department of Energy Measure Installation Strengths  Air sealing prioritized the right areas – 77 percent prioritized sealing at the top and bottom of the envelope.  Attic insulation followed best practices – 77 percent completed attic floor sealing prior to insulation, 88 percent had no gaps or voids, and 95 percent used appropriate materials.

33Managed by UT-Battelle for the Department of Energy Measure Installation Mid-High Ratings AreaRatings Window work 3.5 – 4.0 Door work 2.8 – 3.5 Attic insulation 4.0 Wall insulation 2.9 Basement insulation 3.6 Crawl space work 3.1 – 4.0 Ventilation work 4.1 – 4.8 Thermostat installation 4.1 Air conditioning work 3.0 – 4.0 Water heater work 3.1 – 4.0

34Managed by UT-Battelle for the Department of Energy Measure Installation Opportunities for Improvement  Air sealing diagnostics – 22 percent used the blower door to guide air sealing and 11 percent used the zonal pressure test to affirm appropriate pressure boundaries.  Air sealing – 57 percent sealed all major opportunities. AreaRatings Air sealing 2.4 Heating system work 2.0 – 2.5 Smoke detector installation 2.6 CO detector installation 2.0 Low Ratings

35Managed by UT-Battelle for the Department of Energy Measure Installation Findings Opportunities  Increased use of blower door when air sealing  Respect for clients’ homes (booties, covering furniture)  Crew member safety  Increased assessment of HVAC contractors  Explain CFLs when installing  Client education Strengths  Crew flexibility and dedication (above and beyond work scope)  Crew problem solving  Some very high quality work  Experienced HVAC contractors

36Managed by UT-Battelle for the Department of Energy PROCESS FIELD STUDY FINAL INSPECTION OBSERVATION FINDINGS

37Managed by UT-Battelle for the Department of Energy Final Inspection Observations Check Lists  Home walkthrough  Diagnostic tests*  Combustion safety tests*  Draft tests*  Occupant interaction *Should have been performed, was performed, performed correctly Ratings  Inspection completeness  Testing completeness  Testing quality  Inspector attempt to engage client  Client engagement  Inspector communication skills  Education quantity  Education quality  Inspector efficiency  Inspector professionalism  Inspector cleanliness  Inspector safety practices

38Managed by UT-Battelle for the Department of Energy Final Inspection Occupant Interaction Applicable Number Work Completed #% Visit expected % Adult occupant present % Discussed work performed % Discussed energy/cost savings % Discussed health & safety issues % Discussed job satisfaction %

39Managed by UT-Battelle for the Department of Energy Final Inspection CO Combustion Safety Tests Tests Performed Should Have Been Should Have Been & Was % Performed % Performed Correctly Heating system %97% Water heater %95% Gas stove 26831%100% Space Heater 700%N/A Ambient %100%

40Managed by UT-Battelle for the Department of Energy Number Rated Needs Improvement Excellent Mean Rating Percent with Each Rating Quality of Testing 8816%17%19%13%35%3.3 Final Inspection Testing Quality Rating Design Quality of Testing 1 Inspector test equipment calibration current 2 Tests done correctly 3 Test results used to grade work quality 4 Inspectors proficient in operating test equipment 5 Combustion appliances disabled during blower door and duct testing

41Managed by UT-Battelle for the Department of Energy Final Inspection Strengths  The inspectors discussed the work that was performed – 75 percent discussed the work that was performed with the client.  Inspections were complete – 90 percent examined all accessible rooms.  Blower door testing was usually conducted – 85 percent conducted this test (85 percent of those performed correctly.)

42Managed by UT-Battelle for the Department of Energy Final Inspection Opportunities for Improvement  Zonal pressure test – 33 percent performed the test (87 percent done correctly.)  IR camera – 44 percent used the camera (95 percent done correctly.)  Combustion safety – 31 percent performed this test on the gas stove (100 percent done correctly.)  Worst case draft test – 60 percent performed this test on the heating system (92 percent correctly.)

43Managed by UT-Battelle for the Department of Energy Final Inspection Findings Opportunities  Increased client education – Explain measures installed – Reinforce client action plan  Improved testing quality  Increased assessment of installation quality  Reduced use of auditor to conduct final inspection – additional perspective Strengths  Good job when following work order closely  Time saved when contractor attends  Information to clients – who to call if they have problems  Referrals to additional assistance programs

44Managed by UT-Battelle for the Department of Energy SUMMARY

45Managed by UT-Battelle for the Department of Energy Average Ratings By Agency and Overall

46Managed by UT-Battelle for the Department of Energy Training Needs  Energy bill utilization – to assess needs and educate clients  Building science fundamentals  Critical thinking – assessing unique situations  Combustion safety testing  Zonal pressure testing  IR camera use  Ventilation assessment  Safe work practices – worker safety and lead safe work  Interviewing skills – how to understand the client’s needs  Client education

47Managed by UT-Battelle for the Department of Energy Equipment Needs  GPS  IR camera  Boroscope and fiber optic scope and video  Personal safety equipment

48Managed by UT-Battelle for the Department of Energy Management Opportunities  Standards and procedures  Policy manuals  Forms and checklists