1Managed by UT-Battelle for the Department of Energy Findings from the National Weatherization Evaluation Process Field Study Jackie Berger APPRISE Training: Evaluating Effectiveness and Impact on Work Quality

2Managed by UT-Battelle for the Department of Energy Presentation Overview  Process Field Study design  Audit observation findings  Measure installation observation findings  Final inspection observation findings  Training needs, challenges, and strategies

3Managed by UT-Battelle for the Department of Energy PROCESS FIELD STUDY DESIGN

4Managed by UT-Battelle for the Department of Energy Process Field Study Focus  Audit approach and implementation  Energy education delivery  Measure installation/operation of crews  Post-weatherization quality assurance inspection approach and implementation  Client interaction  Interactions among agency staff  Implications for management, procedures, and training

5Managed by UT-Battelle for the Department of Energy Process Field Study Goals Agency/Staff Weatherization Approach Service Delivery Characteristics Quality of Service Delivery Quantity and Quality of Client Interaction

6Managed by UT-Battelle for the Department of Energy Synthesis Training Experience Procedures Inputs In field application Utilization of knowledge and tools Implementation Client impact Knowledge, behavior, comfort Energy bills Results

7Managed by UT-Battelle for the Department of Energy Field Study Flow Chart Draft Forms and Procedures Test Draft Forms and Procedures Revise Forms and Procedures Convene Expert Panel Revise Forms and Procedures Select Agency Sample Conduct Interviews and Observations Wx Expert and Social Scientist Debriefing Analyze Data and Summaries Report Test Forms and Procedures Revise Forms and Procedures Wx Expert Training Social Scientist Training Revise Forms and Procedures

8Managed by UT-Battelle for the Department of Energy Agency Selection  Number of states per region based on WAP funding  State selection based on part of region and program size  Agencies selected to represent: Urban/RuralEducation } Based on agency survey PY 2008 planned jobsTraining Measure selectionQuality Control

9Managed by UT-Battelle for the Department of Energy Social Scientist Role  Training  One week observation at agency – Weatherization manager and staff interviews – Observations  Client intake  Audit  Measure installation  Final inspection  Written agency summary report  Monthly update calls  Final debriefing

10Managed by UT-Battelle for the Department of Energy Weatherization Expert Role  Training  3-Week Observations at Agency – Audit observation – Measure installation observation – Final inspection observation  Post Observation Data Entry  Monthly Update Calls  Final Debriefing

11Managed by UT-Battelle for the Department of Energy Rating Design Rating1234*5* % of Points needed0%-49%50%-74%75%-89%90%-99%100% Total pointsNumber of Points needed for each rating

12Managed by UT-Battelle for the Department of Energy Process Field Study Observations Number of Observations Social Scientist Weatherization Expert Total Audit Measure Installation Final Inspection Total

13Managed by UT-Battelle for the Department of Energy PROCESS FIELD STUDY AUDIT OBSERVATION FINDINGS

14Managed by UT-Battelle for the Department of Energy Audit Preparation Applicable Observations Information Was Available NumberPercent Demographics % Natural Gas Usage % Bulk Fuel Usage 31723% Electric Usage % End Use Breakout 11076%

15Managed by UT-Battelle for the Department of Energy Audit Introduction IntroLater Intro or Later #%#%#% Explained WAP 7471%66%8077% Explained auditor’s role in WAP 7673%44%8077% Explained client’s role in WAP 3938%99%4846% Asked what client hoped to get from WAP 2726%44%3130% Reviewed energy bill(s) 1212%00%1212% Discussed health and safety 6361%44%6764% Discussed comfort 5957%22%6159% Asked about problems with energy usage 3433%11%3534% Out of 104 observations completed.

16Managed by UT-Battelle for the Department of Energy Audit Introduction Rating Design Quality of Introduction 1Auditor introduced self 2Auditor explained purpose of visit 3Auditor explained WAP process 4Auditor reviewed energy bills with client 5Auditor asked about comfort issues 6Auditor asked about energy issues 7Auditor asked about H&S issues 8Auditor assessed client’s desired outcomes Needs Improvement Excellent Rating12345 Total Points needed Bold Points needed00033

17Managed by UT-Battelle for the Department of Energy Audit Introduction Rating Needs Improvement Excellent Rating12345 Number of Audits Percent of Audits31%32%24%11%2% Mean Rating2.2

18Managed by UT-Battelle for the Department of Energy Audit Home Walkthrough Applicable Number WalkthroughLater Walkthrough or Later #%#%#% Inspected every accessible room %00%10798% Did outside home walk around %33%10093% Had systematic inspection method % % Discussed energy usage with client %76%3936% Discussed actions to reduce usage %76%2321% Discussed $ savings from behavior changes 10844%22%66%

19Managed by UT-Battelle for the Department of Energy Audit Heating Assessment Applicable Observations Work Completed #% Inspected heating system % Discussed whether part of home is cold % Discussed thermostat settings % Discussed night time heat setback % Discussed heat setback when not home % Asked about supplemental heating usage % Discussed heating system health & safety %

20Managed by UT-Battelle for the Department of Energy Audit Water Heater Assessment Applicable Observations Work Completed #% Inspected water heater % Checked hot water temperature at faucet % Measured shower water flow 9900% Discussed efficient hot water usage 9988%

21Managed by UT-Battelle for the Department of Energy Audit Air Leaking and Insulation Diagnostics Applicable Observations Action Taken #% Measured surfaces % Inspected windows % Inspected all accessible attics % Measured insulation in all accessible attics % Created access to inaccessible attics 33310% Measured insulation in exterior walls % Measured insulation in basement/crawlspace % Inspected for all typical bypasses % Visual inspection for air sealing opportunities % Used blower door while inspecting for leaks %

22Managed by UT-Battelle for the Department of Energy Audit Diagnostic Tests Tests Performed Should Have Been Should Have Been & Was % Performed % Performed Correctly Blower door testing %84% Zonal pressure diagnostics %87% Infrared camera %70% Steady state efficiency %98% Air flow test on forced air furnace %93% Proper coolant charge 500%N/A

23Managed by UT-Battelle for the Department of Energy Audit Combustion Safety Tests Tests Performed Should Have Been Should Have Been & Was % Performed % Performed Correctly Ambient CO level in CAZ %98% Ambient CO level outside CAZ %100% Gas/propane/fuel oil leaks %83% Spillage testing on all atmospheric furnaces, boilers, water heaters %95% Heating system CO level %94% Water heating system CO level %93%

24Managed by UT-Battelle for the Department of Energy Audit Combustion Safety Tests Tests Performed Should Have Been Should Have Been & Was % Performed % Performed Correctly Draft test on heating system (worst case, natural conditions) %95% Draft test on water heater (worst case, natural conditions) %88% Measure CAZ for sufficient volume for combustion air %85% Gas/propane ranges CO level %75% Gas/propane stove CO level %77%

25Managed by UT-Battelle for the Department of Energy Audit Exit Interview Applicable Observations Action Taken #% Reviewed bills with client 10177% Provided context (high/low/average) for usage % Summarized findings for client % Discussed health and safety issues % Discussed comfort issues % Discussed measure options % Summarized behavior change opportunities % Discussed potential $ savings from behavior change 10077% Obtained client commitment for behavior change 10077% Recommended behavior changes w/ potential for saving % Provided information on next steps %

26Managed by UT-Battelle for the Department of Energy Audit Exit Interview Rating Design Quality of Exit Interview 1 Auditor explained next steps 2 Auditor reviewed discussed behavior changes 3 Auditor discussed potential $ saved from behavior change 4 Auditor explained client actions with the greatest impact 5 Auditor assessed client’s desired outcomes 6 Auditor obtained client commitments for energy saving actions 7 Client was asked if they have any concerns 8 Auditor explained the work proposed Needs Improvement Excellent Rating12345 Total Points needed Bold Points needed00011

27Managed by UT-Battelle for the Department of Energy Audit Exit Interview Rating Needs Improvement Excellent Rating12345 Number of Audits Percent of Audits55%35%7%2%1% Mean Rating1.6

28Managed by UT-Battelle for the Department of Energy Audit Ratings Number Rated Needs Improvement Excellent Mean Rating Percent with Each Rating Auditor attempt to engage client10045% 30%6%14%5%2.0 Client engagement10040% 22%29%6%3%2.1 Auditor communication skills10015%43%32%2%8%2.5 Quantity of education10061%26%8%5%0%1.6 Quality of education10048%40% 6% 5%1%1.7 Completeness of inspection1038%32% 28% 17%15%3.0 Completeness of testing10439%37% 20% 4%0%1.9 Quality of testing10435%25% 10% 25%6%2.4

29Managed by UT-Battelle for the Department of Energy Audit Ratings Number Rated Needs Improvement Excellent Mean Rating Percent with Each Rating Audit efficiency1040% 10%26%34%31%3.9 Auditor professionalism1031% 10%19%41%29%3.9 Auditor respect of client’s home1044% 11%2% 38%46%4.1 Auditor safety practices10315% 21%24% 14%26%3.2

30Managed by UT-Battelle for the Department of Energy Communication of Audit Findings  Audit write-ups do not convey educational opportunities  Work order not always clear or specific enough for installer to understand the desired outcome  Tools that improved information transfer – Pre-work walk through with installer – Attaching photos of critical areas to the work order – In process check-ups with the installers  Air sealing is a particular area for improved communication  Need for flexibility to change work order when problems are found during installations

31Managed by UT-Battelle for the Department of Energy Audit Findings Opportunities  Utilize energy bills  Increase client engagement  Identify client-specific opportunities for reducing energy use  Auditor understanding of pressure boundaries  Auditor understanding of testing purpose and procedures  Conduct worst case draft testing Strengths  Respect and concern for clients  Explanation of WAP program and process  Use of data collection forms  Auditors meet with contractors to explain audit findings and work scope

32Managed by UT-Battelle for the Department of Energy PROCESS FIELD STUDY MEASURE INSTALLATION OBSERVATION FINDINGS

33Managed by UT-Battelle for the Department of Energy Crews and Contractors Observed NumberPercent Crews 3531% Contractors 7465% Both 54% TOTAL %

34Managed by UT-Battelle for the Department of Energy Measure Installation Preparation Information Available NumberPercent Household demographics 5246% Audit report 5649% Work order 11197% Materials list 7364% Out of 114 observations.

35Managed by UT-Battelle for the Department of Energy Measure Installation Introduction Applicable Number IntroLater Intro or Later #%#%#% Explained WAP %33%2929% Explained purpose of visit %11%7877% Explained planned measures %33%8079% Asked if client had questions %55%5756% Applicable Number NumberPercent Visit expected % Adult occupant present %

36Managed by UT-Battelle for the Department of Energy Measure Installation Window and Door Ratings Number Rated Needs Improvement Excellent Mean Rating Rating – Quality of work12345 Percent with Each Rating Windows sealed128% 0%33%50%8%3.5 Interior storm window work40% 100%0%4.0 Sash kit work10% 100%0% 3.0 Window repair120% 8%42% 8%3.5 Window replacement90% 44%56%0%3.6 Door weather stripping3811% 34%32%21%5%2.8 Doors treated138% 15% 46%15%3.5 Door replacement160% 38% 25%0%2.9

37Managed by UT-Battelle for the Department of Energy Measure Installation Air Sealing Ratings Number Rated Needs Improvement Excellent Mean Rating Rating – Quality of work12345 Percent with Each Rating Attic floor penetrations5810% 41%34%9%5%2.6 Knee walls195% 58%37%0% 2.3 Second floor rim joist30% 33%67%0% 2.7 Basement penetrations to outside248% 46% 0% 2.4 Basement to conditioned space1217% 50%33%0% 2.2 Crawl space to conditioned space140% 64%29%7%0%2.4 Crawl space to outside90% 56%44%0% 2.4 Garage to conditioned space80% 63%38%0% 2.4

38Managed by UT-Battelle for the Department of Energy Measure Installation Attic Insulation Applicable Number Action Taken #% Attic floor sealing complete before % Baffles used % Constant insulation depth achieved % No insulation gaps or voids % Appropriate material used %

39Managed by UT-Battelle for the Department of Energy Measure Installation Attic Insulation Number Rated Needs Improvement Excellent Mean Rating Percent with Each Rating Attic insulation quality574% 18%42%19%18%3.3 Attic insulation cleanliness582% 3%31%17%47%4.0

40Managed by UT-Battelle for the Department of Energy Measure Installation Attic Access Work Number Rated Needs Improvement Excellent Mean Rating Percent with Each Rating Attic hatch work quality486% 23%56%13%2%2.8 Walk-up attic work quality367% 0%33%0% 1.7 Attic access work cleanliness472% 9%40%47%4.3

41Managed by UT-Battelle for the Department of Energy Measure Installation Insulation Ratings Number Rated Needs Improvement Excellent Mean Rating Percent with Each Rating Wall insulation quality323% 13%75%6%3%2.9 Wall insulation cleanliness320% 16%3%19%63%4.3 Siding preservation/interior wall3110% 19%26%29%16%3.2 Basement insulation quality220% 50%36%14%3.6 Basement insulation cleanliness220% 14%18%68%4.5

42Managed by UT-Battelle for the Department of Energy Measure Installation Crawl Space Ratings Number Rated Needs Improvement Excellent Mean Rating Rating – Quality of work12345 Percent with Each Rating Vapor barrier installation210% 29%14%48%10%3.4 Perimeter wall insulation50% 20%60%20%4.0 Vents sealing20% 50% 0%3.5 Crawl space ceiling insulation and air barrier installation 166% 31%13%44%6%3.1

43Managed by UT-Battelle for the Department of Energy Measure Installation Ventilation Ratings Number Rated Needs Improvement Excellent Mean Rating Rating – Quality of work12345 Percent with Each Rating Existing fans ducted to outside150% 7%27%67%4.6 Whole house ventilation30% 33%0%67%4.3 Attic ventilation120% 25%75%4.8 Kitchen ventilation80% 13%0%88%4.8 Bathroom ventilation157% 20%0% 73%4.1

44Managed by UT-Battelle for the Department of Energy Measure Installation Heating System Ratings Number Rated Needs Improvement Excellent Mean Rating Rating – Quality of work12345 Percent with Each Rating Heating system tune-up617% 50%0%33%0%2.5 Heating system repair40% 25% 3.5 Heating system replacement714% 57%14% 0%2.3 80% efficient system installation10% 100%0% %+ efficient system installation520% 60%0%20%0%2.2 Heating system vent correction560% 0%20% 0%2.0

45Managed by UT-Battelle for the Department of Energy Measure Installation Air Conditioning Ratings Number Rated Needs Improvement Excellent Mean Rating Rating – Quality of work12345 Percent with Each Rating Sealing around air conditioners20% 50% 0%3.5 Air conditioner tune-up10% 100%0% 3.0 Air conditioner replacement20% 100%0%4.0 Evaporative cooler repair40% 25%0%50%25%3.8 Evaporative cooler replacement20% 50% 0% 2.5

46Managed by UT-Battelle for the Department of Energy Measure Installation Water Heater Ratings Number Rated Needs Improvement Excellent Mean Rating Rating – Quality of work12345 Percent with Each Rating Hot water temperature adjustment20% 50%0%50%4.0 Hot water heater wrapped140% 29%36% 0%3.1 Hot water heater pipes insulated368% 3%25%19%44%3.9 Hot water heater repaired70% 71%14% 3.4 Hot water heater venting work70% 14%0%71%14%3.9

47Managed by UT-Battelle for the Department of Energy Measure Installation Summary Ratings Number Rated Needs Improvement Excellent Mean Rating Percent with Each Rating Efficiency1130% 11%14%31%44%4.1 Professionalism1132% 13%21%24%40%3.9 Cleanliness1137% 5%19%28%40%3.9 Safety practices11114% 41%33%5%6%2.5 Communication1058% 38%7%22%26%3.2 Attempt to engage client10436% 37%9%17%2%2.1 Client engagement10328% 37%24%7%4%2.2 Education quantity9955% 22%10%7%6%1.9 Education quality9870% 25%2% 1%1.4

48Managed by UT-Battelle for the Department of Energy Measure Installation Findings Opportunities  Increased use of blower door when air sealing  Respect for clients’ homes (booties, covering furniture)  Crew member safety  Increased assessment of HVAC contractors  Explain CFLs when installing  Client education Strengths  Crew flexibility and dedication (above and beyond work scope)  Crew problem solving  Some very high quality work  Experienced HVAC contractors

49Managed by UT-Battelle for the Department of Energy PROCESS FIELD STUDY FINAL INSPECTION OBSERVATION FINDINGS

50Managed by UT-Battelle for the Department of Energy Final Inspection Occupant Interaction Applicable Number Work Completed #% Visit expected % Adult occupant present % Discussed work performed % Discussed energy/cost savings % Discussed health & safety issues % Discussed job satisfaction %

51Managed by UT-Battelle for the Department of Energy Final Inspection Home Walkthrough Applicable Number Walk- through Later Walk- through or Later #%#%#% Examined all accessible rooms %22%8290% Noted CO detector presence/absence %00%5965% Noted smoke detector presence/absence %00%5864% Examined all accessible attic(s) %23%4973% Examined basement %00%4596% Examined all accessible crawl spaces %36%3370% Examined garage %00%1575% Conducted outside home walk-around %22%6774%

52Managed by UT-Battelle for the Department of Energy Final Inspection Diagnostic Tests Tests Performed Should Have Been Should Have Been & Was % Performed % Performed Correctly Blower door % Zonal pressure %87% Duct leakage test %79% Infrared Scan %95% Combustion efficiency test %89% Air handler air flow test %94% Proper coolant charge 300%N/A

53Managed by UT-Battelle for the Department of Energy Final Inspection CO Combustion Safety Tests Tests Performed Should Have Been Should Have Been & Was % Performed % Performed Correctly Heating system %97% Water heater %95% Gas stove 26831%100% Space Heater 700%N/A Ambient %100%

54Managed by UT-Battelle for the Department of Energy Final Inspection Draft Tests Tests Performed Should Have Been Should Have Been & Was % Performed % Performed Correctly WCD heating system %92% WCD water heater %83% WCD Other 700%N/A Spillage Testing %96%

55Managed by UT-Battelle for the Department of Energy Final Inspection Summary Ratings Number Rated Needs Improvement Excellent Mean Rating Percent with Each Rating Inspector attempt to engage client8965% 11% 4%8%1.8 Client engagement8844% 39%10%6%1%1.8 Inspector communication skills8922% 40%20%0%17%2.5 Education quantity8665% 22%7%5%1%1.5 Education quality8663% 20%8%6%3%1.7 Inspection completeness8823% 39%24%11%3%2.3 Testing completeness8938% 37%21%0%3%1.9 Testing quality8816% 17%19%13%35%3.3

56Managed by UT-Battelle for the Department of Energy Final Inspection Summary Ratings Number Rated Needs Improvement Excellent Mean Rating Percent with Each Rating Inspector efficiency 890%16%0%27%57%4.3 Inspector professionalism900% 16%30%29%26%3.6 Inspector cleanliness907% 2%7%0%84%4.5 Inspector safety practices892% 3%25%22%47%4.1

57Managed by UT-Battelle for the Department of Energy Final Inspection Findings Opportunities  Increased client education – Explain measures installed – Reinforce client action plan  Improved testing quality  Increased assessment of installation quality  Reduced use of auditor to conduct final inspection – additional perspective Strengths  Good job when following work order closely  Time saved when contractor attends  Information to clients – who to call if they have problems  Referrals to additional assistance programs

58Managed by UT-Battelle for the Department of Energy PROCESS FIELD STUDY TRAINING FINDINGS NEEDS, CHALLENGES, STRATEGIES

59Managed by UT-Battelle for the Department of Energy Training Needs  Building science fundamentals  Critical thinking – assessing unique situations  Combustion safety testing  Ventilation assessment  Safe work practices – Worker safety – Lead safe work  Client education

60Managed by UT-Battelle for the Department of Energy Training Challenges  More clarity needed on specifications and procedures  Sufficient time is not allocated for training  Pressure for job production  Agency management does not always prioritize training

61Managed by UT-Battelle for the Department of Energy Training Strategies  Increased training resources  Combination of classroom and field training  Greater attendance at national conferences  Emphasis on why tests are conducted  Emphasis on client education  Reinforce with monitoring, assessment, and feedback