1. 1Managed by UT-Battelle for the Department of Energy Findings from the National Weatherization Evaluation Process Field Study Jackie Berger APPRISE Training: Evaluating Effectiveness and Impact on Work Quality

2. 2Managed by UT-Battelle for the Department of Energy Presentation Overview  Process Field Study design  Audit observation findings  Measure installation observation findings  Final inspection observation findings  Training needs, challenges, and strategies

3. 3Managed by UT-Battelle for the Department of Energy PROCESS FIELD STUDY DESIGN

4. 4Managed by UT-Battelle for the Department of Energy Process Field Study Focus  Audit approach and implementation  Energy education delivery  Measure installation/operation of crews  Post-weatherization quality assurance inspection approach and implementation  Client interaction  Interactions among agency staff  Implications for management, procedures, and training

5. 5Managed by UT-Battelle for the Department of Energy Process Field Study Goals Agency/Staff Weatherization Approach Service Delivery Characteristics Quality of Service Delivery Quantity and Quality of Client Interaction

6. 6Managed by UT-Battelle for the Department of Energy Synthesis Training Experience Procedures Inputs In field application Utilization of knowledge and tools Implementation Client impact Knowledge, behavior, comfort Energy bills Results

7. 7Managed by UT-Battelle for the Department of Energy Field Study Flow Chart Draft Forms and Procedures Test Draft Forms and Procedures Revise Forms and Procedures Convene Expert Panel Revise Forms and Procedures Select Agency Sample Conduct Interviews and Observations Wx Expert and Social Scientist Debriefing Analyze Data and Summaries Report Test Forms and Procedures Revise Forms and Procedures Wx Expert Training Social Scientist Training Revise Forms and Procedures

8. 8Managed by UT-Battelle for the Department of Energy Agency Selection  Number of states per region based on WAP funding  State selection based on part of region and program size  Agencies selected to represent: Urban/RuralEducation } Based on agency survey PY 2008 planned jobsTraining Measure selectionQuality Control

9. 9Managed by UT-Battelle for the Department of Energy Social Scientist Role  Training  One week observation at agency – Weatherization manager and staff interviews – Observations  Client intake  Audit  Measure installation  Final inspection  Written agency summary report  Monthly update calls  Final debriefing

10. 10Managed by UT-Battelle for the Department of Energy Weatherization Expert Role  Training  3-Week Observations at Agency – Audit observation – Measure installation observation – Final inspection observation  Post Observation Data Entry  Monthly Update Calls  Final Debriefing

11. 11Managed by UT-Battelle for the Department of Energy Rating Design Rating1234*5* % of Points needed0%-49%50%-74%75%-89%90%-99%100% Total pointsNumber of Points needed for each rating 40-123--4 50-234--5 60-23-4--56 70-34567 8 4-5678 90-45-6789 100-45-78910 110-56-78-91011 120-56-89-101112 130-67-910-111213 140-67-910-121314 150-78-1011-131415

12. 12Managed by UT-Battelle for the Department of Energy Process Field Study Observations Number of Observations Social Scientist Weatherization Expert Total Audit 43112155 Measure Installation 45114159 Final Inspection 3791128 Total 125317442

13. 13Managed by UT-Battelle for the Department of Energy PROCESS FIELD STUDY AUDIT OBSERVATION FINDINGS

14. 14Managed by UT-Battelle for the Department of Energy Audit Preparation Applicable Observations Information Was Available NumberPercent Demographics 1109485% Natural Gas Usage 793544% Bulk Fuel Usage 31723% Electric Usage 1104945% End Use Breakout 11076%

15. 15Managed by UT-Battelle for the Department of Energy Audit Introduction IntroLater Intro or Later #%#%#% Explained WAP 7471%66%8077% Explained auditor’s role in WAP 7673%44%8077% Explained client’s role in WAP 3938%99%4846% Asked what client hoped to get from WAP 2726%44%3130% Reviewed energy bill(s) 1212%00%1212% Discussed health and safety 6361%44%6764% Discussed comfort 5957%22%6159% Asked about problems with energy usage 3433%11%3534% Out of 104 observations completed.

16. 16Managed by UT-Battelle for the Department of Energy Audit Introduction Rating Design Quality of Introduction 1Auditor introduced self 2Auditor explained purpose of visit 3Auditor explained WAP process 4Auditor reviewed energy bills with client 5Auditor asked about comfort issues 6Auditor asked about energy issues 7Auditor asked about H&S issues 8Auditor assessed client’s desired outcomes Needs Improvement Excellent Rating12345 Total Points needed0-34-5678 Bold Points needed00033

17. 17Managed by UT-Battelle for the Department of Energy Audit Introduction Rating Needs Improvement Excellent Rating12345 Number of Audits323325112 Percent of Audits31%32%24%11%2% Mean Rating2.2

18. 18Managed by UT-Battelle for the Department of Energy Audit Home Walkthrough Applicable Number WalkthroughLater Walkthrough or Later #%#%#% Inspected every accessible room 10910798%00%10798% Did outside home walk around 1079791%33%10093% Had systematic inspection method 10810799%--10799% Discussed energy usage with client 1083230%76%3936% Discussed actions to reduce usage 1081615%76%2321% Discussed $ savings from behavior changes 10844%22%66%

19. 19Managed by UT-Battelle for the Department of Energy Audit Heating Assessment Applicable Observations Work Completed #% Inspected heating system 1059994% Discussed whether part of home is cold 1033938% Discussed thermostat settings 1043029% Discussed night time heat setback 1031414% Discussed heat setback when not home 1021010% Asked about supplemental heating usage 1053130% Discussed heating system health & safety 913437%

20. 20Managed by UT-Battelle for the Department of Energy Audit Water Heater Assessment Applicable Observations Work Completed #% Inspected water heater 1019493% Checked hot water temperature at faucet 993939% Measured shower water flow 9900% Discussed efficient hot water usage 9988%

21. 21Managed by UT-Battelle for the Department of Energy Audit Air Leaking and Insulation Diagnostics Applicable Observations Action Taken #% Measured surfaces 1009494% Inspected windows 1009696% Inspected all accessible attics 786988% Measured insulation in all accessible attics 797089% Created access to inaccessible attics 33310% Measured insulation in exterior walls 954749% Measured insulation in basement/crawlspace 745574% Inspected for all typical bypasses 1006262% Visual inspection for air sealing opportunities 1008383% Used blower door while inspecting for leaks 966467%

22. 22Managed by UT-Battelle for the Department of Energy Audit Diagnostic Tests Tests Performed Should Have Been Should Have Been & Was % Performed % Performed Correctly Blower door testing 949197%84% Zonal pressure diagnostics 572442%87% Infrared camera 412049%70% Steady state efficiency 564479%98% Air flow test on forced air furnace 351440%93% Proper coolant charge 500%N/A

23. 23Managed by UT-Battelle for the Department of Energy Audit Combustion Safety Tests Tests Performed Should Have Been Should Have Been & Was % Performed % Performed Correctly Ambient CO level in CAZ 664061%98% Ambient CO level outside CAZ 502448%100% Gas/propane/fuel oil leaks 744662%83% Spillage testing on all atmospheric furnaces, boilers, water heaters 574070%95% Heating system CO level 635079%94% Water heating system CO level 534687%93%

24. 24Managed by UT-Battelle for the Department of Energy Audit Combustion Safety Tests Tests Performed Should Have Been Should Have Been & Was % Performed % Performed Correctly Draft test on heating system (worst case, natural conditions) 514078%95% Draft test on water heater (worst case, natural conditions) 453373%88% Measure CAZ for sufficient volume for combustion air 392154%85% Gas/propane ranges CO level 371643%75% Gas/propane stove CO level 372259%77%

25. 25Managed by UT-Battelle for the Department of Energy Audit Exit Interview Applicable Observations Action Taken #% Reviewed bills with client 10177% Provided context (high/low/average) for usage 991313% Summarized findings for client 1028381% Discussed health and safety issues 1005555% Discussed comfort issues 1003838% Discussed measure options 997980% Summarized behavior change opportunities 1001010% Discussed potential $ savings from behavior change 10077% Obtained client commitment for behavior change 10077% Recommended behavior changes w/ potential for saving 931011% Provided information on next steps 1019695%

26. 26Managed by UT-Battelle for the Department of Energy Audit Exit Interview Rating Design Quality of Exit Interview 1 Auditor explained next steps 2 Auditor reviewed discussed behavior changes 3 Auditor discussed potential $ saved from behavior change 4 Auditor explained client actions with the greatest impact 5 Auditor assessed client’s desired outcomes 6 Auditor obtained client commitments for energy saving actions 7 Client was asked if they have any concerns 8 Auditor explained the work proposed Needs Improvement Excellent Rating12345 Total Points needed0-34-5678 Bold Points needed00011

27. 27Managed by UT-Battelle for the Department of Energy Audit Exit Interview Rating Needs Improvement Excellent Rating12345 Number of Audits5635721 Percent of Audits55%35%7%2%1% Mean Rating1.6

28. 28Managed by UT-Battelle for the Department of Energy Audit Ratings Number Rated Needs Improvement Excellent Mean Rating 12345 Percent with Each Rating Auditor attempt to engage client10045% 30%6%14%5%2.0 Client engagement10040% 22%29%6%3%2.1 Auditor communication skills10015%43%32%2%8%2.5 Quantity of education10061%26%8%5%0%1.6 Quality of education10048%40% 6% 5%1%1.7 Completeness of inspection1038%32% 28% 17%15%3.0 Completeness of testing10439%37% 20% 4%0%1.9 Quality of testing10435%25% 10% 25%6%2.4

29. 29Managed by UT-Battelle for the Department of Energy Audit Ratings Number Rated Needs Improvement Excellent Mean Rating 12345 Percent with Each Rating Audit efficiency1040% 10%26%34%31%3.9 Auditor professionalism1031% 10%19%41%29%3.9 Auditor respect of client’s home1044% 11%2% 38%46%4.1 Auditor safety practices10315% 21%24% 14%26%3.2

30. 30Managed by UT-Battelle for the Department of Energy Communication of Audit Findings  Audit write-ups do not convey educational opportunities  Work order not always clear or specific enough for installer to understand the desired outcome  Tools that improved information transfer – Pre-work walk through with installer – Attaching photos of critical areas to the work order – In process check-ups with the installers  Air sealing is a particular area for improved communication  Need for flexibility to change work order when problems are found during installations

31. 31Managed by UT-Battelle for the Department of Energy Audit Findings Opportunities  Utilize energy bills  Increase client engagement  Identify client-specific opportunities for reducing energy use  Auditor understanding of pressure boundaries  Auditor understanding of testing purpose and procedures  Conduct worst case draft testing Strengths  Respect and concern for clients  Explanation of WAP program and process  Use of data collection forms  Auditors meet with contractors to explain audit findings and work scope

32. 32Managed by UT-Battelle for the Department of Energy PROCESS FIELD STUDY MEASURE INSTALLATION OBSERVATION FINDINGS

33. 33Managed by UT-Battelle for the Department of Energy Crews and Contractors Observed NumberPercent Crews 3531% Contractors 7465% Both 54% TOTAL 114100%

34. 34Managed by UT-Battelle for the Department of Energy Measure Installation Preparation Information Available NumberPercent Household demographics 5246% Audit report 5649% Work order 11197% Materials list 7364% Out of 114 observations.

35. 35Managed by UT-Battelle for the Department of Energy Measure Installation Introduction Applicable Number IntroLater Intro or Later #%#%#% Explained WAP1012626%33%2929% Explained purpose of visit1017776%11%7877% Explained planned measures1017776%33%8079% Asked if client had questions1015251%55%5756% Applicable Number NumberPercent Visit expected 11210997% Adult occupant present 11310593%

36. 36Managed by UT-Battelle for the Department of Energy Measure Installation Window and Door Ratings Number Rated Needs Improvement Excellent Mean Rating Rating – Quality of work12345 Percent with Each Rating Windows sealed128% 0%33%50%8%3.5 Interior storm window work40% 100%0%4.0 Sash kit work10% 100%0% 3.0 Window repair120% 8%42% 8%3.5 Window replacement90% 44%56%0%3.6 Door weather stripping3811% 34%32%21%5%2.8 Doors treated138% 15% 46%15%3.5 Door replacement160% 38% 25%0%2.9

37. 37Managed by UT-Battelle for the Department of Energy Measure Installation Air Sealing Ratings Number Rated Needs Improvement Excellent Mean Rating Rating – Quality of work12345 Percent with Each Rating Attic floor penetrations5810% 41%34%9%5%2.6 Knee walls195% 58%37%0% 2.3 Second floor rim joist30% 33%67%0% 2.7 Basement penetrations to outside248% 46% 0% 2.4 Basement to conditioned space1217% 50%33%0% 2.2 Crawl space to conditioned space140% 64%29%7%0%2.4 Crawl space to outside90% 56%44%0% 2.4 Garage to conditioned space80% 63%38%0% 2.4

38. 38Managed by UT-Battelle for the Department of Energy Measure Installation Attic Insulation Applicable Number Action Taken #% Attic floor sealing complete before 524077% Baffles used 321856% Constant insulation depth achieved 584985% No insulation gaps or voids 585188% Appropriate material used 585595%

39. 39Managed by UT-Battelle for the Department of Energy Measure Installation Attic Insulation Number Rated Needs Improvement Excellent Mean Rating 12345 Percent with Each Rating Attic insulation quality574% 18%42%19%18%3.3 Attic insulation cleanliness582% 3%31%17%47%4.0

40. 40Managed by UT-Battelle for the Department of Energy Measure Installation Attic Access Work Number Rated Needs Improvement Excellent Mean Rating 12345 Percent with Each Rating Attic hatch work quality486% 23%56%13%2%2.8 Walk-up attic work quality367% 0%33%0% 1.7 Attic access work cleanliness472% 9%40%47%4.3

41. 41Managed by UT-Battelle for the Department of Energy Measure Installation Insulation Ratings Number Rated Needs Improvement Excellent Mean Rating 12345 Percent with Each Rating Wall insulation quality323% 13%75%6%3%2.9 Wall insulation cleanliness320% 16%3%19%63%4.3 Siding preservation/interior wall3110% 19%26%29%16%3.2 Basement insulation quality220% 50%36%14%3.6 Basement insulation cleanliness220% 14%18%68%4.5

42. 42Managed by UT-Battelle for the Department of Energy Measure Installation Crawl Space Ratings Number Rated Needs Improvement Excellent Mean Rating Rating – Quality of work12345 Percent with Each Rating Vapor barrier installation210% 29%14%48%10%3.4 Perimeter wall insulation50% 20%60%20%4.0 Vents sealing20% 50% 0%3.5 Crawl space ceiling insulation and air barrier installation 166% 31%13%44%6%3.1

43. 43Managed by UT-Battelle for the Department of Energy Measure Installation Ventilation Ratings Number Rated Needs Improvement Excellent Mean Rating Rating – Quality of work12345 Percent with Each Rating Existing fans ducted to outside150% 7%27%67%4.6 Whole house ventilation30% 33%0%67%4.3 Attic ventilation120% 25%75%4.8 Kitchen ventilation80% 13%0%88%4.8 Bathroom ventilation157% 20%0% 73%4.1

44. 44Managed by UT-Battelle for the Department of Energy Measure Installation Heating System Ratings Number Rated Needs Improvement Excellent Mean Rating Rating – Quality of work12345 Percent with Each Rating Heating system tune-up617% 50%0%33%0%2.5 Heating system repair40% 25% 3.5 Heating system replacement714% 57%14% 0%2.3 80% efficient system installation10% 100%0% 2.0 90%+ efficient system installation520% 60%0%20%0%2.2 Heating system vent correction560% 0%20% 0%2.0

45. 45Managed by UT-Battelle for the Department of Energy Measure Installation Air Conditioning Ratings Number Rated Needs Improvement Excellent Mean Rating Rating – Quality of work12345 Percent with Each Rating Sealing around air conditioners20% 50% 0%3.5 Air conditioner tune-up10% 100%0% 3.0 Air conditioner replacement20% 100%0%4.0 Evaporative cooler repair40% 25%0%50%25%3.8 Evaporative cooler replacement20% 50% 0% 2.5

46. 46Managed by UT-Battelle for the Department of Energy Measure Installation Water Heater Ratings Number Rated Needs Improvement Excellent Mean Rating Rating – Quality of work12345 Percent with Each Rating Hot water temperature adjustment20% 50%0%50%4.0 Hot water heater wrapped140% 29%36% 0%3.1 Hot water heater pipes insulated368% 3%25%19%44%3.9 Hot water heater repaired70% 71%14% 3.4 Hot water heater venting work70% 14%0%71%14%3.9

47. 47Managed by UT-Battelle for the Department of Energy Measure Installation Summary Ratings Number Rated Needs Improvement Excellent Mean Rating 12345 Percent with Each Rating Efficiency1130% 11%14%31%44%4.1 Professionalism1132% 13%21%24%40%3.9 Cleanliness1137% 5%19%28%40%3.9 Safety practices11114% 41%33%5%6%2.5 Communication1058% 38%7%22%26%3.2 Attempt to engage client10436% 37%9%17%2%2.1 Client engagement10328% 37%24%7%4%2.2 Education quantity9955% 22%10%7%6%1.9 Education quality9870% 25%2% 1%1.4

48. 48Managed by UT-Battelle for the Department of Energy Measure Installation Findings Opportunities  Increased use of blower door when air sealing  Respect for clients’ homes (booties, covering furniture)  Crew member safety  Increased assessment of HVAC contractors  Explain CFLs when installing  Client education Strengths  Crew flexibility and dedication (above and beyond work scope)  Crew problem solving  Some very high quality work  Experienced HVAC contractors

49. 49Managed by UT-Battelle for the Department of Energy PROCESS FIELD STUDY FINAL INSPECTION OBSERVATION FINDINGS

50. 50Managed by UT-Battelle for the Department of Energy Final Inspection Occupant Interaction Applicable Number Work Completed #% Visit expected 918897% Adult occupant present 918796% Discussed work performed 896775% Discussed energy/cost savings 892528% Discussed health & safety issues 892629% Discussed job satisfaction 906370%

51. 51Managed by UT-Battelle for the Department of Energy Final Inspection Home Walkthrough Applicable Number Walk- through Later Walk- through or Later #%#%#% Examined all accessible rooms 91 8088%22%8290% Noted CO detector presence/absence 91 5965%00%5965% Noted smoke detector presence/absence 91 5864%00%5864% Examined all accessible attic(s) 67 4770%23%4973% Examined basement 47 4596%00%4596% Examined all accessible crawl spaces 47 3064%36%3370% Examined garage 20 1575%00%1575% Conducted outside home walk-around 91 6571%22%6774%

52. 52Managed by UT-Battelle for the Department of Energy Final Inspection Diagnostic Tests Tests Performed Should Have Been Should Have Been & Was % Performed % Performed Correctly Blower door 867385% Zonal pressure 451533%87% Duct leakage test 452862%79% Infrared Scan 431944%95% Combustion efficiency test 472860%89% Air handler air flow test 321753%94% Proper coolant charge 300%N/A

53. 53Managed by UT-Battelle for the Department of Energy Final Inspection CO Combustion Safety Tests Tests Performed Should Have Been Should Have Been & Was % Performed % Performed Correctly Heating system 543972%97% Water heater 503876%95% Gas stove 26831%100% Space Heater 700%N/A Ambient 451840%100%

54. 54Managed by UT-Battelle for the Department of Energy Final Inspection Draft Tests Tests Performed Should Have Been Should Have Been & Was % Performed % Performed Correctly WCD heating system 402460%92% WCD water heater 453067%83% WCD Other 700%N/A Spillage Testing 412766%96%

55. 55Managed by UT-Battelle for the Department of Energy Final Inspection Summary Ratings Number Rated Needs Improvement Excellent Mean Rating 12345 Percent with Each Rating Inspector attempt to engage client8965% 11% 4%8%1.8 Client engagement8844% 39%10%6%1%1.8 Inspector communication skills8922% 40%20%0%17%2.5 Education quantity8665% 22%7%5%1%1.5 Education quality8663% 20%8%6%3%1.7 Inspection completeness8823% 39%24%11%3%2.3 Testing completeness8938% 37%21%0%3%1.9 Testing quality8816% 17%19%13%35%3.3

56. 56Managed by UT-Battelle for the Department of Energy Final Inspection Summary Ratings Number Rated Needs Improvement Excellent Mean Rating 12345 Percent with Each Rating Inspector efficiency 890%16%0%27%57%4.3 Inspector professionalism900% 16%30%29%26%3.6 Inspector cleanliness907% 2%7%0%84%4.5 Inspector safety practices892% 3%25%22%47%4.1

57. 57Managed by UT-Battelle for the Department of Energy Final Inspection Findings Opportunities  Increased client education – Explain measures installed – Reinforce client action plan  Improved testing quality  Increased assessment of installation quality  Reduced use of auditor to conduct final inspection – additional perspective Strengths  Good job when following work order closely  Time saved when contractor attends  Information to clients – who to call if they have problems  Referrals to additional assistance programs

58. 58Managed by UT-Battelle for the Department of Energy PROCESS FIELD STUDY TRAINING FINDINGS NEEDS, CHALLENGES, STRATEGIES

59. 59Managed by UT-Battelle for the Department of Energy Training Needs  Building science fundamentals  Critical thinking – assessing unique situations  Combustion safety testing  Ventilation assessment  Safe work practices – Worker safety – Lead safe work  Client education

60. 60Managed by UT-Battelle for the Department of Energy Training Challenges  More clarity needed on specifications and procedures  Sufficient time is not allocated for training  Pressure for job production  Agency management does not always prioritize training

61. 61Managed by UT-Battelle for the Department of Energy Training Strategies  Increased training resources  Combination of classroom and field training  Greater attendance at national conferences  Emphasis on why tests are conducted  Emphasis on client education  Reinforce with monitoring, assessment, and feedback