1. Interim Findings and Recommendation from the Whole House Process Evaluation – SCE Contractor Training Assessment May 1, 2012 Presented to:Staff from SCE and PG&E Prepared by: ASW Engineering SBW Consulting, Inc.

2. Part Two of Today’s Presentations SCE Contractor Training Assessment  In-depth examination of SCE’s training designs, methods and materials.  Do they prepare contractor’s to succeed? Context  Rapid change in training underway, so assessment looked at a moving target.  Focused on SCE but there are wider implications.  Recommendations from the overall process evaluation need to be considered before training is changed. 2 May 1, 2012

3. Training Big Picture Contractors must attend training to qualify Performance gaps – a training problem? 3 May 1, 2012

4. Assessment Objectives Effectiveness and appropriateness Do the training objectives address EUC tasks? 4 May 1, 2012

5. Assessment Methods Reviewed program materials (program design, training and reporting) Interviewed key personnel (RHA, ICF) Mapped training objectives to program requirements Assessed discrepancies 5 May 1, 2012

6. Assessment Criteria How well the training and materials:  Supported behavior change  Included motivation to action  Applied adult learning princip les 6 May 1, 2012

7. Adult Learning Principles and Practices Learner orientation, buy in, engagement Learner success engineering Practice, application, and interactivity Learning objectives 7 May 1, 2012

8. Effect of Teaching Styles and Average Retention Rates 8 May 1, 2012

9. Delivery Mode and Average Retention Rates 9 May 1, 2012 Practice and application are crucial to training effectiveness

10. Relating Learning Levels to Performance Outcomes 10 May 1, 2012 Bloom’s taxonomy

11. Required Training Participation Workshop Basic Path Training Advanced Path Training Online Learning Center Mentoring 11May 1, 2012 11

12. Participation Workshop 2-hour session required for at least one person from each contractor – webinar or classroom Orients contractors to goals, scope of work, marketing Describes contractor qualifications, benefits 12 May 1, 2012

13. Participation Workshop Findings Good pitch of program Lacks details on:  Job processing expectations (Job Reporting, EnergyPro, documentation)  Level of rigor for QA/QC 13 May 1, 2012

14. Participation Workshop Recommendations More content on:  EnergyPro requirements  Exposure to job reporting documentation and process Job reporting may change based on other recommendations Refer to:  “Adopt Common Statewide Job Reporting” (Whole House Process Evaluation – PG&E_SCE Interim Report Workbook, Caption Tab, ID=13) 14 May 1, 2012

15. Basic Path Training Mandatory for Basic Path program Exception if BPI Building Analyst on staff 15 May 1, 2012

16. Basic Path Training Findings California Building Performance Contractor’s Association (CBPCA) 3-day course a good match-up between training content and program measures Lacks:  Formal objectives  Adult learning principles 16 May 1, 2012

17. Basic Path Recommendations Incorporate:  Worksheets  More hands-on practice  Ways to demonstrate proficiency (e.g., asbestos identification and insulation measurement)  Formal learning objectives  Adult learning principles 17 May 1, 2012

18. Advanced Path Training EUC requires BPI Building Analyst certification Certification can be obtained with no formal training Certification does not guarantee EUC skill set Dozens of third-party trainings offered 18 May 1, 2012

19. Third Party BPI Training Quality and depth of training ranges widely 0 to 12 days Classroom, online, self-trained Insufficient practice for EUC fieldwork (performance gaps) Loose alignment with some measures (air sealing, weatherization) 19 May 1, 2012

20. Advanced Path Training Findings CBPCA 12-day course provides:  Solid foundation for Advanced Path work  Strong combustion safety curriculum  Some hands-on practice with diagnostic tools Lacks:  Formal training objectives  Adult learning principles  Descriptive text for photos 20 May 1, 2012

21. Advanced Path Recommendations For CBPCA 12-day course:  Add training objectives  Apply adult learning principles  Add descriptive text to photos Supplement BPI training with:  More hands-on practice with field testing procedures, protocols  Practice of program-specific desk work (forms, energy modeling)  Remedial training for specific performance gaps (asbestos, air sealing, etc.) 21 May 1, 2012

22. Concerning HVAC Quality Installation (QI Program) and EUC For minor HVAC repair work, BPI training is adequate For major HVAC repair or replacement, then a permit is required and contractor must meet Title 24 quality standards HVAC QI training addresses Title 24 code 22 May 1, 2012

23. Online Learning Center – A New Resource On-demand training on:  JRT Administration  Energy Modeling  Job Processing  Combustion Safety Testing Mandatory participation; must pass exam 23 May 1, 2012

24. Online Learning Center Findings Well designed training Addresses some performance gap issues Incorporates adult learning features Useful for BPI certified contractors who did not attend CBPCA 12-day course 24 May 1, 2012

25. Online Learning Center Recommendations Record exam scores for multiple employees Expand topics:  BPI and program standards  Field inspection / testing  Asbestos identification  Insulation and weatherization 25 May 1, 2012

26. Mandatory Mentoring Required of one person from each contractor for both program levels Three topics:  Field Work  Job Reporting/Job Processing  Energy Pro 26May 1, 2012

27. Mentoring Findings Not formal training per se Mentoring includes:  Observation of contractor performance  Evaluation and scoring  Feedback 27 May 1, 2012

28. Mentoring Recommendations Standardize mentoring and evaluation protocols QC firm should provide input Provide remedial training adapted to individual needs, based on QA/QC findings Remote mentoring for Energy Pro and Job Reporting 28 May 1, 2012

29. Top 10 Performance Gaps Performance Gaps  Inaccurate insulation R-value  Inability to identify asbestos  Failure to achieve minimum air-sealing  Inaccurate wall, window, and floor measurements  Improper combustion safety testing (CO readings)  Failure to identify combustion appliance hazardous conditions (e.g., gas leaks)  Failure to complete Basic Path measures for Advanced projects 29May 1, 2012

30. Top 10 Performance Gaps Performance Gaps  Inaccurate insulation R-value  Inability to identify asbestos  Failure to achieve minimum air-sealing  Inaccurate wall, window, and floor measurements  Improper combustion safety testing (CO readings)  Failure to identify combustion appliance hazardous conditions (e.g., gas leaks)  Failure to complete Basic Path measures for Advanced projects 30 May 1, 2012

31. Top 10 Performance Gaps (continued) Performance Gaps (continued)  Information does not agree across documents (JRT, Energy Pro, forms)  Inaccurate duct leakage tests  Inaccurate assessment of insulation coverage and quality Refer to:  “Performance Gaps Observed in SCE Whole House Training” (Whole House Process Evaluation Interim Report Workbook, Caption Tab, ID=74) 31 May 1, 2012

32. Proposed Inspector-Trainer Role Establish an internal inspector-trainer within each contractor’s staff—will be responsible for:  Attending all required training  Providing in-house training  Ensuring quality of work Suggestions:  Collaborative effort between QA/QC (RHA) and mentors (ICF) for training inspector-trainers  Develop and provide train-the-trainer materials for inspector- trainers 32 May 1, 2012

33. Proposed Marketing and Sales Training Finding  Little marketing and sales content in current training materials Recommendation  Suggest formal marketing and sales training focused on contractor sales force  Classroom opportunity to practice EUC program sales skills  Opportunity to apply adult learning principles 33 May 1, 2012

34. From Interim Report - Ongoing Safety Training Findings  Safety issues are noted frequently and may be missed by contractors 34 May 1, 2012

35. From Interim Report - Ongoing Safety Training (continued) Recommendations  Provide ongoing safety training to contractors. Based on feedback from inspectors and QC  Issues to emphasize, include: Low house air leakage Possible asbestos in ducts CAZ testing Correct installation of insulation  Refer to: “Ongoing safety training” (Whole House Process Evaluation – PG&E_SCE Interim Report Workbook, Recommendations Tab, ID=17) Relevant To  PG&E and SCE, BIG and ICFI / RHA 35 May 1, 2012