The MIS and Performance Measurement Systems of the Industry Energy R&D Program Dennis Rank Dennis Rank and Associates Presentation to the RTD Interest Group American Evaluation Association Conference Portland, Oregon November 2, 2006

Dennis Rank and Associates2 The IERD A program of Natural Resources Canada (NRCan). A program of Natural Resources Canada (NRCan). Funding to industry for R&D to improve energy efficiency & reduce greenhouse gas (GHG) emissions. Funding to industry for R&D to improve energy efficiency & reduce greenhouse gas (GHG) emissions. Projects cost-shared w/ clients; IERD supplies 35%, on average. Projects cost-shared w/ clients; IERD supplies 35%, on average. Conditionally repayable, depending on risk, commercial status, size of impact, etc. Conditionally repayable, depending on risk, commercial status, size of impact, etc. Currently ~70 projects; two-thirds repayable. Currently ~70 projects; two-thirds repayable.

Dennis Rank and Associates3 Relevance Desirable to track “downstream” socio-economic (S- E) impacts, but few programs succeed. Desirable to track “downstream” socio-economic (S- E) impacts, but few programs succeed. Even “bottom-up” project data are often incomplete or inaccurate, & databases often in very poor shape. IERD uses integrated management information system (MIS) and performance measurement (PM) system. Even “bottom-up” project data are often incomplete or inaccurate, & databases often in very poor shape. IERD uses integrated management information system (MIS) and performance measurement (PM) system. Highly-used and highly-useful, both for management & for tracking S-E impacts. Highly-used and highly-useful, both for management & for tracking S-E impacts. What lessons can be learned from IERD? What lessons can be learned from IERD?

Dennis Rank and Associates4 System Overview M/soft Access, linked to Outlook & Excel. M/soft Access, linked to Outlook & Excel. Very complete suite of administrative & management tools. Very complete suite of administrative & management tools. Goals: (1) meet or exceed reporting requirements; (2) be user-friendly; (3) minimize resources required. Goals: (1) meet or exceed reporting requirements; (2) be user-friendly; (3) minimize resources required. Designed & maintained by full-time database manager. Designed & maintained by full-time database manager.

Dennis Rank and Associates5 Project Data Modules Tracking: basic data on contract, $, contact info., milestones, etc. Tracking: basic data on contract, $, contact info., milestones, etc. Finance & Admin: claim & payment history Finance & Admin: claim & payment history Management: project reports, action items, estimated & actual energy & GHG impacts Management: project reports, action items, estimated & actual energy & GHG impacts Project manager: routine & custom reports Project manager: routine & custom reports

Dennis Rank and Associates6 Operational Aspects Many functions automated or semi-automated; e.g., labels, reminders to clients, reminders to project officers, tracking action items, “red flags”, “policeman’s reports”, etc. Many functions automated or semi-automated; e.g., labels, reminders to clients, reminders to project officers, tracking action items, “red flags”, “policeman’s reports”, etc. Project officers can set up & edit project info. Project officers can set up & edit project info. Database manager responsible for data accuracy & completeness, training, upgrading, routine & custom reports, etc. Database manager responsible for data accuracy & completeness, training, upgrading, routine & custom reports, etc. Program manager ensures system up to date, complete, fully utilized. Program manager ensures system up to date, complete, fully utilized.

Dennis Rank and Associates7 Impacts Tracked Direct energy savings. Direct energy savings. Direct GHG (NOx, CO 2 ) reductions. Direct GHG (NOx, CO 2 ) reductions. Some indirect, knock-on, spin-off benefits (e.g., other uses of the technology not originally intended). Some indirect, knock-on, spin-off benefits (e.g., other uses of the technology not originally intended). Sales revenue and/or cost savings. Sales revenue and/or cost savings. FTEs on project, and/or created long-term. FTEs on project, and/or created long-term. Investment to commercialize. Investment to commercialize.

Dennis Rank and Associates8 Impact Tracking Process For each project, annually & since inception. For each project, annually & since inception. Also at program roll-up level. Also at program roll-up level. Heavy involvement of project officers (e.g., site visits) to obtain & verify information. Heavy involvement of project officers (e.g., site visits) to obtain & verify information.

Dennis Rank and Associates9 Unusual Features Only really useful PM system the author has encountered in 20+ years’ S&T experience. Only really useful PM system the author has encountered in 20+ years’ S&T experience. Data complete, accurate, & up to date – estimates upgraded each year (e.g., w/ actuals on take-up, energy efficiency). Data complete, accurate, & up to date – estimates upgraded each year (e.g., w/ actuals on take-up, energy efficiency). Very high buy-in by project officers. Very high buy-in by project officers. Tracks impacts up to 10 years after project complete (for repayables). Tracks impacts up to 10 years after project complete (for repayables). System in very good running order, reports (even custom ones) easy to produce. System in very good running order, reports (even custom ones) easy to produce.

Dennis Rank and Associates10 Economic Analysis Usefulness Benefits: energy (by type) & GHG data. Benefits: energy (by type) & GHG data. Net Benefits: costs of R&D, operating, job creation, etc. Net Benefits: costs of R&D, operating, job creation, etc. Net Present Value: timing of benefits & costs, program costs. Net Present Value: timing of benefits & costs, program costs. Estimation assumptions captured. Estimation assumptions captured. Easy to identify “big winners” (or losers). Easy to identify “big winners” (or losers). Contact info. makes it easy to obtain more detailed information from clients. Contact info. makes it easy to obtain more detailed information from clients.

Dennis Rank and Associates11 Why Does it Work? System design: System design: On-site expert database designer & manager; On-site expert database designer & manager; First run as prototype, gradually enhanced & upgraded w/ feedback from project officers; First run as prototype, gradually enhanced & upgraded w/ feedback from project officers; Development costs low, as d/base manager has other admin duties; Development costs low, as d/base manager has other admin duties; As opposed to using outside IT help for “one- off” design (but then designer leaves) As opposed to using outside IT help for “one- off” design (but then designer leaves)

Dennis Rank and Associates12 Why Does it Work? Cont’d System operation: System operation: Strong leadership by program manager ensures system is used. Strong leadership by program manager ensures system is used. Oversight, multiple checks, verification ensures data are complete & accurate. Oversight, multiple checks, verification ensures data are complete & accurate. System works well, bug-free. System works well, bug-free. Only useful functions, saving project officers time (~90% savings compared to “by hand”). Only useful functions, saving project officers time (~90% savings compared to “by hand”). Clients for repayables obliged to provide data, & real sanctions if they don’t. Clients for repayables obliged to provide data, & real sanctions if they don’t.

Dennis Rank and Associates13 Transportable? Yes: Yes: Applied R&D programs; Applied R&D programs; R&D done jointly w/ partners (long-term links, “carrots & sticks”); R&D done jointly w/ partners (long-term links, “carrots & sticks”); Non-R&D programs w/ long-term links to clients; Non-R&D programs w/ long-term links to clients; BUT, need on-site database designer & manager. BUT, need on-site database designer & manager.

Dennis Rank and Associates14 Transportable? Cont’d Less so: Less so: Strategic programs (too long a time lag to impacts); Strategic programs (too long a time lag to impacts); Non-repayable grants programs (fewer “carrots or sticks”); Non-repayable grants programs (fewer “carrots or sticks”); Decentralized programs (system training & access more difficult); Decentralized programs (system training & access more difficult); “Sunset” programs (not worth the effort, & no one around to maintain). “Sunset” programs (not worth the effort, & no one around to maintain).

Dennis Rank and Associates15 Conclusions Integrated MIS/PM system is complete, effective, & efficient. Integrated MIS/PM system is complete, effective, & efficient. Highly-used because actually helpful to project officers, expert d/base manager on-site, & strong leadership. Highly-used because actually helpful to project officers, expert d/base manager on-site, & strong leadership. Highly-useful for tracking impacts because data complete, accurate, up-to-date, & easy to verify. Highly-useful for tracking impacts because data complete, accurate, up-to-date, & easy to verify. Transportable to many S&T programs of interest, especially long-running ones w/ applied focus. Transportable to many S&T programs of interest, especially long-running ones w/ applied focus.

Dennis Rank and Associates16 For More Information Dennis Rank Dennis Rank