E-22023 DSM Programs Evaluation Reports
177 passages
DEFINITIONS Accuracy Reflects the proximity of measurements to the true value. Net savings Energy or peak demand savings that can be reliably attributed to a program. This includes effects such as free-ridership and spillover that negative...
AI summary The document defines key terms used in energy efficiency and program evaluation, including accuracy, net savings, net-to-gross ratio, non-sampling error, and peak coincidence factor, providing precise explanations for each.
EfficiencyOne (EOne), an independent, non-profit organization, is responsible for helping Nova Scotians improve the energy efficiency of their homes and workplaces by designing, marketing, and delivering energy efficiency and conservation...
AI summary EfficiencyOne (EOne) manages a DSM program portfolio funded by Nova Scotia Power (NS Power) ratepayers. Econoler, working with multiple partners, evaluated EOne's 2023 DSM programs, confirming significant energy savings and GHG emissions reductions. The evaluation included data collection, market assessments, and participant surveys.
1 Evaluation Scope and Objectives The 2023 Portfolio Evaluation Plan was based on the Evaluation Schedule outlined in the Overall Strategic Evaluation Plan 4 that provides the framework and approach to guide evaluation planning and impleme...
AI summary The 2023 Portfolio Evaluation Plan outlines the approach for evaluating demand-side management (DSM) activities from 2023 to 2025, prioritizing evaluation efforts based on factors such as program savings, uncertainty, changes in program design, regulatory requirements, and evaluation scheduling. It also discusses the three evaluation categories: impact, process, and market evaluations.
Table 1: 2023 Portfolio Evaluation Plan PY2023 Program Components Impact Process Market Residential DSM Program Components Appliance Retirement Condensed Instant Savings Comprehensive Home Energy Assessment Comprehensive Green Heat Condens...
AI summary Table 1 outlines the 2023 Portfolio Evaluation Plan, detailing various program components under residential and BNI DSM programs, including their impact, process, and market evaluations. The plan includes condensed and comprehensive evaluations for initiatives like Appliance Retirement, Home Energy Assessment, and Strategic Energy Management.
1.1 Impact Evaluation Objectives and Scope The impact evaluation activities were aimed at determining: - › Gross electrical energy and peak demand savings at the meter and at the generator - › Net-to-gross ratios (NTGRs), including free-ri...
AI summary The impact evaluation activities aimed to assess electrical energy and peak demand savings, net-to-gross ratios, effective useful life of program measures, and greenhouse gas emissions reductions. Two types of evaluations were conducted: comprehensive and condensed. Comprehensive evaluations were performed every three years for most program components, except for those with stable parameters or small savings contributions.
1.2 Process and Market Evaluation Objectives and Scope A market evaluation was completed in 2023 for mini-split heat pumps (MSHPs) under Green Heat. Market evaluation activities were aimed at achieving the following objective: › Analyze th...
AI summary A market evaluation for mini-split heat pumps under Green Heat was completed in 2023. Four process evaluations were conducted for Green Heat, Custom Retrofit, Strategic Energy Management, and Small Business Energy Solutions to analyze market evolution and program logic models.
2.1 Impact Evaluation The impact evaluations were conducted through a range of activities such as tracking sheet audits, datacollection tool development, energy model reviews assisted by participant follow-up interviews, on-site visits, as...
AI summary The impact evaluations were conducted using various methods including tracking sheet audits, data collection tool development, energy model reviews, participant interviews, on-site visits, and other data collection and analysis activities.
2.1.1 Tracking Sheet Audits The final tracking sheets submitted to the Evaluator by EOne contained both data for all completed projects for 2023 and the tracked results required to calculate final savings. The final tracking sheets were au...
AI summary The final tracking sheets submitted by EOne were audited to ensure data consistency and completeness, leading to corrected tracked values and evaluated savings for each program component.
2.1.2 Data-collection Tool Development and Sampling Strategy Data-collection tool development and sampling were carried out for Instant Savings, Home Energy Assessment, Green Heat, Small Business Energy Solutions, Custom, and Strategic Ene...
AI summary The document outlines the development of data-collection tools and sampling strategies for various energy efficiency programs, including Instant Savings, Home Energy Assessment, Green Heat, and others. The Evaluator created instruments like surveys and interview guides, informed by documentation reviews and staff interviews.
Surveys and Interviews This subsection describes the data-collection activities conducted for impact evaluations. It should be noted that surveys and interviews were often integrated to collect impact, process, and market information as di...
AI summary This subsection outlines the data collection methods used for impact evaluations, focusing on participant surveys conducted between August and December 2023. Surveys were used to gather information on free-ridership and spillover effects, with 392 participants surveyed, primarily via telephone, except for the Instant Savings program which used an online survey.
Table 2: 2023 Participant Surveys Program Component Participants Residential Instant Savings 122 Home Energy Assessment 100 Green Heat 124 BNI Small Business Energy Solutions 46 Total 392 › In-depth interviews with program staff, participa...
AI summary Table 2 outlines the number of participants in various programs in 2023, including residential and business initiatives. In-depth interviews were conducted between August 2023 and January 2024 to evaluate program impact, free-ridership, and spillover effects.
Project Reviews with Follow-up Site Visits or Interviews The Evaluator performed a total of 74 project reviews during the summer and fall of 2023, 51 of which were complemented through site visits and 23 were complemented through phone int...
AI summary The Evaluator conducted 74 project reviews in 2023, including site visits and interviews, to assess programs like Small Business Energy Solutions and Strategic Energy Management. Reviews aimed to validate savings calculations and gather data on free-ridership and spillover effects.
2.1.5 Gross Savings Analysis Gross savings refer to changes in energy consumption resulting from actions taken by participants regardless of their reasons for participating in a program. Upon completion of the impact evaluation activities...
AI summary This section discusses the calculation of gross savings in energy consumption from program participation, including interactive effects and peak demand savings. Interactive effects are calculated based on site-specific and provincial data, while peak demand savings are derived using established ratios and engineering calculations for residential and commercial programs.
ly, the level of participant spillover was established by dividing the total additional savings attributable to the component by the total savings achieved by the component for all survey respondents. When the activities of a program compo...
AI summary The document discusses the calculation of participant spillover for various energy efficiency programs, including how market effects are measured for both participants and non-participants. Spillover levels were updated for some programs in the 2023 evaluation, while others used data from past evaluations.
2.2 Process and Market Evaluation The process and market evaluations were conducted using a range of activities such as program component documentation as well as secondary data reviews, logic model reviews, and in-depth interviews with pr...
AI summary The process and market evaluations for Green Heat and related programs involved activities like program documentation, secondary data reviews, logic model reviews, and in-depth interviews with stakeholders and participants. The Evaluator conducted these tasks to assess the effectiveness of the programs.
Documentation Review The Evaluator reviewed all relevant evaluation and program component-specific documentation such as program manuals, marketing materials, application forms, tracking sheets, and any other information on changes made to...
AI summary The Evaluator reviewed program documentation and conducted staff interviews to assess changes and improvements to program components since the last evaluation.
Data-collection Tool Development and Sampling Strategy As described in Subsection [2.1.2](#page-14-3) above, the Evaluator used an integrated approach to developing datacollection tools that serve all evaluation types where possible. For i...
AI summary The Evaluator developed integrated data-collection tools to serve all evaluation types, such as process, market, and impact evaluations, reducing respondent burden and ensuring integrated evaluation results.
Analysis The results of the process and market evaluation activities were analyzed in relation to the research objectives identified in Subsection [1.2](#page-13-0) above. Results from all evaluation activities were consolidated and triang...
AI summary The analysis consolidated and triangulated results from process and market evaluation activities to ensure findings are based on a preponderance of results, aligning with the research objectives outlined in Subsection 1.2.
Home Energy Assessment - › HEA saw an increase in participation of 47% and an increase in average gross savings per home of 57% compared to 2022, likely driven by the integration of the Canada Greener Homes Grant (CGH Grant). - › Free-ride...
AI summary The Home Energy Assessment (HEA) program saw increased participation and savings, likely due to the Canada Greener Homes Grant. Free-ridership levels for energy efficiency upgrades decreased from 26% in 2020 to 17% in 2023, while solar PV free-ridership remained at 26%. Savings were 8% and 13% higher than those tracked by EOne. Billing analysis results were inconclusive, so 2018 overestimation ratios were retained.
Custom The overall key findings of the Custom impact evaluation were as follows: - › Compared to 2022, Custom participation decreased in 2023 due in part to the lack of participation in the Pay-for-Performance (P4P) service and the removal...
AI summary The Custom impact evaluation found a decrease in participation in 2023 compared to 2022, partly due to the lack of Pay-for-Performance service and the removal of the Retrofit Operational Demand Reduction offer. Free-ridership levels increased for Retrofit and solar PV projects, while New Construction free-ridership remained high. Energy and peak demand savings were adjusted by the Evaluator, resulting in lower savings than those tracked by EOne.
Small Business Energy Solutions - › The number of completed projects in Small Business Energy Solutions decreased by 27% in 2023, while average gross energy savings per project remained flat compared to 2022. - › Following the desk reviews...
AI summary The number of completed Small Business Energy Solutions projects dropped by 27% in 2023, with average energy savings remaining flat. Adjustments were made to energy and peak demand savings for lighting and MSHPs based on evaluations. Free-ridership increased by 5% in 2023, primarily due to lighting projects. Evaluated energy savings slightly differ from tracked savings.
Table 7: 2023 Free-ridership, Spillover, and NTGRs Program Component and Measure Type Free-ridership Levels Spillover Levels NTGR Residential Appliance Retirement Refrigerators 43% 0% 0.57 Freezers 45% 0.55 Air Conditioners 47% 0.53 Small...
AI summary Table 7 presents data on free-ridership, spillover, and net-to-gross ratios (NTGRs) for various energy efficiency programs in 2023, highlighting levels of free-ridership and spillover across different measures and participant categories, with NTGRs indicating the proportion of savings attributable to the program.
4.1 Participant and Partner Satisfaction The 2023 evaluation revealed high participant satisfaction with EOne and its programs. Home Energy Assessment, Green Heat, Custom (regular Retrofit and Solar PV), SEM, and Small Business Energy Solu...
AI summary The 2023 evaluation showed high participant satisfaction with EOne and its programs, including Home Energy Assessment, Green Heat, and Small Business Energy Solutions. Program partners were generally satisfied, though Green Heat contractors expressed lower satisfaction with the program component.
Evaluation Approach The evaluation was aimed at calculating program component gross and net results, namely electrical firstyear and lifetime energy savings, peak demand savings, as well as avoided greenhouse gas (GHG) emissions. [Table](#...
AI summary The evaluation approach focuses on calculating program component gross and net results, including energy savings, peak demand savings, and avoided greenhouse gas emissions. Table 1 summarizes the types of evaluation and corresponding methodologies for each program component.
Instant Savings Findings and Recommendations This subsection presents the key findings and recommendations from the Instant Savings evaluation. 2023 Instant Savings Finding: Instant Savings achieved both of its energy savings and its peak...
AI summary The 2023 Instant Savings program achieved its energy and peak demand savings targets, with increased participation and satisfaction from retailers. However, free-ridership for LED lamps and fixtures increased, while spillover for LED fixtures decreased. Net energy savings were 14% lower than initially tracked due to updated spillover and free-ridership levels. A recommendation is made to review and phase out outdated LED Non-A type lamp models.
INTRODUCTION EfficiencyOne (EOne), an independent, non-profit organization, is responsible for helping Nova Scotians improve the energy efficiency of their homes and workplaces by designing, marketing, and delivering energy efficiency and...
AI summary EfficiencyOne (EOne), a non-profit organization, manages energy efficiency programs funded by Nova Scotia Power (NS Power) ratepayers. EOne's 2023 DSM program portfolio includes the Residential Efficient Product Rebates program, which has two components: Appliance Retirement and Instant Savings. The evaluation report discusses the impact evaluation methodology, including baseline definitions, savings calculation, parameter values, and net-to-gross ratios (NTGRs).
2 ARet Evaluation Approach The 2023 ARet evaluation involved a condensed impact evaluation. The main objectives were as follows: › Calculate gross and net results, namely electrical first-year and lifetime energy savings, peak demand savin...
AI summary The 2023 ARet evaluation focused on calculating gross and net results, including energy savings and avoided GHG emissions. Key research questions were identified to achieve these objectives, with Table 6 outlining the evaluation objectives, research questions, and methods.
Table 6: 2023 ARet Evaluation Approach Evaluation Objectives Research Questions Methodology Calculate gross results › Are the data in the tracking sheet complete, accurate, and consistent? › What are the evaluated first-year and lifetime g...
AI summary This section outlines the evaluation approach for the 2023 ARet program, focusing on calculating both gross and net results through tracking sheet audits, unitary savings reviews, and calculations using evaluation results. It also considers avoided greenhouse gas emissions.
4 ARet Key Findings and Recommendations The 2023 ARet evaluation included a condensed impact evaluation and its main objectives were as follows: › Calculate gross and net ARet results, namely electrical first-year and lifetime energy savin...
AI summary The 2023 ARet evaluation found that the program fell short of its energy savings and peak demand targets by 16% and 13%, respectively. Participation levels were the lowest since 2012, with refrigerator and freezer retirements remaining the main drivers of savings. Evaluated net savings were 9% higher than those tracked by EOne due to updated unitary savings from appliance retirements.
6 Instant Savings Evaluation Approach The 2023 Instant Savings evaluation included a comprehensive impact evaluation whereby NTGRs and market effects were updated respectively through a participant online survey and retailer interviews. Th...
AI summary The 2023 Instant Savings evaluation aimed to collect participant and partner perspectives and calculate gross and net savings, including energy and GHG emissions. Research questions, methods, and sample sizes were outlined in a table.
Table 18: 2023 Instant Savings Evaluation Approach Evaluation Objective Research Question Methodology Collect information on participant and partner perspectives › What is the awareness level about Instant Savings and how did participants...
AI summary This section outlines the 2023 Instant Savings Evaluation Approach, detailing how participant and partner perspectives are collected, how gross and net results are calculated, and the methods used, such as online surveys and tracking sheet audits.
Note on Margin of Error For evaluation activities that yield quantitative results based on a sample, the Evaluator aimed to achieve a maximum margin of error of 10% at a confidence level of 90%. This means that, if measurements were conduc...
AI summary The document discusses the margin of error in quantitative evaluations, aiming for a maximum 10% margin at a 90% confidence level. It explains that this margin reflects precision, not accuracy, and notes that it only accounts for random sampling errors, not non-sampling errors or biases.
8.1 Tracking Sheet Audit To ensure program results were compiled reliably, the Evaluator first performed a tracking sheet audit designed to verify the completeness and consistency of the data submitted by EOne as well as correcting the tra...
AI summary An audit of tracking sheets was conducted to ensure the reliability of program results, verifying the completeness and consistency of data submitted by EOne and correcting tracked savings as needed. The corrected results are detailed in Appendix VIII.
Table 31: 2023 Instant Savings Net-to-Gross Ratios Product Category Free-Ridership Level Margin of Error Spillover Net-to-Gross Ratio LED Lamps 59% 5.9% 9% 0.50 LED Fixtures 49% 6.0% 2% 0.53 All Other Products 0 0 0 1.00 8.3.4 Evaluated Ne...
AI summary Table 31 presents the 2023 Instant Savings Net-to-Gross Ratios for different product categories, including LED lamps, LED fixtures, and all other products. The table highlights free-ridership levels, margins of error, spillover effects, and net-to-gross ratios for these categories.
9 Instant Savings Key Findings and Recommendations As mentioned previously, the main objectives of the 2023 Instant Savings evaluation were as follows: - › Collect information on participant and partner perspectives. - › Calculate gross an...
AI summary The 2023 Instant Savings program exceeded its energy and peak demand savings targets, with 11.476 GWh and 1.459 MW achieved, respectively. Participation increased by 9%, with notable growth in controls and appliance sales. LED lighting remained the primary source of savings, but non-LED replacements are becoming less available. Free-ridership levels for LED lamps and fixtures also increased.
This appendix presents the main results of the tracking sheet audit performed by the Evaluator, which was aimed at: - › Verifying that all data fields required for the evaluation were included and filled out in the tracking sheet submitted...
AI summary This appendix outlines the results of a tracking sheet audit conducted by the Evaluator to ensure data completeness and accuracy in EOne's submitted tracking sheet. The audit verified consistency in parameters used for calculating program results and validated calculation steps.
APPENDIX VIII Instant Savings Tracking Sheet Audit This appendix presents the main results of the tracking sheet audit performed by the Evaluator, which was aimed at: - › Verifying that all data fields required for the evaluation were incl...
AI summary This appendix details the results of an audit of the Instant Savings Tracking Sheet conducted by the Evaluator. The audit aimed to verify the completeness and accuracy of the data submitted by EOne, ensuring consistency in calculation methods and parameters used for program results.
Table 1: Summary of 2023 Existing Residential Program Evaluation Program Evaluation Type Methodology Component Impact Process Market HEA Comprehensive › Participant survey › Tracking sheet audit › Billing analysis › Solar photovoltaic (PV)...
AI summary Table 1 provides a summary of the 2023 evaluation of existing residential programs, including methodologies such as participant surveys, tracking sheet audits, and avoided GHG emission calculations. Programs like HEA, Green Heat, EPI, MHEEP, AMH, and ASFH are evaluated for their impact and process effectiveness.
[Table](#page-195-0) 2 presents the participation levels, net-to-gross ratios (NTGRs), evaluated gross and net savings at the generator, annual GHG emission reductions, as well effective useful life (EUL) values for each program component...
AI summary Table 2 provides data on participation levels, net-to-gross ratios, evaluated gross and net savings, annual GHG emission reductions, and effective useful life values for each program component and for Existing Residential as a whole.
HEA Findings and Recommendations 2023 HEA Finding: Both HEA net electrical energy and demand savings exceeded targets. 2023 HEA Finding: The significant increase in participation and average gross savings per home observed since the introd...
AI summary The 2023 HEA findings show that energy and demand savings exceeded targets, with increased participation and savings per home since the introduction of the Canada Greener Homes Grant. The HEA is well-received, and the evaluation found higher savings due to lower free-ridership. Billing analysis results were inconclusive, and the ORs from 2018 will not be used beyond 2023. A new evaluation approach for 2024 is recommended.
Green Heat Findings and Recommendations This subsection presents the key findings from the Green Heat evaluation. 2023 Green Heat-Finding: Green Heat fell short of its net savings targets in 2023, achieving 80% of its net energy savings ta...
AI summary The 2023 Green Heat evaluation found that the program underperformed its energy and peak demand savings targets, with participation declining by 11% compared to 2022. Mini-split heat pumps saw a 22% drop in participation, while the Canada Greener Homes Grant and HEA programs gained traction due to higher incentives. Customer and contractor satisfaction with Green Heat is low, citing issues with rebates, marketing, and program communication.
EPI Findings and Recommendations This subsection presents the key findings from the EPI evaluation. The Evaluator has no specific recommendation for EPI. 2023 EPI Finding: With 9.357 GWh in net electrical energy savings and 0.920 MW in net...
AI summary The EPI evaluation for 2023 showed that energy and peak demand savings exceeded targets by 13% and 21%, respectively. Participation increased by 16%, leading to higher savings. LED lamps and smart thermostats for mini-split heat pumps were popular measures. Net savings tracked by EOne were consistent with the evaluator's findings.
Table 5: Implementation Status of Past Recommendations for HEA # Past Recommendations Status Comments 2018 HEA-R1 Conduct a billing analysis to review overestimation ratios (ORs) when a sufficient participant sample becomes available. In p...
AI summary The implementation status of the 2018 HEA-R1 recommendation to conduct a billing analysis to review overestimation ratios (ORs) is in progress. Initial planning was for 2022, but it was deferred due to insufficient data. In 2022, the Evaluator researched changes to the methodology, and a new approach was tested in 2023, though results were inconclusive. Further exploration and potential new approaches are planned for 2024.
2 HEA Evaluation Approach The 2023 HEA evaluation comprised a comprehensive impact evaluation. The main objectives of the 2023 HEA evaluation were to: - › Collect information on participant perspectives 3 - › Calculate gross and net result...
AI summary The 2023 HEA evaluation aimed to collect participant perspectives and calculate energy savings and GHG emissions. The Evaluator identified research questions and methods to achieve these objectives, with Table 6 outlining the evaluation's approach.
Table 6: 2023 HEA Evaluation Approach Evaluation Objectives Research Questions Methodology Collect information on participant perspectives › What is the level of satisfaction among participants? › What participants suggest for improving th...
AI summary The 2023 HEA Evaluation Approach outlines objectives to collect participant perspectives, calculate gross and net results, and includes methodologies such as participant surveys, tracking sheet audits, billing analysis, and GHG emission reduction calculations.
3 HEA Participant Perspective A survey was conducted with 100 HEA participants as part of the HEA evaluation. In addition to the questions designed for impact evaluation purposes, survey participants were asked questions regarding satisfac...
AI summary A survey of 100 HEA participants was conducted as part of the HEA evaluation, including questions on satisfaction and recommendations in addition to impact evaluation questions.
3.1 Satisfaction with HEA There was a high level of satisfaction with HEA overall, with an average rating of 8.3. 5 Participants who were less satisfied (27%) mentioned that it took too long to receive their rebate (44%), which was likely...
AI summary Participants expressed high satisfaction with the Home Energy Assessment (HEA) program, with an average rating of 8.3. However, 27% of participants were less satisfied, citing issues such as delays in receiving rebates, unclear program requirements, and difficulties accessing information. The Net Promoter Score (NPS) for HEA was 62, indicating strong likelihood of participants recommending the program.
Overestimation Ratios The savings calculations are mainly based on the difference (reduction) between D and E assessment electrical space heating consumption levels obtained in HOT2000, both of which were adjusted with an OR. The ORs were...
AI summary The document discusses the use of overestimation ratios (ORs) in calculating energy savings from the Home Energy Assessment (HEA) program. A billing analysis was conducted in 2023 to review ORs and the savings calculation approach, but the results were inconclusive. The Evaluator recommends continuing to use the 2018 ORs for the 2023 evaluation due to the lack of definitive findings.
For HEA, participant spillover occurs when participants implement additional measures recommended in their initial energy assessments after their participation in the program component, i.e. after having completed the final energy assessme...
AI summary The document discusses spillover in the Home Energy Assessment (HEA) program, where participants implement additional energy efficiency upgrades after completing the program. A spillover level of 1% was measured in 2023, consistent with the 2020 measurement, based on participant surveys.
The NTGR is calculated using the following equation. NTGR = (1 – % Free-ridership + % Participant Spillover) The NTGR values calculated with the free-ridership and spillover levels established through the 2023 HEA participant survey are pr...
AI summary The NTGR is calculated using a formula that accounts for free-ridership and participant spillover, with values derived from the 2023 HEA participant survey and presented in Table 19.
Table 23: Comparison of 2023 HEA Tracked and Evaluated Savings at the Generator Gross Savings NTGR Net Savings Realization Rate Value Unit Value Value Unit Value Energy Savings Tracked Savings by EOne 30.584 GWh 0.79 24.018 GWh Evaluation...
AI summary Table 23 compares tracked and evaluated energy and peak demand savings from the 2023 Home Energy Assessment (HEA) program. It shows gross and net savings, along with realization rates, highlighting differences between tracked savings by EOne and evaluation results.
5 HEA Key Findings and Recommendations As mentioned previously, the main objectives of the 2023 HEA evaluation were as follows: - › Collect information on participant perspectives - › Calculate gross and net results, namely electrical firs...
AI summary The 2023 HEA evaluation found that the program exceeded energy and demand savings targets, with increased participation and average savings per home. Participant satisfaction was high, though delays in rebate processing were noted. Free-ridership levels were lower than previously tracked, and billing analysis results were inconclusive, prompting recommendations for further exploration and a revised evaluation approach for 2024.
Table 25: 2023 Green Heat Evaluation Approach Evaluation Objectives Research Questions Methodology Collect information on participant and partner perspectives › What is the level of satisfaction with the component among participants and co...
AI summary This section outlines the 2023 Green Heat Evaluation Approach, focusing on participant and partner perspectives, calculating gross and net results, and analyzing the market evolution of MSHPs. The methodology includes surveys, interviews, and data audits to evaluate program effectiveness and identify areas for improvement.
Note on Margin of Error For evaluation activities that yield quantitative results based on a sample, the Evaluator aimed to achieve a maximum margin of error of 10% at a confidence level of 90%. This means that, if measurements were conduc...
AI summary This section discusses the margin of error in evaluations of energy programs, specifically the Green Heat initiative. It outlines the 10% margin of error at a 90% confidence level, explaining that it reflects random sampling errors and does not account for non-sampling errors or biases.
8.2.1 Participants As the results in [Figure](#page-34-0) 12 below indicate, biomass and MSHP participants are highly satisfied with the equipment installed in the home and the selection of equipment eligible for rebate. That said, biomass...
AI summary Participants in the Green Heat program report high satisfaction with equipment and eligibility, but MSHP participants are less satisfied with rebate amounts and program clarity. Biomass participants are more satisfied overall. ETS participants also show high satisfaction.
8.2.2 Contractors The average rating of 6.6 for overall program satisfaction among contractors – with nine out of 20 being highly satisfied and eight providing a more neutral score – reveals that the program component does not perform opti...
AI summary Contractors report mixed satisfaction with the Green Heat program, with an average rating of 6.6. While they are satisfied with eligibility criteria, application forms, and the website, dissatisfaction stems from low incentives, communication issues with EOne, and restrictive eligibility criteria.
Table 27: Elements of Dissatisfaction and Justifications Element of Dissatisfaction (Score of <8) Reasons for Dissatisfaction/What Could Be Improved (#) n=11 Overall program › Communication issues/lack of information from EOne (4) › Eligib...
AI summary The table outlines elements of dissatisfaction with various programs, including communication issues, restrictive eligibility criteria, low incentive levels, and difficulties with online portals and application forms. Respondents highlighted the need for improved marketing, updated equipment lists, and more user-friendly interfaces.
More specifically: - › 11% of Green Heat participants identified a lack of website user-friendliness and related difficulties completing the online application form. - › 9% considered the application form confusing . - › 6% said that there...
AI summary The Green Heat program faces challenges with application completion, including user-friendliness issues, confusion, excessive paperwork, and difficulty finding required information. Contractors assist participants but note problems with incomplete forms and limited computer literacy, particularly among older applicants.
8.6 Suggestions for Improvements Among the participants who have suggestions on how to improve Green Heat: - › 17% suggest increasing the amount of the rebate. - › 11% mention simplifying the application form and associated paperwork. - ›...
AI summary Participants and contractors suggest improving the Green Heat program by increasing rebate amounts, simplifying application processes, and enhancing communication. They also recommend covering the switch from oil to ETS and improving marketing efforts.
9.1 Tracking Sheet Audit To ensure program component results were reliably compiled, the Evaluator first performed a tracking sheet audit aimed at verifying the completeness and consistency of the data submitted by EOne. The verification a...
AI summary The Evaluator conducted a tracking sheet audit to verify the completeness and consistency of data submitted by EOne, ensuring reliable compilation of program component results. Corrective actions are detailed in Appendix XIII, leading to corrected tracked savings results presented in the report.
13 EPI Evaluation Approach The 2023 EPI evaluation comprised a condensed impact evaluation. The main objectives of the 2023 EPI evaluation were as follows: › Calculate gross and net EPI results, namely electrical first-year and lifetime en...
AI summary The 2023 EPI evaluation focused on calculating gross and net energy savings, peak demand savings, and avoided GHG emissions. Key research questions were identified to achieve these objectives, with methods outlined in Table 41.
14.1 Tracking Sheet Audit To ensure program component results were reliably compiled, the Evaluator first performed a tracking sheet audit aimed at verifying the completeness and consistency of the data submitted by EOne. The verification...
AI summary A tracking sheet audit was conducted to verify the completeness and consistency of data submitted by EOne, ensuring reliable compilation of program component results. Corrective actions taken by the Evaluator are detailed in Appendix XVI, and the reported tracked savings reflect these corrections.
Table 57: 2023 MHEEP Evaluation Approach Evaluation Objectives Research Questions Methodology Calculate gross results › Are the data in the tracking sheet complete, accurate, and consistent? › What are the heating and building envelope mea...
AI summary This section outlines the evaluation approach for the 2023 MHEEP, focusing on calculating gross and net results through tracking sheet audits, unitary savings reviews, and calculations using evaluation results and a net-to-gross ratio of 1.
21 AMH Evaluation Approach The 2023 evaluation consisted of a condensed impact evaluation. The main objectives of the 2023 AMH evaluation were as follows: › Calculate gross and net AMH results, namely electrical first-year and lifetime ene...
AI summary The 2023 AMH evaluation focused on calculating gross and net results, including energy savings, peak demand savings, and avoided GHG emissions. The evaluation involved identifying key research questions and mapping them to methods and sample sizes.
22.1 Tracking Sheet Audit To ensure program results were reliably compiled, the Evaluator first performed a tracking sheet audit aimed at verifying the completeness and consistency of the data submitted by EOne as well as correcting the tr...
AI summary An audit of tracking sheets was conducted to ensure the reliability of program results, verifying the completeness and consistency of data submitted by EOne and correcting tracked savings as needed. The corrected results are detailed in Appendix XX.
22.3.1 Evaluated Net Savings Net savings are defined as the changes in energy use that are specifically attributable to AMH. Since spillover and free-ridership effects were considered nil, the net energy savings are equal to the gross savi...
AI summary The AMH program exceeded its electrical energy and peak demand savings targets by 17% and 118% respectively, with net savings equal to gross savings due to nil spillover and free-ridership effects.
26.1 Tracking Sheet Audit To ensure program component results were reliably compiled, the Evaluator first performed a tracking sheet audit aimed at verifying the completeness and consistency of the data submitted by EOne as well as correct...
AI summary The Evaluator conducted a tracking sheet audit to verify the completeness and consistency of data submitted by EOne and correct tracked savings as necessary. The results of this audit are detailed in Appendix XXII, and the tracked savings figures in the report reflect these corrections.
26.3.1 Evaluated Net Savings Net savings are defined as the changes in energy use that are specifically attributable to ASFH. Since spillover and free-ridership effects were considered nil, the net energy savings are equal to the gross sav...
AI summary The document evaluates the net savings of the Affordable Single-Family Homes (ASFH) program, noting that it fell short of its electrical energy and peak demand savings targets by 59% and 30% respectively. Net savings are defined as energy use changes directly attributable to the program, with no spillover or free-ridership effects considered.
Home Energy Assessment Appendix I HEA Participant Survey Questionnaire Appendix II HEA Participant Survey Results Appendix III HEA Tracking Sheet Audit Appendix IV HEA Detailed Methodology and Billing Analysis Results Appendix V HEA Algori...
AI summary The document contains appendices related to the Home Energy Assessment (HEA), including survey questionnaires, results, tracking sheets, methodology, free-ridership calculations, spillover analysis, reporting requirements, and 2023 recommendations.
Table 1: Overview of Data Collection Activity Descriptor This Instrument Instrument Type Telephone survey Estimated Time to Complete 15 min Target Audience Participants whose projects generated electrical savings in 2023 Expected Number of...
AI summary The document outlines two tables related to a data collection activity. Table 1 describes a telephone survey targeting participants in energy efficiency projects in 2023, while Table 2 lists research objectives and corresponding sections. The text indicates that the document is to be adapted by Econoler.
Responses of 'Don't know' and 'Refused' are excluded from the mean calculation C7A, B,C. I would like you to rate the importance of the following three factors in your decision to install your solar PV system. Use a scale from 0 to 10 wher...
AI summary The text discusses the importance of factors influencing the decision to install a solar PV system, with respondents rating each factor on a scale from 0 to 10. The context involves customer feedback and program evaluation in a regulatory proceeding.
APPENDIX III HEA Tracking Sheet Audit This appendix presents the main results of the tracking sheet audit performed by the Evaluator, which was aimed at: - › Verifying that all data fields required for the evaluation were included and fill...
AI summary This appendix outlines the results of a tracking sheet audit conducted by the Evaluator to ensure data completeness and accuracy in EOne's submission, including verification of parameters and calculation consistency for program results.
This appendix summarizes all the recommendations made by the Evaluator as part of the 2023 HEA evaluation. Section Recommendations Executive Summary Recommendation #1: Continue exploring the factors potentially responsible for the inconclu...
AI summary This appendix summarizes recommendations from the 2023 HEA evaluation, including the need to explore factors affecting billing analysis results and reconsider evaluation approaches for 2024 due to changes in program participant characteristics.
[READ AND ROTATE (F1 + F2 TO F4) AND (F5 + F6 TO F8) SEQUENCES] - F1. Before participating in the Heating System Rebates program in 2023, had you at any time in the past participated in any Efficiency Nova Scotia program or service? - 1. Y...
AI summary The text includes a survey question about prior participation in Efficiency Nova Scotia programs and a follow-up question about the influence of that participation on the decision to install heating equipment, indicating a focus on customer engagement and program impact evaluation.
[0-10 SCALE WITH END POINT LABELS– RECORD NUMBER] - 98. Don't Know - 99. Refused - G3. [I[F H2<](#page 1-1)8] What was the main reason you were not more satisfied with the program overall? [PROBE FOR SPECIFIC REASON. RECORD FIRST MENTION A...
AI summary The text outlines survey questions related to customer satisfaction with the Heating System Rebates program, focusing on aspects such as program clarity, application process, rebate amounts, and communication with Efficiency Nova Scotia. Respondents are asked to rate their satisfaction on a scale from 0 to 10.
Aspects of the Program Score 0 = Not at all satisfied 10 = Completely Satisfied Reason If 7 or less, please share the reason(s) for your score. a. The overall program /10 What could be improved? b. The list of equipment that qualifies for...
AI summary The document outlines aspects of a program that require evaluation, including the overall program, eligibility criteria, incentive levels, and application form. Each aspect is rated on a scale from 0 to 10, with reasons for scores below 7.
APPENDIX XIII Green Heat Tracking Sheet Audit This appendix presents the main results of the tracking sheet audit performed by the Evaluator, which was aimed at: - › Verifying that all data fields required for the evaluation were included...
AI summary This appendix details the results of a tracking sheet audit conducted by the Evaluator to verify the accuracy and completeness of data submitted by EOne, focusing on program results such as energy and peak demand savings.
APPENDIX XVIII MHEEP Tracking Sheet Audit This appendix presents the main results of the tracking sheet audit performed by the Evaluator, which was aimed at: - › Verifying that all data fields required for the evaluation were included and...
AI summary This appendix outlines the results of a tracking sheet audit conducted by the Evaluator to ensure the completeness and accuracy of data submitted by EOne. The audit focused on verifying that all required data fields were included and that the calculations for program results were consistent with previous evaluations.
APPENDIX XX AMH Tracking Sheet Audit This appendix presents the main results of the tracking sheet audit performed by the Evaluator, which was aimed at: - › Verifying that all data fields required for the evaluation were included and fille...
AI summary This appendix outlines the results of a tracking sheet audit conducted by the Evaluator to ensure data completeness and accuracy in the tracking sheet submitted by EOne. The audit focused on verifying consistency in parameters used for calculating program results and validating calculation steps.
APPENDIX XXII ASFH Tracking Sheet Audit This appendix presents the main results of the tracking sheet audit performed by the Evaluator, which was aimed at: - › Verifying that all data fields required for the evaluation were included and fi...
AI summary This appendix outlines the results of a tracking sheet audit conducted by the Evaluator to verify data completeness and accuracy in the tracking sheet submitted by EOne. The audit focused on ensuring consistency in parameters used for calculating program results and validating calculation steps.
3.1 Tracking Sheet Audit To ensure program component results were reliably compiled, the Evaluator first performed a tracking sheet audit aimed at verifying the completeness and consistency of the data submitted by EOne. The results obtain...
AI summary The Evaluator conducted a tracking sheet audit to verify the completeness and consistency of data submitted by EOne, ensuring reliable compilation of program component results. The findings are detailed in Appendix I, and the report refers to corrected tracked savings.
3.3.1 Free-ridership Free-ridership occurs when participant homeowners or participant builders would have implemented energy efficiency measures in the absence of the program component. The assessment of the NHC free-ridership level is bas...
AI summary Free-ridership in the NHC program is assessed through self-reported surveys of participant homeowners and builders. The 2023 free-ridership level was calculated using data from the 2019 evaluation and the proportion of NHC applicants in 2023. The overall free-ridership level for homeowners and builders combined was 24%.
This appendix presents the main results of the tracking sheet audit performed by the Evaluator, which was aimed at: - › Verifying that all data fields required for the evaluation were included and filled out in the tracking sheet submitted...
AI summary This appendix outlines the results of a tracking sheet audit conducted by the Evaluator to ensure data completeness and accuracy in EOne's submissions. The audit verified consistency in parameters used for calculating program results and validated calculation steps.
INTRODUCTION EfficiencyOne (EOne), an independent, non-profit organization, is responsible for helping Nova Scotians improve the energy efficiency of their homes and workplaces by designing, marketing, and delivering demand-side management...
AI summary EfficiencyOne (EOne) is an independent, non-profit organization that delivers demand-side management (DSM) programs in Nova Scotia through the Efficiency Nova Scotia (ENS) franchise. EOne's 2023 DSM program portfolio includes the Efficient Product Rebates program, which is evaluated using parameters such as unitary savings values, adjustment ratios, and net-to-gross ratios.
1 BER Overview This section describes Business Energy Rebates (BER), follows up on past evaluation recommendations, and presents participation history.
AI summary This section outlines the Business Energy Rebates (BER) program, discusses past evaluation recommendations, and provides participation history data.
1.2 Follow-up on Past Evaluation Report Recommendations The Evaluator evaluated BER in previous years and issued improvement recommendations. [Table](#page-137-1) 5 below provides a summary of the implementation status of past recommendati...
AI summary The Evaluator has assessed the Business Energy Rebates (BER) program and provided improvement recommendations. Table 5 outlines the implementation status of past recommendations, with only one from 2022 still in progress.
3.1 Tracking Sheet Audit To ensure program component results were reliably compiled, the Evaluator first performed a tracking sheet audit aimed at verifying the completeness and consistency of the data submitted by EOne. The results obtain...
AI summary A tracking sheet audit was conducted to verify the completeness and consistency of data submitted by EOne, ensuring reliable compilation of program component results. Corrected tracked savings results are presented in Appendix I.
4 Impact Evaluation for Instant Rebates The objectives of the 2023 Instant Rebates impact evaluation were to determine gross and net electrical energy and peak demand savings.
AI summary The 2023 Instant Rebates impact evaluation aimed to assess gross and net electrical energy and peak demand savings, providing insights into the effectiveness of the program.
This appendix presents the main results of the tracking sheet audit performed by the Evaluator, which was aimed at: - › Verifying that all data fields required for the evaluation were included and filled out in the tracking sheet submitted...
AI summary This appendix outlines the results of a tracking sheet audit conducted by the Evaluator, focusing on verifying data completeness and accuracy of program results, including energy and peak demand savings calculations, and presents corrected tracked savings in Table 1.
APPENDIX II BER Instant Rebates Tracking Sheet Audit This appendix presents the main results of the tracking sheet audit performed by the Evaluator, which was aimed at: - › Verifying that all data fields required for the evaluation were in...
AI summary This appendix outlines the results of a tracking sheet audit conducted to verify the completeness and accuracy of data submitted by EOne for the BER Instant Rebates program. The audit focused on ensuring consistency in parameters and calculation methods used to determine energy and peak demand savings.
Table 1: Summary of 2023 Custom Incentives Program Evaluation Program Evaluation Type Methodology Component Impact Process Market Custom Comprehensive X - Retrofit › Participant phone interviews (New Construction) › Consultant and Business...
AI summary In 2023, the Strategic Energy Management (SEM) and Energy Management Information Systems (EMIS) programs were merged into a single component. The evaluation of the Custom Incentives Program in 2023 included comprehensive assessments of retrofit and new construction projects, involving interviews, project reviews, and calculations for greenhouse gas (GHG) emissions reduction.
Program Performance Table 2 below presents the participation levels, average net-to-gross ratios (NTGRs), evaluated gross and net savings at the generator, annual GHG emission reductions, as well as average EUL values for each program comp...
AI summary Table 2 provides data on program participation levels, net-to-gross ratios, gross and net savings, GHG emission reductions, and average effective useful life values for each program component and Custom Incentives as a whole.
Custom General Key Findings and Recommendations 2023 Custom Finding: Custom achieved 21.746 GWh in net electrical energy savings and 3.053 MW in net peak demand savings at the generator in 2023, thus falling short of the planned electrical...
AI summary In 2023, Custom fell short of its energy and peak demand savings targets, partly due to decreased participation and adjustments made during project reviews. Free-ridership levels increased for Retrofit and solar PV projects, while spillover effects were minimal. Evaluated savings were lower than those tracked by EOne.
SEM Findings and Recommendations This section provides the key findings and recommendations from the SEM evaluation. The recommendations are also outlined in Appendix XIX. 2023 SEM-Finding: SEM achieved 3.381 GWh in net electrical energy s...
AI summary The 2023 SEM evaluation found that the program achieved energy savings above targets, with high participant satisfaction. However, it lacks metrics for capital project realization and faces barriers like time commitment and staffing. Recommendations include tracking capital projects and improving communication on energy management's value.
Program Tracked and Evaluated Savings [Table](#page-191-0) 3 below compares EOne tracked energy and peak demand savings to evaluated savings at the generator. It also includes the realization rate, representing the ratio of evaluated net s...
AI summary The text discusses the comparison between tracked energy and peak demand savings by EOne and evaluated savings at the generator, along with realization rates and NTGR values, which are calculated as the ratio of net savings to gross savings.
EfficiencyOne (EOne), an independent, non-profit organization, is responsible for helping Nova Scotians improve the energy efficiency of their homes and workplaces by designing, marketing, and delivering demand-side management (DSM) for No...
AI summary EfficiencyOne (EOne), a non-profit organization, delivers energy efficiency programs in Nova Scotia through the Efficiency Nova Scotia (ENS) franchise. EOne's 2023 Custom Incentives program, including Custom and Strategic Energy Management (SEM) components, was evaluated by Econoler. The evaluation focuses on baseline definitions, savings calculation methodologies, parameter values, and net-to-gross ratios.
Pay-for-Performance (P4P) - Financial support for reductions in electricity consumption achieved through a variety of capital and behavioural energy efficiency upgrades - •Incentives are based on verified facility energy performance - Orga...
AI summary The Pay-for-Performance (P4P) program provides financial support for electricity consumption reductions through energy efficiency upgrades. Eligibility is based on annual consumption and project-generated savings. Savings claims are categorized into partial, single-year, and multiyear projects, with adjustments applied in 2023. Combined programs aim for 22.730 GWh of net energy savings and 5.280 MW of peak demand savings in 2023.
1.2 Follow-up on Past Evaluation Report Recommendations The Evaluator evaluated Custom in previous years and issued improvement recommendations. [Table](#page-195-0) 5 below provides a summary of the implementation status of each recommend...
AI summary This section discusses the follow-up on past evaluation report recommendations, providing a summary of the implementation status of recommendations from the 2021 and 2022 Evaluation Reports. A table is referenced to detail the progress made on these recommendations.
Table 5: Implementation Status of Past Recommendations for Custom # Recommendations Status Comments 2022 New Construction R2 Update the Program Guide to include the approach proposed in the 2022 evaluation for fan power baselines in multi-...
AI summary Table 5 outlines the implementation status of past recommendations for custom programs. Specifically, it notes that Recommendation 2022 New Construction R2 has been completed, with EOne revising the New Construction Program Guide to include a new approach to the fan power baseline for multi-unit residential buildings (MURBs).
2 Custom Evaluation Approach The 2023 Custom evaluation comprised a comprehensive impact evaluation for Retrofit, Building Optimization, and New Construction. For Building Optimization, given its smaller contribution to Custom savings, NTG...
AI summary The 2023 Custom evaluation focused on assessing the impact of Retrofit, Building Optimization, and New Construction programs. It aimed to calculate energy savings, develop a method for solar PV projects, and gather participant perspectives and motivations. The evaluation used NTGR results for Building Optimization and outlined research questions, methods, and sample sizes.
Table 6: 2022 Custom Evaluation Approach Evaluation Objectives Research Questions Methodology Collect information on participant and partner perspectives on Retrofit › What is the level of satisfaction with the different elements of Custom...
AI summary This section discusses the 2022 Custom Evaluation Approach, focusing on collecting information on participant and partner perspectives on Retrofit. It outlines research questions related to satisfaction, impact of decarbonization and electrification objectives, challenges, and ways to improve Custom. The methodology includes interviews with Business Development Managers, participants, and consultants.
Participant Phone Interviews To collect information on participant free-ridership and perspectives, the Evaluator conducted participant phone interviews with 13 regular Retrofit participants, six solar PV participants, 7 and eight New Cons...
AI summary The Evaluator conducted phone interviews with participants in Retrofit, solar PV, and New Construction programs to gather information on free-ridership and perspectives. Interview guides are provided in appendices for each participant group.
Note on Margins of Error For evaluation activities that yield quantitative results based on a sample, the Evaluator aimed to achieve a maximum margin of error of 10% at a confidence level of 90%. This means that if measurements were conduc...
AI summary The note explains the margin of error used in the evaluation of DSM programs, aiming for a 10% margin at a 90% confidence level. This applies to savings adjustment ratios for Retrofit and New Construction, but not for Building Optimization, as all 2023 projects were fully reviewed. The note clarifies that precision relates to reproducibility, not accuracy.
3 Retrofit Process Evaluation The results of the Retrofit process evaluation are separated into two main subsections. Subsection [3.1](#page-4-0) summarizes participant and consultant perspectives, while Subsection [3.2](#page-14-0) is foc...
AI summary The Retrofit process evaluation is divided into two subsections: one summarizing participant and consultant perspectives, and the other focused on the logic model review.
Consultant Perspectives Consistent with participant feedback, consultants indicated that project costs and insufficient returns on investment are the primary barriers to businesses carrying out energy efficiency projects (n=6). Two consult...
AI summary Consultants identified project costs, insufficient returns on investment, and lengthy timelines as key barriers to energy efficiency projects for businesses. They suggested standardizing forms and providing additional training for EOne staff to reduce administrative burdens and improve processing times.
3.1.6 Measurement and Verification
AI summary This section addresses the importance of measurement and verification in energy efficiency programs. It outlines the need for accurate tracking of energy savings and the methodologies used to ensure program effectiveness.
Participant Perspectives Since solar PV projects are now distinguished from regular Retrofit projects within the Retrofit offer, participant satisfaction results are presented separately for each project category. Participants are satisfie...
AI summary Participants expressed high satisfaction with Custom Retrofit projects, with solar PV participants scoring higher than regular Retrofit participants. EOne received positive feedback for its responsiveness and technical expertise but faced criticism regarding rebate amounts and clarity in energy savings calculations.
Table 8: Reasons for Dissatisfaction Factor Assessed Reasons for Dissatisfaction/What Could Be Improved (#) n=4 Value of the service relative to the time and cost involved › Incentives too low/should be more value for the customer (1) › Cu...
AI summary The table outlines reasons for dissatisfaction with retrofit programs, citing issues such as low incentives, complex processes, expensive equipment, and poor customer support. Consultants report that the programs are too time-consuming, not financially rewarding, and lack adequate support and transparency.
3.2.1 Logic Model The main objective of the logic model is to link program objectives to the barriers thereto and identify actions or offers to address the barriers. The Evaluator examined the Retrofit logic model to assess alignment betwe...
AI summary The logic model aims to connect program objectives with barriers and identify actions to address them. The Evaluator reviewed the Retrofit logic model to assess alignment with observed barriers and program offers, identifying gaps and improvement opportunities.
Objectives According to the program manual, Retrofit and Custom Incentives more broadly have the following objectives: - › Influence electrical energy efficiency and system peak demand reduction projects in Nova Scotia - › Build awareness...
AI summary The objectives of Retrofit and Custom Incentives programs are outlined, emphasizing energy efficiency and system peak demand reduction. However, the logic model fails to adequately represent the third and fourth objectives, particularly the barriers customers face and the diversification of EOne program offerings.
Barriers and Actions The logic model indicates the primary barrier to achieving program objectives is that projects are "not eligible for standard EOne programs". While this is a legitimate barrier to the execution of large and complex ene...
AI summary The logic model for the Retrofit program underestimates barriers such as project eligibility, cost, and lack of technical support. It also fails to account for program actions like SEM and OEMs, which provide project management resources. The model assumes Retrofit participation will lead to spillover, but this is not well-supported.
[Table](#page-16-2) 10 summarizes the impact evaluation approach for each project category under Retrofit. Results for the three project categories are presented below and are then combined as the aggregated results for Retrofit in Subsect...
AI summary Table 10 outlines the impact evaluation approach for Retrofit project categories. It excludes partial savings projects and presents results for three categories, which are later combined into aggregated results for Retrofit in Subsections 4.2.7 and 4.3.4.
4.3.1 Free-ridership In the case of Retrofit, free-ridership occurs when participants would have still implemented energy efficiency upgrades and measures in the absence of the service. For compressed air leak audit projects, the NTGR was...
AI summary The document discusses the free-ridership assessment for Retrofit and solar PV projects, explaining how participants might have implemented energy efficiency measures independently of the program. The 2021 NTGR was used for compressed air leak audits due to missing 2022 and 2023 data. Free-ridership levels were assessed using self-reporting via phone interviews, with adjustments made based on participant influence from EOne programs. The evaluation used a sample size of 13 out of 20, and results were calculated using an algorithm in Appendix V.
4.3.2 Spillover For Retrofit, participant spillover occurs when participants implement eligible energy efficiency measures due to influence from previously participating in the service without receiving any kind of additional support. The...
AI summary The document discusses spillover effects in the Retrofit program, where participants implement energy efficiency measures independently after participating in the service. The Evaluator used phone interviews and an algorithm to assess spillover, estimating 2% for regular Retrofit and 1% for solar PV projects.
Table 18: Evaluated 2023 Retrofit NTGRs Project Category Free-ridership Level Spillover Level NTGR Regular Retrofit 25% 2% 0.77 Solar PV 38% 1% 0.63 Compressed Air Leak Audit 16% 0% 0.84 \ Free-ridership and spillover for compressed air le...
AI summary Table 18 presents the evaluated 2023 Retrofit NTGRs for different project categories, including free-ridership and spillover levels. Values for compressed air leak audit projects are from the 2021 evaluation as they were not measured in 2023.
Peak Demand Savings To establish evaluated peak demand savings, the Evaluator averaged the monthly peak demand reductions calculated in the revised energy models for the months of December, January, and February. The differences between tr...
AI summary The Evaluator calculated evaluated peak demand savings by averaging monthly peak demand reductions in December, January, and February. Discrepancies between modelled and tracked demand savings were significant, with most projects differing by over 20%. After adjustments, the average adjustment ratio was 0.776 with a 12.6% margin of error.
Table 28: Evaluated 2023 Building Optimization NTGR Value for Savings Claimed in 2023 Free-ridership Level 9% (±7.2%) Participant Spillover Level 0% NTGR 0.91 Free-ridership and spillover levels were not measured in 2022 or 2023, results f...
AI summary Table 28 evaluates the 2023 Building Optimization NTGR, showing a free-ridership level of 9% (±7.2%) and a spillover level of 0%. The NTGR value is reported as 0.91, with data from the 2021 Custom evaluation used due to the absence of measurements in 2022 and 2023.
General Custom Key Findings and Recommendations 2023 Custom Finding: Custom net electrical energy and peak demand savings fell short of targets in 2023. Custom achieved 21.746 GWh in net electrical energy savings and 3.053 MW in net peak d...
AI summary In 2023, Custom's net electrical energy and peak demand savings fell short of targets by 4% and 42% respectively. Participation in Custom programs decreased due to the lack of Pay-for-Performance (P4P) service and the removal of the Retrofit Operational Demand Reduction offer. Project reviews led to adjustments in energy and peak demand savings for Retrofit, New Construction, and Building Optimization.
New Construction Key Findings and Recommendations 2023 New Construction Finding: Project reviews revealed continued room for improvement in the model review process. The reviews of New Construction projects revealed that the building energ...
AI summary The 2023 New Construction evaluation found that while building energy models aligned with design specs, site visits revealed less efficient equipment was installed. Free-ridership remained high at 31%, with participants citing external factors as motivators for energy efficiency. EOne plans a process evaluation for 2024.
9.2 Follow-up on Past Evaluation Report Recommendations The Evaluator evaluated SEM in previous years and issued improvement recommendations. [Table](#page-49-0) 32 provides a summary of the implementation status of the recommendations pre...
AI summary This section discusses the follow-up on past evaluation report recommendations, specifically focusing on the implementation status of recommendations issued in the 2022 Evaluation Report. A table is referenced to summarize the progress made.
10 SEM Evaluation Approach The 2023 SEM evaluation comprised a comprehensive impact evaluation. The main objectives of the SEM evaluation were as follows: - › Calculate SEM gross and net results, namely electrical first-year and lifetime e...
AI summary The 2023 SEM evaluation aimed to calculate energy savings, peak demand reductions, and GHG emissions avoided, while collecting participant perspectives and updating the program logic. Research questions, methods, and sample sizes were outlined in a table.
Table 33: 2023 SEM Evaluation Approach Evaluation Objectives Research Questions Methodology Collect information on participant and partner perspectives › What is the level of satisfaction with SEM? › To what extent do SEM participants adop...
AI summary Table 33 outlines the 2023 SEM Evaluation Approach, focusing on collecting participant perspectives, updating program logic, calculating gross and net results, and identifying barriers and improvements for the SEM program. The methodology includes interviews, audits, and calculations.
Logic Model Review The logic model review was aimed at determining whether SEM was achieving its goals in the way that it was anticipated at the time of design. The Evaluator reviewed the logic model presented in the program manual and com...
AI summary The logic model review assessed whether the Strategic Energy Management (SEM) program was achieving its intended goals as designed. The Evaluator compared the assumptions in the program manual with actual implementation findings from the BDM, Service Provider, and participant interviews.
Tracking Sheet Audit Prior to performing the savings review, the Evaluator audited the final 2023 tracking sheet to ensure it was complete and data entry was consistent. The detailed protocol used for the tracking sheet audit and the resul...
AI summary The Evaluator conducted an audit of the final 2023 tracking sheet to ensure its completeness and consistent data entry. The audit protocol and results are detailed in Appendix XVI.
Project Reviews In January 2023, a total of six project reviews were conducted, one for each participant that generated savings. Four of these project reviews were supported by phone interviews, while site visits were conducted for the oth...
AI summary In January 2023, six project reviews were conducted for participants generating savings. Four reviews were supported by phone interviews, and two involved site visits. Econoler conducted desk reviews and phone interviews, while Equilibrium Engineering conducted site visits for two new participants. The Evaluator contacted the Service Provider for clarifications.
11.2 Logic Model Review The logic model review was aimed at determining whether SEM was achieving its goal in the way that it was anticipated at the time of design. To do so, the Evaluator first assessed which elements of the CEE SEM Minim...
AI summary The logic model review evaluated whether the Strategic Energy Management (SEM) program was achieving its intended goals as designed. The Evaluator compared the CEE SEM Minimum Elements completed by participants with findings from interviews with BDM, Service Providers, and participants.
12.1 Tracking Sheet Audit To ensure program component results were reliably compiled, the Evaluator first performed a tracking sheet audit aimed at verifying the completeness and consistency of the data submitted by EOne. The results obtai...
AI summary A tracking sheet audit was conducted to verify the completeness and consistency of data submitted by EOne, ensuring reliable compilation of program component results. The corrected tracked savings results are presented in Appendix XVI.
13 SEM Key Findings and Recommendations As mentioned previously, the main objectives of the 2023 SEM evaluation were as follows: - › Calculate SEM gross and net results, namely electrical first-year and lifetime energy savings, peak demand...
AI summary The 2023 SEM evaluation found that the program exceeded its energy and peak demand savings targets, maintained stable participation levels, and received high participant satisfaction. The bottom-up M&V approach did not hinder the program's energy management objectives.
CONCLUSION Table 39 presents the participation levels, net-to-gross ratios (NTGRs), evaluated gross and net savings at the generator, annual GHG emission reductions, as well effective useful life (EUL) values for each Custom Incentives pro...
AI summary The conclusion section references Table 39, which provides data on participation levels, net-to-gross ratios, evaluated savings, GHG emission reductions, and effective useful life values for the Custom Incentives program and its components.
Custom Appendix I Retrofit and SEM Interview Guide with BDMS Appendix II Retrofit and Solar Participant Interview Guide Appendix III Retrofit Consultant Interview Guide Appendix IV Custom Tracking Sheet Audit Appendix V Retrofit Algorithm...
AI summary The document outlines various appendices related to energy efficiency programs, including interview guides, tracking sheets, algorithms for free-ridership and spillover calculations, project review protocols, and recommendations for a custom 2023 initiative.
RETROFIT-SEM PROGRAM EVALUATION
AI summary The document introduces the RETROFIT-SEM Program Evaluation, which assesses the effectiveness of the Retrofit and Strategic Energy Management (SEM) programs. It outlines the evaluation process and its importance in ensuring program success and compliance with energy efficiency goals.
APPENDIX IV Custom Tracking Sheet Audit This appendix presents the main results of the tracking sheet audit performed by the Evaluator, which was aimed at: - › Verifying that all data fields required for the evaluation were included and fi...
AI summary This appendix outlines the results of a tracking sheet audit conducted by the Evaluator to verify the completeness and accuracy of data submitted by EOne. The audit focused on ensuring consistency in parameters used for calculating program results and validating calculation steps.
APPENDIX V Retrofit Algorithm for Free-Ridership Calculation Question Response Score INTRODUCTION I am with and we are conducting an evaluation of the Custom Retrofit program offered by Efficiency Nova Scotia. This interview should take up...
AI summary This text is an introduction to an interview conducted by Efficiency Nova Scotia to evaluate the Custom Retrofit program. The interview aims to understand the decision-making process of an organization regarding its energy efficiency project, specifically a retrofit project at a specified location.
Reminder of Evaluation Goals and Key Principles While developing the checklist, the Evaluator kept in mind the key evaluation goals and the five guiding principles presented in the 2020-2022 Overall Strategic Evaluation Plan. Notably for N...
AI summary The Evaluator focused on reviewing key measures contributing the most savings in complex programs like Custom New Construction, without a comprehensive review of entire models or M&V procedures, as outlined in Principle 4 of the 2020-2022 Overall Strategic Evaluation Plan.
Table 1: Participant Interview Questionnaire and Free-ridership Algorithm Question (From the Custom New Construction Participant Interview Guide) Response Score INTRODUCTION I am with and we are performing an evaluation of the Custom New C...
AI summary The text outlines an interview questionnaire used to evaluate the Custom New Construction program by Efficiency Nova Scotia, aiming to understand participants' decision-making processes in designing energy-efficient buildings beyond code requirements.
B. Free-ridership and Spillover Series of questions to be asked for each measure. - B1. Without the support from EPS, would you have identified this energy efficiency opportunity? - 1. Yes - 2. No - 98. Don't know - 99. Refused - B2. [Ask...
AI summary This section outlines a series of questions to assess free-ridership and spillover effects related to energy efficiency projects. It asks whether support from EPS and Efficiency Nova Scotia influenced the identification and completion of these projects.
Interviewed participants were asked free-ridership validation questions. These questions are listed at the end of the protocol. - € Efficiency Nova Scotia ECONOL≣R Project Review Protocol 1. General Information Interview/Site Visit Date: P...
AI summary The document outlines a protocol for reviewing energy efficiency projects, including general information, participation history, and impact evaluation. It includes sections for recording details about the project, participants, and energy consumption drivers.
Table 1: Summary of 2023 Direct Installation Program Evaluation Program Evaluation Type Methodology Component Impact Process Market Small Business Energy Solutions Comprehensive1 X - › EOne staff interviews: Energy auditor, business develo...
AI summary The 2023 Direct Installation Program Evaluation for Small Business Energy Solutions includes comprehensive impact and process evaluations. The methodology involves interviews, surveys, audits, and calculations to assess program performance and effectiveness.
SBES Findings and Recommendations This subsection presents the key findings from the SBES evaluation. 2023 SBES-Finding: Net electrical energy and peak demand savings did not meet targets for 2023. 2023 SBES-Finding: Participation in SBES...
AI summary The 2023 SBES evaluation found that net electrical energy and peak demand savings did not meet targets, participation decreased, and free-ridership for DIY increased. Recommendations include improving the CIS and implementing communication strategies for participants nearing project expiration. Satisfaction levels remain high, but stakeholders are dissatisfied with processing times.
Table 3: Comparison of 2023 SBES Tracked and Evaluated Savings at the Generator 3 Gross Savings NTGR Net Savings Realization Rate Value Unit Value Unit Energy Savings Tracked Savings by EOne 8.915 GWh 0.86 7.629 GWh 98% Evaluation Results...
AI summary Table 3 compares the tracked and evaluated energy and peak demand savings from the 2023 SBES program, including the CDI pilot. Tracked savings by EOne are compared to evaluation results, showing slight differences in gross and net savings, along with realization rates.
INTRODUCTION EfficiencyOne (EOne), an independent, non-profit organization, is responsible for helping Nova Scotians improve the energy efficiency of their homes and workplaces by designing, marketing, and delivering demand-side management...
AI summary EfficiencyOne (EOne), a non-profit organization, manages demand-side management (DSM) programs through the Efficiency Nova Scotia (ENS) franchise. The report evaluates the Direct Installation program, specifically the Small Business Energy Solutions (SBES) component, focusing on baseline definitions, savings calculation methodologies, parameter values, and net-to-gross ratios (NTGRs).
1.2 Follow-up on Past Evaluation Report Recommendations No major past recommendations remained unimplemented for SBES.
AI summary The document indicates that no major past recommendations from the evaluation report remain unimplemented for the Small Business Energy Solutions (SBES) program.
2 SBES Evaluation Approach The 2023 SBES evaluation consisted of a comprehensive impact evaluation and a process evaluation. The evaluation for the Commercial Direct Install (CDI) pilot was a condensed evaluation as it was evaluated in a p...
AI summary The 2023 SBES evaluation included a comprehensive impact and process evaluation, with the Commercial Direct Install (CDI) pilot being a condensed evaluation due to prior assessment. The main objectives were to collect perspectives and calculate energy savings, peak demand savings, and GHG emissions.
Table 5: 2023 SBES Evaluation Approach Evaluation Objectives Research Questions Methodology Collect information on participant, partner, and staff perspectives › What is the level of satisfaction with the service among participants and con...
AI summary Table 5 outlines the 2023 SBES Evaluation Approach, which includes collecting participant and contractor perspectives, calculating gross and net results, and evaluating program effectiveness. It involves various methods such as interviews, surveys, tracking sheet audits, and site visits.
Participant Survey In November 2023, Narrative Research conducted a telephone survey with a total of 46 SBES participants. The average length of the survey was 15.4 minutes. The questionnaire used is presented in Appendix II and complete s...
AI summary In November 2023, Narrative Research conducted a telephone survey with 46 SBES participants. The survey had a 15.4-minute average duration, and a margin of error of 11.4% at 90% confidence level due to a small sample size and limited response rate. The Evaluator found the results acceptable for a qualitative process evaluation.
Tracking Sheet Audit Prior to performing the savings calculation review, the Evaluator performed an audit of the final 2023 tracking sheet to ensure it was complete and the entered data were consistent. The detailed protocol used for the t...
AI summary An audit of the final 2023 tracking sheet was conducted to ensure completeness and data consistency prior to performing a savings calculation review. The audit protocol and results are detailed in Appendix VI.
Note on Margins of Error For evaluation activities that yield quantitative results based on a sample, the Evaluator aimed to achieve a maximum margin of error of 10% at a confidence level of 90%. This means that, if measurements were condu...
AI summary The document discusses the margin of error in evaluation activities, aiming for a 10% margin at a 90% confidence level. It explains that this margin of error reflects the precision of measurements, not their accuracy, and that it only accounts for random sampling errors, not nonsampling errors.
3 SBES Process Evaluation This section presents the results of surveys and interviews with SBES participants, dropped-out participants (businesses that applied to SBES but whose projects had not been completed after a minimum of 8 months),...
AI summary This section evaluates the SBES program through surveys and interviews with participants, dropped-out participants, contractors, and program staff, presenting results by topic and stakeholder group.
3.3 Participant and Contractor Satisfaction
AI summary This section discusses participant and contractor satisfaction in the context of energy efficiency programs. It likely includes metrics and feedback mechanisms used to evaluate the effectiveness of these initiatives.
Table 10: Satisfaction with SBES Overall Overall Satisfaction with SBES 2017 2019 2021 2023 Sample Size 70 50 50 46 Mean 8.8 9.2 9.1 8.8 Don't know/refused excluded from calculations Table 11: Satisfaction with Aspects of SBES 20 17 20 19...
AI summary Table 10 and Table 11 provide data on customer satisfaction with the Small Business Energy Solutions (SBES) program across different years, highlighting sample sizes, mean satisfaction scores, and feedback on various aspects of the program such as audit reports, financing, contractor quality, and rebate processes.
Table 12: Participant Suggestions to Improve SBES Participant Recommendations to Improve SBES 2017 2019 2021 2023 Sample Size 70 50 50 46 Offer financing/help with cost 1% - 4% 63% Offer more measures eligible for rebates 4% 10% 2% 15% Inc...
AI summary Table 12 outlines participant recommendations for improving the Small Business Energy Solutions (SBES) program from 2017 to 2023. The data shows a significant increase in the percentage of participants suggesting the need for financing options, with 63% in 2023 compared to 1% in 2017. Other recommendations include increasing rebates, improving marketing, simplifying the program, and enhancing communication.
4 SBES Impact Evaluation The objectives of the 2023 SBES impact evaluation were to determine gross and net electrical energy and peak demand savings.
AI summary The 2023 SBES impact evaluation aimed to assess gross and net electrical energy and peak demand savings, providing insights into the effectiveness of the Small Business Energy Solutions program.
4.1 Tracking Sheet Audit To ensure program component results were reliably compiled, the Evaluator first performed a tracking sheet audit aimed at verifying the completeness and consistency of the data submitted by EOne. The results obtain...
AI summary A tracking sheet audit was conducted to verify the completeness and consistency of data submitted by EOne, ensuring reliable compilation of program component results. The findings are detailed in Appendix VI, and the report refers to corrected tracked savings validated by the Evaluator.
4.2.1 Summary of Adjustments As part of the 2023 evaluation, the Evaluator conducted desk reviews complemented by on-site visits (n=42) to establish adjustment ratios and determine evaluated savings. In total, 25 lighting projects, 12 HVAC...
AI summary As part of the 2023 evaluation, the Evaluator conducted desk reviews and on-site visits to 42 projects to determine adjustment ratios for lighting, HVAC, and other measure categories. These ratios were used to revise savings estimates and extrapolate results for categories with acceptable margins of error.
4.3.2 Participant Spillover For SBES, spillover occurs when participants implement eligible energy efficiency measures due to the influence of previously participating in the program component without having received any kind of additional...
AI summary The evaluation of the SBES program found no spillover effects in 2023, consistent with previous years. The Evaluator used desk reviews, on-site visits, and phone interviews to assess spillover, and an algorithm is detailed in Appendix IX.
4.4 Realization Rate [Table](#page-2-0) 24 compares the tracked energy and peak demand savings values established through this evaluation to those calculated in the 2023 tracking sheet. It also includes the realization rate, representing t...
AI summary Section 4.4 discusses the realization rate, which is the ratio of evaluated net savings to tracked net savings for both energy and peak demand savings. This is compared to values from the 2023 tracking sheet in Table 24.
5 SBES Key Findings and Recommendations The main objectives of the 2023 SBES evaluation were as follows: - › Collect information on participant and partner perspectives - › Calculate gross and net SBES results, namely electrical first-year...
AI summary The 2023 SBES evaluation found that net energy and peak demand savings fell short of targets by 37% and 28%, respectively. Participation decreased by 27%, and free-ridership for DIY projects increased to 20%. The Evaluator made minor adjustments to savings calculations and recommended improving the CIS data entry process. Participant satisfaction remained high.
C. Challenges and Opportunities C1. In your experience, what are the most common concerns or participation challenges faced by potential participants in SBES that cause them not to participate? How do you address these concerns and challen...
AI summary The text outlines a series of questions posed to stakeholders regarding the Smart Building Energy Solutions (SBES) program, focusing on challenges faced by potential and registered participants, contractor availability, industry sectors with energy improvement potential, participant feedback, and ways to improve the program experience.
- 2. No [SAY "PERHAPS YOU CAN HELP ME ANYWAY." CONTINUE] Hello, my name is and I am calling from Narrative Research, a Halifax-based survey research company. We are performing an evaluation of energy efficiency services and programs provid...
AI summary A representative from Narrative Research is contacting a business that participated in Efficiency Nova Scotia's Small Business Energy Solutions Program to gather feedback on the program's effectiveness. The information will be used to evaluate and improve the program.
0-10 SCALE WITH END POINT LABELS– RECORD NUMBER - 98. (Don't Know) - 99. (Refused) - F3. [IF [F2<](#page-26-0)8] What is the most important reason you are not more satisfied with the program overall? [PROBE FOR SPECIFIC REASON. ACCEPT MULT...
AI summary This section of the regulatory proceeding document includes survey questions and response options related to customer satisfaction with the Small Business Energy Solutions (SBES) program. Respondents are asked to rate their satisfaction with various aspects of the program and provide recommendations for improvement.
C. Free-ridership for Audit Path C1. BEFORE learning about the Small Business Energy Solutions rebates and financing and having your facility evaluated by a Small Business Energy Auditor, had your business already decided to install the sa...
AI summary This section of the regulatory proceeding document addresses free-ridership in the context of the Small Business Energy Solutions Program, asking whether businesses had already decided to install energy-efficient upgrades before being evaluated by a Small Business Energy Auditor.
F2. Using a scale from 0 to 10 where 0 is "not at all satisfied" and 10 is "completely satisfied" how would you rate your satisfaction with the program overall?\ 2023 Sample Size 46 MEAN 8.8 Mean excludes responses of 'Don't know' and 'Ref...
AI summary The survey results indicate a high level of satisfaction with the program, with a mean score of 8.8 out of 10 in 2023, based on a sample size of 46 respondents. The mean excludes responses of 'Don't know' and 'Refused'.
F5. Do you have any recommendations for improving the Small Business Energy Solutions program? Anything else? 2023 Sample Size 46 Offer financing/help with cost 63% Offer more measures eligible to rebates 15% Increase rebates 9% Monitor th...
AI summary The Small Business Energy Solutions (SBES) program received feedback from 46 respondents in 2023. The majority (63%) suggested offering financing or help with costs, while other recommendations included expanding eligible measures, increasing rebates, improving application monitoring, enhancing marketing, and simplifying the program.
[VOLUNTEERED] - 98. Don't Know - 99. Refused - a. The number of steps required to complete the program - b. The time it took to get your application approved - c. The range of equipment that qualifies for incentives - d. The helpfulness of...
AI summary The text presents a series of survey questions aimed at assessing the experience of participants in the Small Business Energy Solutions program, focusing on program efficiency, support received, challenges faced, and areas for improvement.
[ASK FOR EACH RESPONSE IN E3A-I= 5-10 (RESPONDENT INDICATES SPILLOVER - THEY WERE INFLUENCED BY PROGRAM TO INSTALL EE EQUIPMENT WITHOUT REBATES)] - E4. Would it be ok if someone from the program evaluator, Econoler, called within the next...
AI summary The respondent indicates that they were influenced by a program to install energy efficiency equipment without receiving rebates. They are being asked if a program evaluator from Econoler can contact them for more specific questions about the equipment installed.
APPENDIX VI SBES: 2023 Tracking Sheet Audit This appendix presents the main results of the tracking sheet audit performed by the Evaluator, which was aimed at: - › Verifying that all data fields required for the evaluation were included an...
AI summary This appendix outlines the results of a tracking sheet audit conducted by the Evaluator to verify the completeness and accuracy of data submitted by EOne, ensuring consistency in parameters and calculation steps used to determine program results.
Sampling Methodology As part of the 2023 evaluation, the Evaluator conducted 42 desk reviews of SBES projects complemented by on-site visits at the facilities of participants. In total, 25 lighting projects, 12 HVAC projects, and five proj...
AI summary The 2023 evaluation involved 42 desk reviews and on-site visits for SBES projects, including 25 lighting, 12 HVAC, and five other measure projects. A stratified random sampling plan was used to reduce variance, based on a tracking sheet with 1,160 projects from 298 participants.
Table 1: FR – Audit Path C3. If your business had not received the free audit of your facility by a certified energy auditor, would you have identified the opportunity for the energy efficient upgrades that you installed? IF 1. Yes: EMPTY...
AI summary The audit path table examines whether businesses would have identified energy efficiency opportunities without the SBES program and includes inconsistency tests and free-rider calculations to assess program effectiveness and participation impact.
This appendix summarizes all the recommendations made by the Evaluator as part of the 2023 SBES evaluation. Section Recommendations Executive Summary Recommendation #1: Consider improvements to the Customer Information System (CIS), namely...
AI summary This appendix outlines two recommendations from the 2023 SBES evaluation: improving the Customer Information System (CIS) to handle missing entries and extract more data for validation, and implementing communication strategies for participants nearing the end of their project terms.
Evaluation Approach The evaluation was aimed at calculating program component results, namely new and total available DR capacity. [Table](#page-71-1) 1 summarizes the types of evaluation conducted for each program component and the corres...
AI summary The evaluation approach focuses on calculating program component results, specifically new and total available demand response (DR) capacity, with Table 1 summarizing the types of evaluation and corresponding methodology for each program component.
Table 1: Summary of Demand Response Program Evaluation Program Component Evaluation Type Methodology Process Market Impact Residential Demand Response Comprehensive › Tracking sheet audit › Water heater controller meter data analysis › Mea...
AI summary The document presents a summary of the Demand Response Program Evaluation, including methodology and performance metrics. It outlines evaluation components for residential and BNI Demand Response programs, and mentions participation levels and available capacity for each component.
Table 12: 2023 BNI DR Evaluation Approach Evaluation Objectives Research Questions Methodology Establish available DR capacity results for the DR Aggregator pathway › Are the data in the tracking sheet complete, accurate, and consistent? ›...
AI summary Table 12 outlines the 2023 BNI DR Evaluation Approach, focusing on evaluating the accuracy and completeness of data related to demand response (DR) capacity results for the DR Aggregator pathway. The methodology includes a tracking sheet audit, meter data analysis, project reviews, and follow-up interviews.
Note on Margin of Error For evaluation activities that yield quantitative results based on a sample, the Evaluator aimed to achieve a maximum margin of error of 10% at a confidence level of 90%. This means that, if measurements were conduc...
AI summary The evaluation of quantitative results from sampled data aimed for a 10% margin of error at a 90% confidence level. This margin of error reflects the precision of measurements, not their accuracy. For BNI DR projects, no margin of error was calculated as all projects were reviewed directly without extrapolation.
7.2.1 Meter Data Analysis and Project Reviews
AI summary The section discusses meter data analysis and project reviews, focusing on evaluating energy usage patterns and the effectiveness of various programs and initiatives. It highlights the importance of accurate data in assessing program performance and making informed decisions.
This document presents the detailed results of the tracking sheet audit performed by the Evaluator, which was aimed at: - › Verifying that all data fields required for the evaluation were included and filled out in the tracking sheet submi...
AI summary The document outlines the results of a tracking sheet audit conducted by the Evaluator to verify the completeness and accuracy of data submitted by EfficiencyOne. The audit focused on ensuring consistent calculation methods and verifying that tracked results aligned with previous evaluations.
This document presents the detailed results of the tracking sheet audit performed by the Evaluator, which was aimed at: - › Verifying that all data fields required for the evaluation were included and filled out in the tracking sheet submi...
AI summary The document outlines the results of a tracking sheet audit conducted by the Evaluator to ensure the accuracy of data submitted by EOne. The audit verified that all required data fields were included and that calculations were consistent with previous results. A table shows corrected available DR capacity, but no changes were made by the Evaluator, leading to no difference between the tracked and corrected values.
KEY DEFINITIONS Accuracy Reflects the proximity of measurements to the true value. Equipment life The number of years installed equipment will operate before it fails. Equivalent CO2 A unit of measurement indicating the amount of carbon di...
AI summary The text defines key terms related to energy efficiency and program evaluation, including accuracy, equipment life, equivalent CO2, equivalent effective useful life, evaluated savings, and evaluation plan. These definitions are crucial for understanding program performance and energy savings calculations.
E-32023 Savings Verification Review
32 passages
SAVINGS VERIFICATION REVIEW Of Efficiency Nova Scotia Program Year 2023 Evaluation Results Report for the Nova Scotia Utility and Review Board 07/26/2024 H. Gil Peach & Associates LLC H. Gil Peach, PhD John Mitchell, BS Yvonne J. Whitelaw,...
AI summary This document presents the 2023 evaluation results for the Efficiency Nova Scotia program, prepared by H. Gil Peach & Associates LLC for the Nova Scotia Utility and Review Board. It focuses on savings verification processes and program performance analysis.
Mission Statement With extensive experience in North America, we can provide the full range of evaluation, verification, policy, management, planning, regulatory adaptation services – wherever and whenever there is a need.
AI summary The organization emphasizes its extensive North American experience and offers a comprehensive range of services, including evaluation, verification, policy development, management, planning, and regulatory adaptation, available to meet needs wherever they arise.
H. Gil Peach & Associates LLC 16232 NW Oak Hills Drive Beaverton, Oregon 97006 USA E-mail: [[email protected]](mailto:[email protected]) Website: [www.peachandassociates.biz](http://www.peach...
AI summary H. Gil Peach & Associates LLC submitted a report titled 'Savings Verification Review of Program Year 2023 Evaluation Results' to the Nova Scotia Utility and Review Board, authored by H. Gil Peach and others, evaluating energy savings programs in July 2023.
I. How Savings Verification fits in the Policy, Planning, Program Cycle It can be useful in approaching evaluation to review how the Policy, Planning and Program cycle works and where evaluation and savings verification fit.[2](#page-6-3)...
AI summary The document outlines the Policy, Planning, Program cycle, emphasizing Savings Verification's role in evaluating Demand-Side Management (DSM) programs. Efficiency Nova Scotia administers DSM programs, succeeding Nova Scotia Power in 2010. Econoler, as the evaluator, conducts independent assessments and savings verification, ensuring data accuracy and methodological rigor.
II. Resource Acquisition and Other Evaluation Frameworks Efficiency Nova Scotia programs are almost entirely resource acquisition programs that treat saved energy as equivalent to generated energy. This is the original framework for the en...
AI summary Efficiency Nova Scotia's energy efficiency programs are evaluated as resource acquisition initiatives, treating saved energy as generated energy. Econoler, an international consulting firm, evaluates these programs. Market transformation frameworks, pioneered by BC Hydro and NUTEK, evolved to include code development and regulatory bans on inefficient practices.
III. Evaluation Standards In this section we highlight three standards for evaluation: independence, transparency, and the desirability of familiarity with current evaluation guidelines. - (1) Independent Evaluator In energy efficiency eva...
AI summary The section outlines three evaluation standards: independence, transparency, and adherence to current protocols. Independence requires an impartial evaluator to ensure credibility. Transparency mandates public access to reports and methods. Evaluation protocols align with the Uniform Methods Project and other guidelines. The Evaluator's 2023 process meets these standards, supported by Efficiency Nova Scotia's procedures.
IV. Summary of Evaluated Savings Results Evaluation of demand savings, first year energy savings, and lifetime net energy savings at the generator as reported by the Evaluator are summarized in this section.
AI summary This section summarizes evaluated metrics including demand savings, first-year energy savings, and lifetime net energy savings at the generator, as reported by the Evaluator. These results form part of the regulatory proceeding analysis.
Sector Contributions In Figure 4, net peak demand savings at the generator is split about 40% from BNI and about 60% from the Residential sector. 14 Taken as a whole, the multiplier for Residential programs is seventeen, the multiplier for...
AI summary The document discusses sector contributions to demand and energy savings, noting BNI and Residential sectors' roles. BNI accounts for 60% of net peak demand savings and 54% of lifetime energy savings, while Residential contributes 40% of peak savings and 46% of lifetime savings. Multipliers for Residential (17) and BNI (14) programs indicate average savings lifetimes.
VI. Savings Verification Approach The savings verification review was conducted as follows: - We focus on the "installed" annual energy savings and demand reductions. These are the annualized value of savings and demand reductions from the...
AI summary The savings verification process focuses on annualized energy savings from installed measures, reviews evaluation methods (interaction, free-ridership, spillover, net-to-gross), and includes 39 site visits for the 2023 review year. Calculations were not verified, but methodological approaches were assessed.
VII. General Findings - Each program impact evaluation conducted by the Evaluator is both competent and sufficient. The structure and format of each impact evaluation is consistent. This helps to maintain a high standard of practice. - Eac...
AI summary The Evaluator's program impact evaluations are competent and consistent, featuring executive summaries, methodological models, and proper citations. They appropriately developed energy savings and carbon offset analyses, demonstrating command of evaluation guidelines. Limited market evaluation was conducted.
VIII. General Recommendations 1. Savings Verification Recommendation No. 1: The Savings Verification study recommends acceptance of the 2023 evaluation results for energy savings and for demand-reduction for all programs with the understan...
AI summary The Savings Verification study recommends accepting 2023 energy savings and demand-reduction evaluation results for all programs, pending a literature review on smart thermostats for the 2024 Measure Assessment Update. This acceptance is conditional on the Evaluator's upcoming analysis.
IX. Individual Program Review There are fourteen individual programs. For Savings Verification, we focus on impact evaluation.
AI summary The section discusses the review of fourteen individual programs, emphasizing the focus on Savings Verification through impact evaluation.
recycled by ARCA Canada Inc., which is the delivery agent for the program. Eligible appliances must be in working condition and must be ten years of age, or older. Participants must own the appliances that will be retired. The program also...
AI summary The Appliance Retirement Program (ARET) recycled 4,260 appliances in 2023, a 19% decline from 2022, with most being refrigerators and freezers. The program did not meet its energy savings target, achieving 84% of the goal. Free ridership remains high, reducing effective savings by nearly 50%. Customer satisfaction remains high despite declining participation and savings.
2. Instant Savings Instant Savings is administered as an in-store discount program for energy-efficient products. The program is implemented through a delivery agent, Summerhill Group, Inc., and participating national and independent retai...
AI summary Instant Savings is an in-store discount program for energy-efficient products administered by Summerhill Group, Inc., with participation from retailers. The 2023 evaluation showed net energy and peak demand savings exceeded targets despite lower NTGR for LEDs. LED baseline shifts to 2025 will reduce program savings claims, though new product additions (e.g., ENERGY STAR dishwashers) offset market saturation.
3. Home Energy Assessment (HEA) Home Energy Assessment (HEA) is a component of the Existing Residential Program suite of offerings. This program encourages homeowners to increase the efficiency and comfort of their homes by providing rebat...
AI summary The Home Energy Assessment (HEA) program, part of the Existing Residential Program, offers rebates for energy efficiency retrofits and uses 'test-in/test-out' audits. A 2023 evaluation found 35% of 4,085 participants failed to complete post-retrofit assessments, with 54% of these still completing efficiency measures. The Evaluator recommends surveying unconverted D assessments to assess spillover savings.
The Evaluator found that Green Heat: - Achieved net electrical energy savings of 2.918 GWh, and 2.561 MW in net peak demand savings, although savings are less than planned net electrical energy savings of 3.629 GWh and net peak demand savi...
AI summary The Evaluator found that Green Heat achieved 2.918 GWh energy savings but fell short of planned targets. Participation dropped 11% in 2023, with customers preferring HEA/CGH over Green Heat due to higher incentives. Contractors reported program confusion, low incentives, and portal issues. Recommendations include improving marketing, website usability, communication, and increasing incentives for low-income households. The MSHP market is growing, with 40% of Nova Scotia households projected to use heat pumps by 2023.
5. Residential Efficient Product Installation (EPI) In 2023 the Evaluator conducted a condensed impact evaluation for the Efficient Product Installation (EPI) program, a component of EOne's Residential DSM Program. EPI provides free direct...
AI summary The EPI program, part of EOne's Residential DSM Program, provides free energy-efficient product installations. In 2023, eligible items included LED lamps, smart thermostats, and insulation. Funding comes from electricity ratepayer funds and the Low Carbon Economy Fund. A pilot for smart thermostats was paused due to installer withdrawal, and air sealing kits were no longer offered.
6. Mi'kmaw Home Energy Efficiency Program MHEEP provides energy efficiency upgrades to band-owned homes in the thirteen (13) Mi'kmaw communities in Nova Scotia, at no cost to the participant or the communities. Energy efficiency upgrades a...
AI summary The Mi'kmaw Home Energy Efficiency Program (MHEEP) provides no-cost energy efficiency upgrades to Mi'kmaw communities in Nova Scotia, funded by electricity ratepayers and the province. Evaluations show increased participation (38% rise in 2023) and energy savings (52% increase in gross energy savings), though net electrical energy savings fell short of targets. The program uses EnerGuide audits and collaborates with EOne and community partners.
8. Affordable Single-Family Homes (ASFH) The Affordable Single-Family Housing (ASFH) program was launched in 2023 and provides health and safety (moisture control) and energy efficiency retrofits (building envelope, appliances, and heat pu...
AI summary The Affordable Single-Family Homes (ASFH) program, launched in 2023, provides free retrofits for income-qualified homes in Nova Scotia. It partners with Housing Nova Scotia and uses EPI for additional measures. The first-year evaluation showed energy and demand savings below targets (59% and 30% shortfalls), attributed to new operations and lack of guidance. No recommendations were provided.
9. New Residential Program The New Residential Program has a single program component that is called the New Home Construction (NHC) program. This program provides incentives to builders and homeowners to construct new homes above the curr...
AI summary The New Residential Program's New Home Construction (NHC) component offers incentives for constructing energy-efficient homes. Participants register, undergo energy assessments, and receive rebates based on performance tiers. The 2023 evaluation showed significant energy and demand savings exceeding targets, though reasons for overachievement are unclear. The program achieved 5.281 GWh net energy savings and 1.909 MW peak demand reduction.
10. BNI Efficient Products Rebates (BER) This program serves the business, non-profit and institutional sector (BNI sector). The Efficient Product Rebates program operates the Business Energy Rebates (BER) program. This program has two com...
AI summary The BNI Efficient Products Rebates (BER) program offers rebates for energy-efficient products to the BNI sector via point-of-sale and mail-in applications. Qualifying products include LED lighting, sensors, and HVAC systems. 2023 data shows a 19% increase in point-of-sale rebates, driven by higher adoption of LED linear lamps and in-place fixtures.
11. BNI Custom Incentives Program (Custom) For 2023, the BNI Custom Incentives Program consists of two components, Custom and Strategic Energy Management (SEM). The Custom program is comprised of four components: Retrofit, New Construction...
AI summary The BNI Custom Incentives Program (2023) includes Retrofit, New Construction, Pay-for-Performance, and Building Optimization. Retrofit supports efficiency improvements, with rooftop solar contributing 13% of savings. 107 projects were included, with some incomplete. Measures included solar PV, refrigeration, HVAC, and limited building envelope improvements.
12. BNI Strategic Energy Management (SEM) Strategic Energy Management (SEM) is an approach for integrating energy management into business practice – so that a focus on continually advancing energy-efficiency becomes an integral aspect of...
AI summary Strategic Energy Management (SEM) integrates energy efficiency into business practices, inspired by Japanese Kaizen, and has expanded to include facilities like hospitals. In 2023, the EMIS program was incorporated into SEM.
13. BNI Small Business Energy Solutions Program (SBES) The BNI Direct Installation Program has a single program component, Small Business Energy Solutions (SBES). SBES is available to businesses that use less than 350,000 kWh annually. The...
AI summary The BNI SBES program offers two participation paths (audit and DIY) for small businesses. In 2023, 83% of participants used the DIY path, with 467 total projects—a 27% drop from 2022. Energy savings (7.451 GWh) and demand reduction (1.491 MW) fell short of targets (63% and 72% achieved), prompting mid-year target adjustments. The Commercial Direct Installation pilot remained in testing.
14. Demand Response In Q1 2023 Efficiency Nova Scotia integrated two pilot demand response (DR) programs that had been operating since 2021 into a single DR program offering. The DR program has two components, (1) Residential and (2) Busin...
AI summary Efficiency Nova Scotia integrated two demand response (DR) pilot programs into a single DR program in 2023, comprising residential and BNI components. Residential enrollments doubled previous pilot levels, but metering issues slightly reduced savings. BNI achieved high engagement but fell short of first-year targets, prompting over-enrollment adjustments due to budget constraints.
X. Recommendations from Fieldwork In multiple site visits for Program Year 2023 it was discovered that some confusion between similarly named programs and program delivery functions had occurred. Specifically, in regard to the Green Heat p...
AI summary During 2023 fieldwork, confusion between the Green Heat program and the Canada Greener Homes Grant (CGH) led to missed federal grants due to deadline misunderstandings. Most visits showed good workmanship, but some audited sites revealed patterns of reduced program/customer interaction, leading to miscommunication and lost delivery opportunities.
General Recommendations SVR-1 : The Savings Verification study recommends acceptance of the 2023 evaluated results for energy savings and peak demand reduction for all programs.
AI summary The Savings Verification study recommends accepting the 2023 evaluated results for energy savings and peak demand reduction across all programs. This endorsement supports the effectiveness of energy efficiency initiatives in Nova Scotia.
3. BNI Custom Incentives Program (Custom) SVR-4 : Contract for an extensive New Construction program process evaluation with a specific focus on recent declines in participation rates and potential geographic delivery biases. This evaluati...
AI summary The BNI Custom Incentives Program requires a process evaluation focusing on declining participation rates, geographic delivery biases, and logic model assumptions in a changing economic environment.
4. BNI Strategic Energy Management (SEM) SVR-5 : SEM program administrators must incorporate Pre/Post metering or some comparable form of submetering or EMIS system reporting as a robust form of energy data collection to validate savings v...
AI summary The SEM program requires administrators to implement Pre/Post metering or EMIS systems for accurate energy data collection. Additionally, a comprehensive survey of past SEM and Custom Retrofit participants is recommended to assess persistent spillover savings from prior program participation.
5. Small Business Energy Solutions (SBES) SVR-7 : Shifts in program delivery and promotion away from account manager interactions and the subsequent decline in Audit-path project numbers are negatively impacting program performance and sho...
AI summary SVR-7 highlights declining Audit-path project numbers due to reduced account manager interactions, advocating for regional contractors with performance-based reimbursements to boost participation. SVR-8 recommends a non-participant spillover survey to inform future rebate programs and identify additional measures.
rgy Laboratory. NREL/ SR-7A40-68562. [(http://www.nrel.gov/docs/fy17osti/68562.pdf)](http://www.nrel.gov/docs/fy17osti/68562.pdf). This version supersedes the 2013 and 2015 versions of the protocol. Dimetrosky, S.; Parkinson, K.; Lieb, N....
AI summary The text lists references to energy efficiency protocols, evaluation reports, and academic works on demand-side management and innovation diffusion. Key entities include the National Renewable Energy Laboratory (NREL), Energy & Resource Solutions, and authors like Dimetrosky and Rogers. Topics focus on energy efficiency evaluation methods and demand-side management frameworks.
XIII. Appendix: Questions to ask of DSM Evaluations - (1) Is the evaluator free of conflicts of interests? - (2) Does the independent evaluator have control over methods and measurement approaches? - (3) Is the evaluation oriented to the g...
AI summary This appendix outlines 25 questions to assess the quality of Demand Side Management (DSM) evaluations, focusing on transparency, methodological rigor, conflict of interest avoidance, data accuracy, and stakeholder engagement. Key areas include evaluation independence, documentation, statistical validity, and proper handling of energy data and simulations.