Topic:"Performance Monitoring" in M08604

Attachment 1, Page 1 of 20 Update on Implementation of 2013-2015 Evaluation Recommendations

AI summary This document provides an update on the implementation of recommendations from the 2013-2015 evaluation. It outlines progress made and any remaining actions required to address the findings of the evaluation.

or high delta EnerGuide. The Evaluator commends ENSC on this action and therefore recommends that the results of these reviews be tracked in the DSMDS. 2013 Evaluation Adapt the next tracking system (DSMDS) to the QA program: A QA program...

AI summary The Evaluator commends ENSC for adapting the next tracking system (DSMDS) to the QA program and recommends tracking results in the DSMDS. ENS agrees, noting that full implementation will occur in 2014. In the interim, results can be tracked on a spreadsheet with identification numbers for merging with the DSMDS and to support less experienced CEAs.

building commissioning a mandatory element of Custom NC and/or including a full energy simulation review to ensure the accuracy of the model inputs. 2014 Evaluation Adapt the next tracking system (DSMDS) to the QA program: A QA program PP-...

AI summary The text discusses the adaptation of the DSMDS tracking system to a QA program by ENSC in 2014. ENS agrees with the recommendation, noting that the QA process will involve reviewing simulation results and tracking outcomes. The Evaluator suggests using a spreadsheet for interim tracking and including CEA identifiers for better support and training.

results should include the identification of the CEAs who conducted each assessment to provide more support or training to CEAs with less experience. 2015 Evaluation Continue working towards the standardization of measurement and Custom-R2...

AI summary The text discusses the need for standardizing measurement and verification (M&V) procedures for Custom Retrofit programs. ENS has taken steps to standardize its M&V policy and improve documentation, but inconsistencies and incomplete documentation remain a concern. The Evaluator recommends that M&V strategies be included in the Project Development Agreement (PDA) and that all relevant documentation be stored in the DSMDS.

DEFINITIONS Accuracy Reflects the proximity of measurements to the true value. The ratio of evaluated results to tracked results. This ratio expresses the Adjustment ratio adjustment made to tracked savings or other tracked values such as...

AI summary The text defines key terms related to energy efficiency measurement and verification, including accuracy, base case, displaced wattage, and EnerGuide ratings. These definitions are critical for evaluating program effectiveness and ensuring reliable energy savings calculations in regulatory proceedings.

rtain period before being installed, such as LED bulbs. The energy savings that occur over the lifetime of an energy efficiency Lifetime energy measure. Lifetime energy savings account for a measure’s effective useful savings life and any...

AI summary The text defines key terms related to energy efficiency program evaluation, including lifetime energy savings, line loss factor, net energy savings, and peak coincidence factor. These metrics are critical for measuring program effectiveness, accounting for system losses, market effects, and energy savings attribution.

Efficiency Nova Scotia Executive Summary The demand savings that coincide in time with the expected annual peak Peak demand savings demand period of the entire electricity system. Precision The extent to which repeated measurements agree w...

AI summary The document defines key terms related to energy efficiency programs, including peak demand savings, random sampling error, tracked savings, and unitary savings. It emphasizes metrics for evaluating program effectiveness and measurement accuracy in demand-side management initiatives.

TABLE OF CONTENTS INTRODUCTION ................................................................................................................................... 1 1 EVALUATION OBJECTIVES AND SCOPE ...........................................

AI summary The document outlines the structure of an evaluation report, detailing objectives, methodology (including interviews, surveys, on-site visits, and analysis techniques), and impact evaluation results. It focuses on assessing program effectiveness through data collection and analytical methods.

Errors arising during the course of all survey activities other than sampling. Non-sampling error Non-sampling errors tend to lead to bias in measurements and are very difficult to quantify. Peak coincidence The percentage of measure deman...

AI summary The text discusses various statistical and measurement concepts in energy efficiency programs, including non-sampling errors, precision, sample size, and tracked savings. It also defines terms such as peak coincidence factor, unitary savings, and secondary market impacts, which are relevant to energy efficiency tracking and evaluation.

ndation, if properly implemented, will improve the results and facilitate identifying major equipment changes made between any two reporting dates. The Adjustments Made to Tracked Incremental Savings Incremental energy savings were tracked...

AI summary The document discusses adjustments made to tracked incremental savings for four EMIS projects. Projects 2 and 3 required recalculations by the Evaluator, resulting in an energy adjustment ratio of 0.89, while Projects 1 and 4 did not require adjustments as the service provider's calculations aligned with the participants’ processes.

. Altogether, these services generated 21.005 GWh in net energy savings and 3.637 MW in net peak demand savings at the generator in 2017, with 78 projects generating savings and 58 projects completed. The 2017 evaluation has included a tec...

AI summary The 2017 evaluation of energy efficiency services in Nova Scotia showed 21.005 GWh in net energy savings and 3.637 MW in net peak demand savings. Improvements in documentation led to a closer adjustment ratio for Retrofit services, though some projects had insufficient M&V activities. Overall, adjustments reduced ENS’s tracked value by 9 percent, with positive feedback on the Retrofit service and BDM’s support.

g eligibility for your customers. 1 What problems or issues, if any, have you faced? 2 What has worked well for you? 3 What problems or issues, if any, have your customers faced? Project No. 6138 6 Custom Incentives Program Efficiency Nova...

AI summary The document outlines a series of questions related to the Custom Incentives Program under Efficiency Nova Scotia, focusing on the evaluation of the 2017 DSM (Demand Side Management) initiative. It explores challenges and successes at various stages, including scoping studies, feasibility studies, implementation, and the M&V (Monitoring and Verification) phase.

DEFINITIONS Accuracy Reflects the proximity of measurements to the true value. The ratio of evaluated results to tracked results. This ratio expresses the Adjustment ratio adjustment made to tracked savings or other tracked values such as...

AI summary The text defines key terms related to energy efficiency measurement and verification, including accuracy, base case, displaced wattage, and EnerGuide ratings. These definitions are critical for evaluating program effectiveness and ensuring reliable energy savings calculations in regulatory proceedings.

standing of roles and responsibilities; adequate resources and support for marketing staff; and effective communication among marketing team members and between the marketing team and other ENS staff. Indicators: Marketing staff and progra...

AI summary The text outlines ENS's focus on ensuring marketing staff understand their roles, have adequate resources, and maintain effective communication. It emphasizes indicators like staff adherence to brand guidelines, workload management, and data collection methods such as IDIs and Google Analytics to evaluate program implementation effectiveness.

Marketing plan and campaign materials reviews; IDIs with ENS marketing staff and program managers; review of Google Analytics metrics; marketing tracking system and program databases. ENS’s long-term expectations for this activity are the...

AI summary ENS evaluates marketing strategies, emphasizing integrated delivery, brand consistency, and responsive campaign adjustments. Key indicators include coordinated materials and staff effectiveness. Tracking systems like Basecamp are used for campaign management, with data collected via IDIs and analytics reviews.

; users provide feedback and receive prompt replies to their system/process update requests. Data sources and data collection methods: IDIs with marketing staff, and program managers. As marketing staff and others gain more experience with...

AI summary Efficiency Nova Scotia (ENS) outlines short-, medium-, and long-term goals for tracking systems and processes used in marketing activities. These systems aim to improve cross-program collaboration, adapt to market changes, and streamline reporting. Success indicators include user satisfaction, system effectiveness, and accurate data collection via In-depth Interviews (IDIs) and report reviews.

communicating with the creative agency and overseeing project execution. ENS also has a digital marketing specialist dedicated to building sophisticated digital analytics capabilities and dashboards. The communications team operates simila...

AI summary ENS merged its marketing and communications teams, enhancing collaboration and cohesion. The marketing team includes a digital specialist, while the communications team focuses on stakeholder engagement and educational initiatives. The merger was positively received, though full impacts are still emerging. Research Into Action evaluated the marketing team but not the communications team.

Indicator Table—Enhance/Develop Marketing Communications Tracking Systems Page C-1 Marketing Evaluation Short-Term Outcomes Indicators Data Sources & Collection Approaches 1. Relevant staff find tracking • Tracking systems allow easy data...

AI summary This document outlines the short-term outcomes and indicators for enhancing and developing marketing communications tracking systems. It emphasizes the importance of user-friendly systems, regular usage by staff, and the ability of systems to capture and store necessary information for marketing efforts.

] Q25. Do you typically have one key contact at each customer’s organization, or do you work with more than one individual? How does this vary for local versus international organizations? Q26. Is/are the key contact/s typically in decisio...

AI summary The text consists of a series of questions posed to Business Development Managers regarding customer interactions, data management, and challenges and successes of the BD team. The questions focus on customer engagement strategies, key contacts, support needs, customer satisfaction, and the impact of new systems on communication and workflow.

CA will be provided by EfficiencyOne to NS Power. It incorporates the requirements of the Act, with the exception of the term of one year in accordance with the EPIA, which provide that the Agreement: (a) Not be terminable or terminated un...

AI summary EfficiencyOne will provide a contract to NS Power under the EPIA, with terms excluding a one-year termination clause. The 2019 DSM targets align with the 2016-2018 plan, incorporating quarterly performance reporting on energy savings and spending. Pilots and stakeholder collaboration are emphasized in the 2019 plan.

he EECA 38 without the prior written approval of NSPI. 8 Supply Agreement 1 10.2 Where EfficiencyOne subcontracts any part of the EECA, EfficiencyOne shall preserve 2 and protect the rights of the Parties under this Agreement with respect...

AI summary The agreement outlines EfficiencyOne's obligations to protect NSPI's rights when subcontracting EECA work, ensuring confidentiality of personal information, and indemnifying NSPI against misuse. It emphasizes responsibility for subcontractor actions and compliance with CASL.

ion by EfficiencyOne contrary to the terms of 20 this Agreement or the Act, or in consequence of EfficiencyOne's contravention of CASL. 21 12. PERFORMANCE REQUIREMENTS AND EVALUATIONS 22 12.1 EfficiencyOne's performance under the terms of...

AI summary The text outlines contractual obligations between EfficiencyOne and NS Power under the Electricity Efficiency and Conservation Act, including performance evaluation requirements set by the UARB, force majeure provisions, and compliance with CASL. It emphasizes adherence to legal standards and mitigation of disruptions.

47 Performance Requirements 48 I. UARB-APPROVED PERFORMANCE TARGETS, THRESHOLDS, AND 49 INDICATORS 50 51 a) Performance Targets and Thresholds: 52 i. EfficiencyOne is deemed to be in substantial compliance with the UARB- 53 approved Plan P...

AI summary The document outlines UARB-approved performance targets, thresholds, and indicators for EfficiencyOne. It specifies that 90% achievement on two key performance targets—incremental annual net energy savings and peak demand savings—is required for substantial compliance. If targets are not met, a regulatory process will be initiated. Additional indicators include lifetime savings, ratepayer benefits, spending, customer satisfaction, and low-income program metrics.

rcentages, the top approximately 40% of DSM portfolio programs produce 80% of net first-year energy savings. 7 Page Verification Review of Program Year 2017 Evaluation Results indicated in Figure 4, both as a multiple of first-year net ene...

AI summary The top 40% of DSM portfolio programs generate 80% of first-year net energy savings, per evaluation results. Table 2 quantifies 2017 net energy savings at the generator, with Figure 4 illustrating multiplier effects on savings. The text focuses on verifying program performance metrics.

Direct Small Business Energy Solutions 7.887 7.887 Installation Installation BNI Subtotal 76.877 Overall DSM Portfolio Total 130.749 Codes & Standards 21.914 Total (DSM Portfolio plus Codes & Standards 152.663 Table Notes Due to year to ye...

AI summary The 2017 DSM program evaluation highlights energy savings contributions from Business Energy Rebates, Custom, Instant Savings, and Residential Efficient Product Installation, accounting for 73% of total net lifetime GWh. The report notes no SEM results in 2017 due to timing, with data sourced from Table 5 of the 2017 DSM Program Evaluation Executive Summary.

, 2014-2016 Assessment of Automated M&V Methods, Summary of Work (http://eis.lbl.gov/auto- mv.html). 23 Some vendors will claim that they have proprietary and confidential methods based on engineering models perhaps combined with big data...

AI summary The text critiques proprietary methods in automated M&V (Monitoring and Verification), emphasizing transparency and validation. It warns against over-reliance on algorithmic approaches in 'Evaluation 2.0,' which may compromise accuracy and invite misstatements of energy savings. Physical measurements and evaluator independence are stressed as critical, with the 2017 Econoler evaluation cited as a transparent example.

of the evaluator, including the evaluator’s independence in control of methods of analysis for evaluation. The 2017 Econoler evaluation has no problems of this kind. 3. Transparency DSM evaluation plans and reports are made public and are...

AI summary The evaluation of the 2017 DSM program emphasizes the independence of evaluators, transparency in reporting, and the use of a Technical Resource Manual (eTRM). Efficiency Nova Scotia’s eTRM is noted as a technical advance, while Econoler’s independent reviews are highlighted to prevent circularity in energy savings calculations.

t are the savings that should be realized when each of the projects operates for a full year.] These numbers come from the tracking system and from the evaluator’s adjustments. • Generally, we did not check the evaluator’s mathematical cal...

AI summary The evaluation of Efficiency Nova Scotia programs in 2017 is deemed comprehensive and methodologically sound, with Econoler's analyses adhering to industry frameworks. Key findings highlight the value of lifetime net energy savings assessments at the generator level, enhancing the understanding of program impacts. The review focused on methodological rigor rather than mathematical recalculations, except in limited cases.

tically rely on the eTRM, though sometimes values from the eTRM may be used. The evaluator independently is responsible for unitary savings values and effective measure lives. The use of a Technical Resource Manual (here an electronic TRM...

AI summary The text emphasizes the importance of independent evaluation to avoid circularity when using the eTRM for savings calculations. Econoler is credited with independently verifying unitary savings values and effective measure lives, ensuring evaluation integrity. The Savings Verification study recommends accepting 2017 evaluation results for all programs.

lot project, appliances owned by the Halifax Regional School Board (HRSB) were audited and collected by the Appliance Retirement Program. Savings from these two additions are included in this program. The 2017 evaluation used documentation...

AI summary The 2017 evaluation of the Appliance Retirement Program included audits of appliances owned by the HRSB, visits to recycling facilities, and improvements to metering protocols. Net energy and peak demand savings were found to be higher than previously reported, with recommendations for continued improvements in metering and tracking processes.

g e Verification Review of Program Year 2017 Evaluation Results Incentives are based on simple cost payback periods and are tailored to meet the situation of individual customers. Custom has three primary components: retrofit, new construc...

AI summary The verification review of Program Year 2017 evaluation results discusses incentives based on cost payback periods and program components such as retrofit, new construction, and building optimization. Econoler conducted reviews and found issues with M&V data and made adjustments based on past trends and assumptions.

ding participant phone interviews). This included review of updated annual reports and savings calculation workbooks as well as baseline reviews. Free- ridership and spillover were assumed to be zero. Econoler recommends that Efficiency No...

AI summary Econoler recommends that Efficiency Nova Scotia improve tracking and documentation of energy savings, including the use of baseline regressions and M&V, and ensure that incremental savings are tracked rather than cumulative savings. These measures are deemed necessary to ensure accurate savings calculations and better capture energy savings.

405.9 GWh 62.4 MW 90% 80% 80% 23.7 MW 70% 66% Percent of 2016-2018 Target 130.7 GWh 60% 50% 40% 26.0 MW 30% 136.9 GWh 20% 10% 0% Energy savings (2016-2017 Energy savings (2016-2018 Peak demand savings (2016- Peak demand savings (2016- actu...

AI summary The text presents energy savings and peak demand savings data for 2016-2018, highlighting progress toward targets. It mentions that 2016 savings are verified, while 2017 savings are evaluated and pending verification. The data is presented in percentages and gigawatt-hours.

s responsibility for service relationships with the people and institutions that ENSC serves? What are the restrictions on scope and to what extent are they needed or unnecessary? How and how well does ENSC maintain good faith and quality...

AI summary The text raises questions about ENSC's service quality, organizational culture, and employee experiences, including concerns about good faith service, unintended consequences, work environment, career opportunities, and the presence of cynicism or irony within the organization.

NON-CONFIDENTIAL 1 Request IR-03: 2 3 Page 14 rows 10 through 13 of the Evidence states, “In the context of the development of the 4 2019 DSM Resource Plan as a continuation plan, EfficiencyOne is proposing largely the same 5 performance t...

AI summary EfficiencyOne clarifies that the 2016-2018 DSM Resource Plan did not include total spending or ratepayer benefits as performance targets but as indicators. Savings from pilot programs (e.g., demand response) are included in 2019 targets, though lifetime GWh savings are not yet a target despite improved tracking capabilities.

NON-CONFIDENTIAL 1 Request IR-04: 2 3 Please refer to Table 2 – Proposed Performance Indicators and Targets on page 15 of the 4 Evidence. 5 a) Please explain why EfficiencyOne does not propose 2019 Performance Targets for 6 lifetime saving...

AI summary EfficiencyOne's response to IR-04 explains that 2019 DSM targets are a continuation of 2016-2018 plans, maintaining existing performance metrics aligned with a resource acquisition framework. It emphasizes consistency with NS Power's mandate and rejects new 2019 targets for lifetime savings and customer satisfaction, citing continuity rather than new proposals.

NON-CONFIDENTIAL 1 Lifetime energy savings, total ratepayer benefits, total spending, and customer 2 satisfaction provide valuable context and insight into how ENS is performing, but 3 should be categorized as Performance Indicators as opp...

AI summary ENS argues that lifetime energy savings, ratepayer benefits, spending, and customer satisfaction should be Performance Indicators, not Targets, due to methodological adjustments, correlation with other metrics, budget constraints, and limited ex-post measurement feasibility. EfficiencyOne supports including lifetime savings in the next three-year plan but not as a Target yet.

does not reflect achievement of the DSM Plan 21 objectives in the same capacity as the two proposed Performance Targets. 22 23 b) No, these metrics were proposed and approved as Performance Indicators. 24 25 c) Confirmed – the value of 127...

AI summary EfficiencyOne clarifies that the DSM Plan's objectives are not met at the same level as proposed Performance Targets. Metrics were approved as Performance Indicators, with specific targets of 127.2 GWh for energy savings and 20.2 MW for demand savings. EfficiencyOne manages its portfolio to meet UARB-approved targets and adheres to the 2016-2018 Consensus Agreement requiring explanations for significant variances.

Failure to submit Weekly Risk Reports may result in progress payments being withheld from the Consultant until such time that the Weekly Reports are submitted in a timely and accurate manner. COMPLETING THE REPORT When the Consultant encou...

AI summary The document outlines the requirements for submitting Weekly Risk Reports by the Consultant to EfficiencyOne, emphasizing the consequences of non-compliance, such as withheld progress payments. It also details the evaluation criteria for the Consultant's performance based on the accuracy and timeliness of these reports.

.................................... ................................... 6 6 5. NOTIFICATION OF SIGNIFICANT CHANGES..................................................................6 7 6, SAFETY.................................. .............

AI summary The text outlines contractual clauses covering notification of changes, safety, property protection, environmental compliance, EfficiencyOne's covenants, subcontractor management, confidentiality, performance requirements, force majeure, indemnity, liability limits, and insurance provisions.

ts and omissions of Subcontractors and of persons directly or indirectly 5 employed by them, as if they were acts and omissions of persons directly employed by 6 EfficiencyOne. 7 10.3 Nothing contained in the Contract Documents shall creat...

AI summary The contract outlines EfficiencyOne's obligations regarding subcontractor liability, confidentiality of NSPI's personal information, performance evaluation under UARB guidelines, and force majeure provisions.

gets 127.2 j 20.2 _ _ 11 12 13 The Performance Targets will be achieved in accordance with the EECA Plan as set out 14 in Schedule E. 15 16 For certainty, in accordance with the performance requirements set cut in Schedule “C” attached her...

AI summary The document outlines performance targets under the EECA Plan, requiring EfficiencyOne to achieve 90% or more on both performance targets for substantial compliance. If this threshold is not met, a regulatory process will be triggered. Schedule B details compensation aspects of the 2019 DSM Plan filing.

Supply Agreement for Compliance Filing 45 SCHEDULE C 46 47 Performance Requirements 48 1. UARB-APPROVED PERFORMANCE TARGETS. THRESHOLDS. AND 49 INDICATORS 50 51 a) Performance Targets and Thresholds: 52 i. EfficiencyOne is deemed to be in...

AI summary This document outlines the performance requirements for EfficiencyOne under the UARB-approved Plan, including targets and indicators for energy savings, peak demand reduction, ratepayer benefits, and customer satisfaction, with specific reporting obligations and triggers for regulatory processes.

Request IR-4: - 32 Please refer to Table 2 - Proposed Performance Indicators and Targets on page 15 of the - 33 Evidence. - 34 35 36 a. Please explain why EfficiencyOne does not propose 2019 Performance Targets for lifetime savings, total...

AI summary Request IR-4 questions EfficiencyOne (E1) about missing 2019 performance targets for metrics like lifetime savings and customer satisfaction, whether prior years had such targets, and consequences for unmet thresholds. Request IR-5 demands data on program performance (lifetime benefits, TRC/PAC results) and explains assumptions in the 2019 DSM Resource Plan compared to 2016-2018 plans.

ts and omissions of Subcontractors and of persons directly or indirectly 5 employed by them, as if they were acts and omissions of persons directly employed by 6 EfficiencyOne. 7 10.3 Nothing contained in the Contract Documents shall creat...

AI summary The agreement outlines subcontractor liability, confidentiality obligations, performance evaluation criteria under the UARB, and force majeure provisions. EfficiencyOne is responsible for securing NSPI's personal information and indemnifying NSPI for breaches, including CASL violations. Performance is measured via UARB-established metrics in Schedule C.