Comments on 5/13 call

1. What baseline improvements would increase performance, entry, and demand flexibility innovation?

The California Efficiency + Demand Management Council (“Council”) and Leapfrog Power, Inc. (“Leap”) appreciate this opportunity to comment on the discussion from the May 13 Demand and Distributed Energy Market Integration (“DDEMI”) Working Group.

The Council and Leap appreciate the presentation given by the Department of Market Monitoring (“DMM”) regarding its concerns about demand response (“DR”) baselines.  Unfortunately, it posed several questions without providing suggested solutions.   

The Council and Leap agree with DMM that, to the extent that existing DR baselines are inadequate in some way, the first option should be to modify them rather than simply create new ones.  However, DMM has not adequately explained the risk incurred by retaining the current number of DR baselines, or even expanding their number should it be necessary.  In the absence of any evidence (e.g., additional CAISO operational costs, diminished DR baseline accuracy, etc.), DMM’s concern appears to be a solution looking for a problem.  Also, when considering whether to reduce the number of DR baselines, the CAISO should consider the significant amount of effort and time that is required to create new DR baselines and get FERC approval (if DR baselines are not moved out of the CAISO Tariff and into a Business Practice Manual) should it be discovered that some eliminated baselines could have been used in the future.  In any event, any consideration of eliminating DR baselines should wait until the very end of the DDEMI process because only then will the updated DR baseline landscape be apparent.

As a general principle, good DR baselines will increase performance, entry, and innovation.  On a conceptual level, the primary role of DR baselines is to ensure accurate performance measurement of DR resources.  As such, well-designed baselines will accurately measure DR performance.  In turn, accurate measurement will encourage good DR performance (by ensuring compensation commensurate with performance).  Additionally, well-designed baselines will attract entry by accurately accommodating the inherent heterogeneity among participants.  This heterogeneity can be reflected in frequency of dispatch, weather-dependence, and technology type.  Similarly, well-designed baselines can help to facilitate innovation by ensuring accurate performance measurement of new technologies (e.g., the growing range of direct-controlled smart devices and BTM energy storage) and use patterns. 

One helpful exercise may be to assess current baseline gaps by explicitly identifying the various use categories (which would be defined by factors such as customer type, underlying technologies, dispatch frequency, etc.) and assessing whether the existing suite of baselines addresses all of these needs. 

In the meantime, the Council and Leap offer some thoughts on specific baseline improvements. 

Device-Level Measurement to Increase Performance, Entry, and Demand Flexibility Innovation: The Council and Leap continue to support device-level measurement as a measure that, though it is not a baseline per se, can lead to increased performance, entry, and demand flexibility innovation.  The Council and Leap discussed the potential benefits of device-level measurement at the April 7 DDEMI Working Group but reframes those most relevant to the three criteria highlighted in this question:

• Increase Performance: Faster and more accurate measurement of load curtailment by eliminating reliance on the IOUs to provide customer meter data
• Entry: Potential to dramatically increase DR participation in the CAISO market by eliminating the requirement for customers to go through the IOU Share My Data process in order to enroll with a third-party DR providers
• Demand Flexibility Innovation: Allow multiple providers to serve devices across a premise, increasing customer choice of DR providers and also allowing each device to be dispatched during the hours of the day when they can provide the greatest benefit to the grid

The benefits of device-level measurement outlined above underscore the importance of moving forward with Problem Statement #5, which CAISO’s May 13 presentation indicated that they were seeking stakeholder feedback on.  The Council and Leap believe that this problem statement is important to include, and that its phrasing in the May 13 presentation is appropriate.  The ability to register specific devices in the DRRS, as opposed to the customer-level meter, is important, and will become increasingly important as more and more homes end up with multiple devices behind-the-meter, most of which are controlled by different entities.  Furthermore, the authorization process for sharing meter-level data constitutes a significant barrier to third-party participation with CAISO.  As PG&E states in a recent reply to its Advice Letter 7577-E, the requirement for customers to complete the Share My Data authorization, as well as to provide their PG&E login information and account number, creates a notably more arduous enrollment process for customers.  Problem Statement #5 critically addresses the fact that customers registering at the service account level (via utility data sharing procedures) hinders enrollment in wholesale-integrated DR programs, and that developing alternative processes for device-level registration would substantially increase DR resources’ entry into the market.

Universal Access to Control Groups for Increased Dispatch Frequency: Today, most DR resources participating in the CAISO market use 5-in-10 or 10-in-10 baselines, comparing a customer’s actual load during an event to their recent load profiles.  While generally effective for users with stable load patterns and infrequent dispatch, this method is inadequate for the new generation of more frequently-dispatchable DR which uses thermal and electric energy storage, electric vehicles (“EVs”), and/or directly-controlled smart devices.  These technologies can dispatch more frequently, but under current measurement methods, frequent dispatching distorts baseline calculations, creating challenges for DR to dispatch in the CAISO market with greater regularity.  One potential solution would be the greater use of control group baselines because the constituent customers are not impacted by frequent DR dispatch.  However, in addition to the problem described by Pacific Gas and Electric Company (“PG&E”) that is created by requiring control group participants to be registered in the Demand Response Registration System (“DRRS”), there are significant barriers to third parties accessing large-scale non-participant data, as described in the response to Question 2 below.   

A middle-ground approach to consider would be the use of “prescriptive baselines.” This approach would introduce a new type of “control group” methodology in CAISO, but one where the control group is built in advance using historic customer load data rather than constructed in response to individual DR events.  Prescriptive baselines have been used in the CEC’s Demand Side Grid Support (“DSGS”) program since summer 2023 and discussed in detail by Leapfrog Power in its presentation at the March 3 DDEMI Working Group.  In the DSGS Incentive Option 3 participation model for residential batteries, the CEC established a static or “prescriptive” baseline for all batteries participating in the program.  This prescriptive baseline is based on the average battery usage for residential battery customers in California, and it is updated every two years to ensure it accurately reflects current customer battery use patterns.  If used for Proxy Demand Resources (“PDRs”), this baseline methodology would allow batteries to be dispatched as frequently as they are able to supply energy to the grid, calculating their performance using average battery demand as a counterfactual.  This would streamline DR performance calculations while maintaining accuracy, because at the aggregate level, a resource composed of hundreds or thousands of batteries should have a counterfactual demand close to the overall state average.

CAISO already allows for DR performance by electric vehicles and distributed batteries to be measured using device-level data (via the EVSE and MGO baseline methodologies, respectively).  It would be relatively simple for CAISO to allow customers using these baseline methodologies to measure their performance against average energy use for EVs and batteries in California, which are relatively uniform within customer types.  Since the CEC already has experience constructing these types of baselines for the DSGS program, it could support the development and subsequent updates to prescriptive baselines in wholesale markets, potentially as part of its Integrated Energy Policy Report that it submits to CAISO on a biannual basis.

This approach could also be expanded to other technologies (e.g., thermal energy storage, smart thermostats, and heat pump water heaters) and customer types.  Because prescriptive baselines are set up using historic load data, it is possible to design finely-tuned comparisons based on a number of conditions.  For example, a household’s response to a DR dispatch on a 100-degree day would be assessed against the average load of similarly-sized customers at the same temperature over previous years.

2. Why are the control group and weather matching baselines not widely used?

The largest challenge with using control groups from the perspective of third-party DR providers is the absence of access to non-participant customer data.  As for-profit businesses, DR providers cannot afford to allocate significant budget to recruiting customers whose sole purpose would be to act as a control group.  Every enrolled customer is a source of revenue, so their load curtailment must be utilized to the greatest extent possible.  From the IOU perspective, PG&E’s March 3 Proposal to Revise Demand Response Control Group Settlement Methodology indicates that another barrier to control group use is the required enrollment of these customers in the DRRS.

As described by a stakeholder during the May 13 DDEMI Working Group, weather-matching baselines can be problematic due to the difficulty of finding comparable weather conditions when a DR event occurs during extreme heat.  Also, whenever comparable weather conditions do occur within the same time frame as a DR event, it is likely that DR resources were dispatched on that day as well.  From an operational standpoint of a DR provider, the 90-day lookback period is sometimes prohibitively long to manage for large numbers of customers and individual resources.

1. What baseline improvements would increase performance, entry, and demand flexibility innovation?

Marin Clean Energy (MCE) appreciates the opportunity to comment on the questions presented by the Department of Market Monitoring during the May 13th Demand and Distributed Energy Market Integration (DDEMI) Working Group meeting. As discussed in MCE’s response to question two, the following improvements to the control group method would increase performance, entry, and demand flexibility innovation:

• Removing or modifying the requirement to register non-participant end users for load-serving entities (LSEs) with existing access to non-participant data;
• Facilitating use of standardized load profile data sets for control group formation;
• Addressing other prohibitive barriers such as requiring 20 pre-event days and the use of a new control group for each event; and
• Allowing for limited and data-based adjustments to control group baselines to account for known differences between the control and treatment group.

Streamlining an LSEs’ use of its customer data or enabling use of standardized load profiles for control group formation would significantly reduce computational barriers, lower costs for market entrants, and accelerate scaled deployment of innovative demand flexibility programs.

Additionally, MCE encourages the CAISO to consider soliciting proposals for, and authorizing, pilot programs to test alternative baseline approaches to the control group methodology. Pilots could be implemented in the near term by continuing to settle on an existing approved performance evaluation methodology (PEM) while simultaneously evaluating performance using alternative baseline techniques. MCE is currently developing an alternative baseline method and encourages the CAISO to authorize a pilot program that would allow for testing MCE’s alternative baseline in settlement. MCE’s approach is being designed to enable daily signal participation and enhance the performance value by rewarding more frequent load-shifting, and is further discussed below.

2. Why are the control group and weather matching baselines not widely used?

MCE has no comment on the weather matching baseline at this time, nor can MCE speak to why the baselines are not widely used by market participants at-large. Rather, MCE’s comments are focused on the barriers that MCE has encountered in employing the control group method.

MCE’s primary barrier with the control group method is captured in performance evaluation methodology (PEM) problem statement six, which states that the “[r]equirement for control group end users to be registered in the Demand Response System limits use of non-participating end users within a control group and is in conflict with consumer data privacy rules.”^[1] The requirement to register non-participant end users for use in control group analysis significantly slows and complicates the process. MCE believes this likely contributes to overall low utilization of the control group method despite its known advantages.^[2] To register a non-participant end user, the user must “opt-in” to having their data used in the Demand Response Registration System (DRRS). This is an inhibitory and unnecessary step.

As the user won’t be participating in or benefiting from the demand response (DR) program, there is no reason or incentive for them to opt-in. Further, as an LSE, MCE already has access to the requisite non-participant customer data – data which MCE already uses in CAISO settlements. Removing the registration or opt-in requirement for LSEs with existing access to non-participant data would lead to increased utilization of the control group method.

MCE recognizes that the CAISO needs to otherwise verify the customer exists and is not registered in another DR program. MCE encourages exploration of other mechanisms to conduct those verifications, such as by exploring the solutions offered by PG&E in their comments on the April 7th DDEMI working group meeting. MCE also recognizes that access to non-participant data for third-party demand response providers (non-LSEs) is an issue, but MCE believes this is outside the scope of the DDEMI working group and is currently being addressed at the California Public Utilities Commission.^[3]

The registration requirement also forces a program to be designed in one of two ways. The first would be to create an internal control group by sending no signals to a group of participants. However, this decreases the value of program participation and decreases the value of the program as a whole. The other option would be to conduct the costly and arduous process of registering enough non-participant end users in the DRRS to create a pool of control group participants that is sufficiently larger than your participant group. Once enough control group participants are registered, next there are significant and costly computational requirements to validate and settle on the control group method. It requires significant resources to store all the data used for control group formation and all the calculation logs for generating baseline performance and comparison to the treatment group.

MCE agrees with the premise behind the prescriptive baseline concept proposed by Leap.^[4] Access to statewide, standardized load profile data sets^[5] for control group formation would both circumvent the registration issue and reduce the computational power required to utilize the control group method. However, unlike Leap’s proposal for state-level baseline development, MCE believes that the state should create standardized state-level load profiles, but that demand response providers (DRPs) should maintain the ability to calculate the baseline for their DR programs.

MCE encourages the working group to move forward with problem statement six, but notes that the non-participant registration issue is not the only challenge with scaling the control group method. Additional barriers faced with this method include the requirement to create a new control group for each event, the requirement for 20 pre-event days, and the inability to adjust the baseline to account for pre-existing differences between the control and treatment group.

Requiring the formation of new control groups for each event is inefficient and further increases the costs and computational power required to implement the control group method. Currently, each control site can only be used in a single control group. Thus, the pool of potential control sites for each subsequent participant is reduced until the pool is eventually exhausted. MCE is currently working on developing a modified methodology, which in part evaluates how many control groups a site may be used in before it begins to create impactful bias. The requirement for 20 pre-event days further limits utilization of the control group method because it limits use of sites that receive daily signals. MCE’s modified methodology uses Bayesian statistical methods to create evolving baselines that require less non-participation performance data. This methodology trains an existing baseline with new information, rather than creating one from scratch for each event, and is designed to use intermittent non-participation days to update an evolving baseline.

Lastly, there is currently no way for a DRP to modify the baseline to account for pre-existing differences between the control and treatment group. Allowing for limited and data-based adjustments to control group baselines to account for known differences would further reduce the cost and computational power required to utilize the control group method.

^[1] Demand and Distributed Energy Market Integration (DDEMI) Working Group (WG) Presentation, California ISO, May 13, 2025: https://stakeholdercenter.caiso.com/InitiativeDocuments/Presentation-Demand-Distributed-Energy-Market-Integration-May-13-2025.pdf

^[2] See Demand Response Advanced Measurement: Methodology Analysis of Open-Source Baseline and Comparison Group Methods to Enable CAISO Demand Response Resource Performance Evaluation, Recurve Analytics, Inc., February, 2022: https://www.caiso.com/Documents/Demand-Response_Advanced_Measurement_Methodology_updated_Feb_2022.pdf

^[3] See California Public Utilities Commission (CPUC) Data Working Group established under R. 22-11-013, Rulemaking to Consider Distributed Energy Resource Program Cost-Effectiveness Issues, Data Access and Use, and Equipment Performance Standards: https://www.laregionalcollaborative.com/data-working-group/

^[4] DDEMI WG Presentation - Prescriptive Baselines in CAISO, Leap, March 3, 2025: https://stakeholdercenter.caiso.com/InitiativeDocuments/Presentation-Leap-Prescriptive-Baselines-Mar-03-2025.pdf

^[5] Similar to the Systemwide Dynamic Load Profiles that community choice aggregation programs use for billing determinants, for example.

1. What baseline improvements would increase performance, entry, and demand flexibility innovation?

Improvements to the control group methodology would facilitate its increased utilization in demand response (DR) wholesale settlement calculations, resulting in the following benefits:

1. Enhanced Performance Accuracy: Based on CAISO’s 2017 Baseline Accuracy Report[1] the Control Group is the most accurate baseline methodology available and provides the best representation of what would have happened in the absence of the DR program.
2. Encouraging Broader DRP Entry: With more accurate performance measures, DR program participants are more likely to have confidence in the effectiveness of DR programs, leading to increased participation. Control groups provide a transparent and verifiable method for measuring DR performance, which can help to build trust between participants, demand response providers, and system operators.
3. Fostering Demand Flexibility Innovation: By providing a more accurate and reliable measure of DR performance, control groups can encourage the development and adoption of new and innovative demand flexibility strategies, including the integration of distributed energy resources (DERs) into the grid. They provide a better counter-factual than day matching baselines for frequently dispatched resources.  They also help to improve grid stability and reliability by ensuring that demand reduction is accurately measured, therefore enabling more accurate forecasting of DR load reduction capabilities.

PG&E outlines steps to improve the existing Control Group baseline methodology in its March 3, 2025 proposal[2]:

• Explicitly delineate between Hold-out versus Matched Control Group approaches.
• For matched control group, do not require registration of control group customers in the Demand Response System.
• A baseline adjustment should be allowed for control groups to account for error.
• A second validation period should be used to better account for Demand Response event days.  Additional validation metrics should be used depending on customer types to justify sufficient bias, confidence, and precision.

Both CAISO and third-party stakeholders noted potential challenges with PG&E’s proposal. In the April 7, 2025 Working Group meeting, CAISO noted the following:[3]

• ISO Challenge: verifying that the control group customers haven't also? been registered in another DR program?
• ISO Challenge: verifying the existence of the control group customers

In addition, stakeholders submitted comments noting that removing DRRS registration requirement would help IOUs but not third party DRPs who do not have access to non-participant interval data, indicating the need for control group load profiles that are accessible to all DR providers, not only the IOUs.[4]

In response to this feedback, PG&E suggests the working group explore the development of aggregated, synthetic control groups consisting of anonymized data which could be verified by CAISO and used by all demand response providers.

[1] California ISO Baseline Accuracy Working Group Proposal, Nexant, June 6, 2017, Ch. 3, pg. 4: https://stakeholdercenter.caiso.com/InitiativeDocuments/2017BaselineAccuracyWorkGroupFinalProposalNexant.pdf

[2] Proposal to Revise Demand Response Control Group Settlement Methodology, Pacific Gas & Electric Company, March 3, 2025: https://stakeholdercenter.caiso.com/InitiativeDocuments/Presentation-PGE-Settlement-Methodology-Mar-03-2025.pdf

[3] Presentation – Demand and Distributed Energy Market Integration – Apr 07, 2025, California ISO, April 7, 2025: https://stakeholdercenter.caiso.com/InitiativeDocuments/V2-Presentation-Demand-Distributed-Energy-Market-Intgeration-Apr7-2025.pdf

[4] Comments on Demand and Distributed Energy Market Integration, California Efficiency + Demand Management Council, Leapfrog Power, Inc., March 28, 2025: https://stakeholdercenter.caiso.com/Comments/AllComments/07dbeaeb-176a-4b78-9c30-21669fda6eac

2. Why are the control group and weather matching baselines not widely used?

PG&E describes the current constraints of the Control Group baseline methodology in its “Proposal to Revise Demand Response Control Group Settlement Methodology.”

• It is operationally and legally challenging for Load Serving Entities (LSEs) who act as Demand Response Providers to implement a “Matched” control group as it would require registration of customers in the Demand Response Registration System who do not enroll in a LSE’s DR program.
• The Tariff and Business Practice Manual does not articulate baseline adjustments to account for error.
• The validation methods described in the Tariff, Business Practice Manual, and Demand Response Registration User Guide do not adequately account for bias in the control group.
• As noted in stakeholder comments, third-party DRPs do not have access to candidate control group customer interval data.

1. What baseline improvements would increase performance, entry, and demand flexibility innovation?

The Department of Market Monitoring (DMM) has advocated for the working group to revise current demand response baseline methodologies prior to the addition of new ones. Based on responses received from the CAISO during past stakeholder meetings, PacifiCorp agrees to evaluate what is currently available in each baseline type (control group, day-matching, weather matching, etc.) and refining to better fit the needs of stakeholders today and in the years to come. The DMM presented that currently the day-matching baseline methodology is what is widely used today. The majority of PacifiCorp’s demand response fleet also use day-matching as it has found success in its implementation. From presentations and conversations thus far in the working group, it is apparent that the onboarding process for registering demand response programs has shown to be a significant challenge faced by market participants and that the metering infrastructure required to enact demand response programs are more stringent than of those of scheduling coordinator metering entities (SCMEs) as described in Portland General Electric's presentation.  

In past comments, PacifiCorp has expressed support for evaluating Leap’s proposal on prescriptive baselines. It is PacifiCorp’s understanding that prescriptive baselines provide a less stringent performance evaluation methodology. Additionally, in past comments, PacifiCorp has requested the CAISO to explore with stakeholders the requirements that would be needed for implementing a reliability demand response programs paradigm for market participants outside of California. PacifiCorp looks forward to discussing this topic in a future working group meeting. 

2. Why are the control group and weather matching baselines not widely used?

PacifiCorp finds the control group baselines costly, difficult to employ and maintain. To accommodate the control group baseline, PacifiCorp would need a large enough sample size to achieve a reasonable control group that is difficult to achieve for programs that do not have enough participants to randomize the subgroups and perform analysis. Weather matching baselines are narrow as there are other factors that need to be considered in addition to weather, for example, irrigation programs load can vary depending on how recently the crop is irrigated and time of year crops are growing. PacifiCorp has found greater success in implementing day-matching baselines for its current programs; however, day-matching implicitly includes weather matching as similar weather are likely to materialize on similar days for when the programs would be dispatched.