Comments on Preliminary Portfolio Analysis

1. Provide a summary of your organization's comments on the preliminary portfolio analysis:

The Alliance for Retail Energy Markets (“AReM”) provides these comments on the Resource Adequacy Enhancements: Supplemental Report – Preliminary Portfolio Assessment (“Supplemental Report”), issued November 6, 2020.  AReM is a regulatory alliance of electric service providers (“ESPs”) active in the California retail direct access market as load-serving entities (“LSEs”).  AReM’s members are also active participants in the CAISO’s markets, including the buying, selling or scheduling of power on a wholesale basis.  (This submittal represents the position of AReM, but not necessarily that of a particular member or any affiliates of its members with respect to the issues addressed herein.

The CAISO’s Resource Adequacy (“RA”) Enhancements Initiative has been underway since October 2018 and involves possibly extensive revisions to the RA program, such as (“UCAP”) and Maximum Import Capability (“MIC”) changes for RA imports.  As LSEs with Scheduling Coordinators, AReM members must comply with any changes adopted by the CAISO and approved by FERC.  At the same time, AReM members are subject to the RA requirements promulgated by the California Public Utilities Commission (“CPUC”), a Local Regulatory Authority (“LRA”).  The CPUC is currently considering its own extensive revisions to the RA program in Track 3 of R.19-11-009.  The CAISO has been active in that proceeding and has proposed significant revisions to the CPUC’s RA program, as have other parties.  In addition to the R.19-11-009 proceeding, the CPUC has just issued a new rulemaking (R.20-11-003) to address emergency reliability needs for 2021 and 2022.

AReM understands and appreciates that the CAISO, like the CPUC, is re-thinking RA requirements to take into account the lessons learned from the August rolling outages.  In the Supplemental Report, the CAISO proposes that a new, interim “secondary resource adequacy reliability requirement must be promptly instituted to ensure sufficient RA capacity is available across the net-load peak hours” to be implemented for the 2022 RA compliance year (Supplemental Report, p. 18).   The CAISO describes this new requirement as an “additional planning reserve margin that must be met with RA resources across” the critical evening net-load peak hours (Supplemental Report, p. 6). 

This proposal, in particular, is of concern to AReM because of its potential to lead to conflicting CPUC and CAISO RA requirements being imposed on LSEs, as well as the possibility of unintended consequences from lack of adequate vetting or overlapping requirements.  AReM strongly recommends that, if the CAISO wishes to pursue this proposal for the 2022 RA compliance year, it immediately submit a detailed proposal for consideration in Track 3 of R.19-11-011 to ensure that the proposal is vetted there as well as in the CAISO stakeholder process.  This will facilitate coordination of the two agencies emergency response actions.

AReM also notes that the CPUC approved changes to the maximum cumulative capacity (“MCC”) buckets in D.20-06-031 specifically to address California’s increased reliance on use-limited resources to meet reliability needs.  Those changes have been implemented for the 2021 RA compliance year and should mitigate some of the CAISO’s concerns by ensuring that more RA resources will be available 24x7 and requiring certain RA resources to be available to dispatch from 4:00 to 9:00 pm.  AReM is unaware if this new required type of dispatch behavior was taken into account in the CAISO’s modeling conducted for the Supplemental Report.  This is a clear example of how the CPUC and CAISO efforts need to be aligned.

AReM also notes that the Supplemental Report contains a proposal for the CAISO to conduct a monthly evaluation of RA resources to determine if they are sufficient to meet the expected net load peak hours, or whether there is a collective deficiency.  AReM believes that this approach may have some merit in that it would provide a measured and targeted way to prepare for anticipated extreme weather events.  However, as stated above, the proposal cannot be developed in a vacuum without evaluation of other changes that will impact LSEs’ RA requirements, and must be implemented in such a way that CPUC and CAISO requirements on LSEs are aligned.  Not doing so will add further complexity to RA procurement and higher costs for customers by both the CAISO and CPUC rushing to make changes to the RA program without considering what the other agencies are doing.

Finally, to the extent the CAISO considers any LSE-specific RA requirements, such as setting additional Planning Reserve Margins (“PRM”) or changing monthly RA requirements, the CAISO must take into account the customer load profiles of ESPs, which serve primarily commercial and industrial loads.  ESPs’ loads are non-coincident, peaking earlier in the day.  Thus, ESPs’ contribution to the system peak and net demand peak differ from that of other LSEs, whose peaks more closely align with the CAISO’s peaks.  Accordingly, to ensure proper cost causation, the CAISO must consider the differences in ESPs’ customer load profiles when proposing to revise the PRM or monthly RA requirements. 

2. Provide your organization’s feedback on the Overview of the CAISO’s Production Simulation Model topic as described in section 4:

3. Provide your organization’s feedback on the defining “deficiency” topic as described in section 4.1:

4. Provide your organization’s feedback on the iterations and output topic as described in section 4.2:

5. Provide your organization’s feedback on the model details topic as described in section 4.3 and all relevant subsections:

6. Provide your organization’s feedback on the results topic as described in section 5:

7. Provide your organization’s feedback on the interim needs topic as described in section 6:

8. Provide your organization’s feedback on the proposed foundational framework as described in section 7:

9. Additional comments on the preliminary portfolio analysis:

1. Provide a summary of your organization's comments on the preliminary portfolio analysis:

CalCCA appreciates the CAISO’s efforts in setting up, performing and sharing the results of the RA Enhancements Preliminary Portfolio Analysis (Portfolio Analysis). It is obvious that a lot of effort and thought was put into this work. We offer below some observations and questions related to the Portfolio Analysis.

2. Provide your organization’s feedback on the Overview of the CAISO’s Production Simulation Model topic as described in section 4:

CalCCA continues to support the use of a modified version of the production simulation model CAISO uses for the Summer Assessment for the portfolio assessment. CalCCA believes, however, that there is considerable value in modeling only the shown RA resources in all hours (i.e., peak and off-peak), as well as a scenario that includes all WECC resources, in and outside of California. The purpose of the portfolio assessment is to identify instances in which the shown RA resources would not be able meet the performance metric, which cannot be ascertained if non-RA resources are included in the assessment. At the same time, it is critical to understand the impact of including all WECC resources, which have every incentive to offer their available energy and capacity into the CAISO markets due to regional and diurnal diversity. The performance of the entire WECC portfolio thus should be considered when determining the appropriate service level reliability target in this stakeholder process, and understanding both scenarios is critical to understanding the risks of not meeting the performance target solely with the RA Portfolio. CalCCA also believes including the Thermal Portfolio provides a useful benchmark for comparison.

3. Provide your organization’s feedback on the defining “deficiency” topic as described in section 4.1:

CalCCA supports the use of a metric to identify resource deficiencies that would have triggered a Stage Two Emergency but suggests CAISO refrain from characterizing this as a loss of load expectation as these are not equivalent concepts. Nevertheless, it seems reasonable to identify instances in which the model shows there would be inadequate capacity to meet load plus non-spinning reserves plus spinning reserves plus regulation, since CAISO would need to obtain emergency supply in these circumstances. As CAISO has noted, the service level reliability criteria to apply and the cost tradeoffs are critical elements that need further investigation in coordination with local regulatory authorities, since these entities are responsible for making the cost/benefit determination in consultation with CAISO as the Balancing Authority.

4. Provide your organization’s feedback on the iterations and output topic as described in section 4.2:

CalCCA understands that the Portfolio Assessment modeled performance of the July 2020 shown RA fleet using stochastic production simulation with 2,000 month-long iterations to compute the probability of a portfolio deficiency with hourly, daily and monthly granularity.  The hourly level data can be used to assess the hours in which CAISO is most likely to need additional capacity and the duration of the deficiencies. The daily level results show the probability of a deficiency within a day and identifies the magnitude of the largest daily deficiency. CalCCA understands that the more granular data increases the robustness of the simulation results.

5. Provide your organization’s feedback on the model details topic as described in section 4.3 and all relevant subsections:

Section 4.3.1 CAISO System: CalCCA supports the use of a WECC-wide model to capture regional interactions that have a critical impact on CAISO’s access to external resources. While CAISO does not have access to the same level of detailed information about the non-CAISO loads and resources, the regional model data is readily available to CAISO and already is used extensively by CAISO in the transmission planning process and as part of its operational planning. CalCCA suggests net imports be limited to the amount of Shown RA resources during both peak and off-peak hours for at least one of the scenarios. If imports are thus limited, CAISO may be able to consider running a California-only model for the Shown RA portfolio analysis, since the remaining WECC resources will not be relevant to the analysis. This reduction in the model should significantly reduce model run-time and thus allow for potentially more complex modeling assumptions, such as the correlation between loads and intermittent resource availability. CalCCA supports focusing on system level requirements and continuing to separately use the specialized assessment performed for local capacity requirements. We note, however, that in many local capacity areas/sub-areas, CAISO needs all or nearly all available local resources and these resources might not be included on RA showings. If CAISO ultimately is likely to have access to these resources (e.g., via backstop procurement), the impact of these resources ultimately being procured needs to be factored in to avoid over procurement of system resources. That is, parties should not be encouraged to cure system resource adequacy deficiencies that would then be rendered moot by CAISO’s backstop procurement of needed local capacity resources.

Section 4.3.2 Load Inputs: CAISO has developed 175 hourly load profiles based on historical weather data. The distribution of loads depicted in Figure 1 for the July production simulation appear to be a reasonable representation of the distribution of potential load outcomes. CalCCA expects that if CAISO applies a similar approach to developing the load distributions for other months, it likewise will result in reasonable representations of the range of potential outcomes to include in future simulations.

Section 4.3.3 Resource Inputs: CalCCA understands that CAISO has included historical production profiles for wind and solar resources in the RA Scenario, rather than using the ELCC MW for these resources. We agree that this is an appropriate approach for evaluating the impact of these Shown RA resources. Applying the ELCC MW would understate their expected impact during some hours and overstate their impact during other hours, greatly reducing the meaningfulness of the analysis. For the Thermal Scenario, CAISO has replaced the wind and solar resources with a representative mix of thermal resources, grossed up for the 15% planning reserve margin. CalCCA suggests that a more appropriate approach would be to replace the shown wind and solar ELCC MW with the thermal resources on a MW-for-ELCC MW basis. Had the wind and solar resources not been available to be shown, parties could have met their RA obligation by providing an equivalent MW amount of thermal resources without having to account for the actual production profile of the solar/wind resources and without having to account for the planning reserve margin beyond the amount already incorporated into the RA requirement.

In addition to the RA and Thermal Scenarios, CalCCA suggests CAISO should include two additional scenarios. The first would attempt to model the anticipated impact of the RA Enhancements UCAP proposal by grossing up the amount of conventional resources to address the expected increase in the RA requirement to account for the UCAP outages. We understand that this would be a hypothetical scenario and might also need to consider an adjustment to the overall obligation related to the potential decrease in the PRM from 15% to 10% associated with implementation of UCAP. We believe, however, that it would be a useful scenario for parties to consider informing the discussions related to development of the service level reliability criteria. The second scenario would be one in which all WECC-wide loads and resources would be modeled, whether or not they were included in RA showings. This scenario would be similar to the approach taken for the summer assessment to aid in identifying the potential risks related to the Shown RA deficiencies identified in the other scenarios.

CalCCA understands CAISO intends to model Hydro resources using similar water year production. This appears to be a reasonable approach, but if CAISO were to perform the portfolio modeling following the annual showings (rather than just for monthly showings), additional hydro uncertainty might need to be introduced into the modeling.

6. Provide your organization’s feedback on the results topic as described in section 5:

CalCCA appreciates the CAISO’s presentation of the results for the RA Scenario and the Thermal Scenario, which illustrate how the performance of the different RA fleets differ throughout the day (and, presumably, would differ throughout the year). These type of results (supplemented with additional analysis of portfolio performance in other months) help provide guidance for future procurement to help address the identified deficiencies.  For example, the finding that over 90% of the days with deficiencies in both scenarios had deficiencies less than four hours duration suggests that four-hour storage resources could be useful for mitigating a large portion of the deficiencies.

The results suggest that the additional output of the modeled solar and wind resources above their ELCC contribute to reduced chances of RA deficiencies during some intervals, while the converse is true during other intervals. As noted in response to question 5, modeling a UCAP scenario in which the amount of shown RA resources is increased to align with the increased UCAP RA requirements is likely to show there will be reduced levels of deficiencies once UCAP is implemented as part of the RA Enhancements.  CalCCA believes that it is important for CAISO to include such a UCAP scenario in future analyses so parties can evaluate portfolio performance that incorporates this key element of the RA Enhancements proposal.

7. Provide your organization’s feedback on the interim needs topic as described in section 6:

CalCCA appreciates that the CAISO agrees that a net-load peak RA requirement is essential, as proposed by SCE and CalCCA in Track 3b of the CPUC RA proceeding. Our expectation is that with the SCE/CalCCA approach, the gross load peak requirement will no longer be needed. Additional portfolio modeling that includes representative RA portfolios that would meet such a net-load requirement would be useful to ascertain whether it is likely to be necessary to maintain a gross load peak requirement.  However, CalCCA is concerned that CAISO’s proposal to develop an interim net-load peak RA requirement for the 2022 RA year lacks specificity and a proposed means of implementation and will divert resources from finalizing the RA Enhancements needed for the 2023 RA year implementation. Under such a short timeframe, the proposal is also not likely to have a meaningful impact on the amount of new RA resources that can be procured by the 2022 RA year. LSEs need an appropriate amount of lead-time to comply with such a requirement, which should be developed in concert with the CPUC. 

8. Provide your organization’s feedback on the proposed foundational framework as described in section 7:

CalCCA appreciates CAISO’s discussion of the granularity of the RA program as a necessary component for determining the appropriate reliability standard and agrees that California’s monthly RA construct introduces complexities in the application of that standard. Further, the monthly compliance construct leaves very little time for deficiencies to be addressed if identified in a month-ahead assessment. An annual standard could provide greater efficiency in RA procurement and a longer runway for addressing potential shortfalls. Finally, as noted in CAISO’s report, even under a monthly construct an annual standard is required to ensure all months are bound by a single guiding reliability standard. While CalCCA supports further exploration of the relative advantages of an annual standard, the RA period chosen must ultimately be consistent between the CPUC and CAISO.

9. Additional comments on the preliminary portfolio analysis:

1. Provide a summary of your organization's comments on the preliminary portfolio analysis:

CDWR may provide its comments on the portfolio assessment after the CAISO performs additional studies and the results from such studies are available.

CDWR acknowledges that to establish resource procurement obligations, it is necessary to determine an acceptable level of service reliability given the probability of a capacity shortfall and potential for involuntary load shedding. CAISO wants to make a decision on the granularity of the RA program (monthly or, seasonal or, annual), application of annual planning standard, and desired service level reliability target.

With regard to the granularity of RA program, as the portfolio deficiency test is done on a monthly granularity, it would be prudent to apply RA requirements based on the monthly granularity of deficiency test as well. Monthly granularity may also be supported by the type of resources and their varying monthly availability. For the desired service level target, it would be prudent to compare with  a WECC standard  like Loss of Load Probability (LOLP) for the same target.

2. Provide your organization’s feedback on the Overview of the CAISO’s Production Simulation Model topic as described in section 4:

WECC adequacy standard is set at LOLP of ≤5%. LOLP includes firm hydro and thermal, market supplies, non-declared utility resources, and buyback provisions on load. WECC considers LOLP as the likelihood of taking emergency actions, not necessarily curtailment. WECC LOLP may have included all resources such as non-RA resources, but  non-RA economic energy is excluded in CAISO portfolio analysis. If the WECC adequacy standard (LOLP) is  inclusive of such resources, it may be worthwhile for CAISO to consider including similar resources for setting similar adequacy standard.

3. Provide your organization’s feedback on the defining “deficiency” topic as described in section 4.1:

CAISO proposes to define “deficiency” as the shortfall in non-spin, spin, and regulation at stage 2 emergency. Resources that can provide these shortfalls should be allowed to be eligible to provide RA under the RA enhancement proposal. For example, a participating load that provides RA today by offering non-spin when contingency occurs, should be able to provide RA as it does today without requiring the must offer obligation of various  new products such as reliability capacity up/down.

4. Provide your organization’s feedback on the iterations and output topic as described in section 4.2:

Are all the resources’ metered generation capped at shown RA capacities? If the generation is metered generation capped at RA capacity, there may be certain hours in which RA capacity generation was not dispatched (no metered generation) due to the market clearing process. In addition, the same resource may also be shown below NQC values for a month leaving unused but reliable non-RA capacity from the same resource. How are these two realistic factors going to affect the analysis result?

5. Provide your organization’s feedback on the model details topic as described in section 4.3 and all relevant subsections:

It is apparent that resource input in the model is not limited to shown RA capacity in case of wind and solar. Solar is 333 MW above the shown RA capacity. Contrary to the  study objective to exclude non-RA capacity, the model analysis does include some part of non-RA capacity from wind and solar. Why is the non-RA capacity from other traditional RA resources within the CAISO system  would considered to be unreliable? In fact, the WECC adequacy standard of ≤5% LOLP may have included all resources under the category of “market supplies”.

 The proposal states, “Maximum production levels for dispatchable hydro units are capped at the shown NQC. While the resource may be capable of producing more than NQC, the CAISO’s objective was to test the shown RA values. Additionally, based on the CAISO’s preliminary review of hydro resource availability, NQC seems to provide a reasonable cap on its overall maximum availability.” This statement creates confusion as to the whether cap will be based on the shown NQC or NQC; Shown NQC ≤NQC; If capped at NQC, unused reliable non-RA capacity would be included. However, a cap based on shown NQC may not include all the capacity based on maximum availability when shown NQC < NQC.

6. Provide your organization’s feedback on the results topic as described in section 5:

No comment.

7. Provide your organization’s feedback on the interim needs topic as described in section 6:

The current flexible RA framework is designed to address the duck curve which is based on the net load. Are the existing flexible RA framework and other flexible products not sufficient to address the net load peak requirements?

8. Provide your organization’s feedback on the proposed foundational framework as described in section 7:

It would be interesting to see  how much of 8,637 MW shortfall could have been reduced if the non-RA reliable available capacity from RA resources was included as model input.

CAISO identifies two core challenges that must be addressed:

1. Establishing a defined reliability criteria or loss-of-load expectation that determines procurement targets and backstop procurement trigger,
2. Determining the quantity and attributes of capacity needed to address a portfolio deficiency.

CAISO seeks decision on key issues as following:

1. The correct granularity of the RA program: Annual, Seasonal, or Monthly?
2. The application of an annualized planning standard
3. The desired service level reliability target

   Granularity of RA program: The benefit of a monthly RA program is that it helps address uncertainty of forecast, weather related variations, and resource types such as variable energy resources. If an annual or seasonal only RA program is administered, how will the benefits of monthly  RA program be maintained? In a monthly RA program, resources are allowed to modify the RA capacity during a month compared to annual filing based on the current availability (within the NQC). This flexibility allows for better reliability and resource utilization. Under seasonal or annual only RA program, this flexibility may not be there if monthly showings are eliminated.

    Application of planning standard: Further data analysis on procurement targets for peak and off-peak  months can inform what would be an appropriate alternative.

Desired service level reliability standard : The desired service level reliability standard is defined by determining an acceptable loss of load probability (LOLP) when setting its RA procurement targets. The July RA analysis shows approximately 3% as acceptable LOLP. If further analysis is performed for all 12 months, the monthly variations of LOLP would be observed with a range that may allow to compare with seasonal and annual standard. Also reference to WECC adequacy standard may provide a better judgement on what planning standard should be.

9. Additional comments on the preliminary portfolio analysis:

No further comments.

1. Provide a summary of your organization's comments on the preliminary portfolio analysis:

The preliminary Portfolio Analysis is a good start to better understand how the fleet of Resource Adequacy (RA) resources meets the reliability requirements under different load assumptions.  However, as explained in more detail below, the methodology needs to improve the definition of an outage, resolve various inconstant assumptions, and address missing modeling inputs. 

It would be helpful to get clarity as to how the CAISO will utilize this analysis and what information will be provided to load serving entities so they can avoid deficiencies or collective deficiencies.  LSEs need to have clear instructions on how resources will be counted and the required reliability targets, and if the instructions are followed LSEs should have a reasonable expectation they will not be subject to deficiency costs after they make the RA showing.  The current proposal lacks clarify on how it will actually improve the ability of LSEs to make the necessary procurement.

The CAISO should continue to refine the approach before including it in any RA Enhancements proposal.  It is better to take the time to get it right than implement a rushed proposal that may suffer from unknown flaws.  This effort will also require more coordination with the California Public Utilities Commission (CPUC).

2. Provide your organization’s feedback on the Overview of the CAISO’s Production Simulation Model topic as described in section 4:

 On page 7, CAISO mentions that the “model does not currently model use-limitations that extend beyond a single day”.  This could be a significant shortcoming with a growing storage fleet, the need to properly use pondage hydro, and to assure effective use of demand response programs.  Unless the model can properly utilize use-limited resources, the results will be not be as useful to determine the sufficiency of the RA fleet.

3. Provide your organization’s feedback on the defining “deficiency” topic as described in section 4.1:

A Stage 3 Emergency occurs when non-firm load is subject to rotating outages.  The CAISO is proposing to define a Stage 2 Emergency as “when there is inadequate capacity to meet the aggregate of non-spin, spin, regulation, and load.”  During the CPUC RA Track 3B call on November 23, 2020, Donald Brooks of the CPUC mentioned that the CPUC’s modeling used as the definition of an outage when load, regulation, and spin could no longer be maintained.  The difference appears to be whether non-spin should be utilized before interrupting load.  The CAISO and the CPUC need to coordinate on which definition is appropriate to use in reliability modeling for planning purposes so both organizations are using a consistent definition.   

4. Provide your organization’s feedback on the iterations and output topic as described in section 4.2:

            CLECA does not have comments on this section at this time.

5. Provide your organization’s feedback on the model details topic as described in section 4.3 and all relevant subsections:

During the call on November 12, CAISO clarified that the load assumptions are not correlated with the wind and solar profiles.  The implication is that there is a possibility that for an extremely hot day scenario, load will be matched with a wind or solar profile that occurred during a cooler day when the output is higher than would be expected during a heatwave.[1]  This could distort the results by developing an infeasible scenario combining load and wind or solar profiles.  CAISO has not provided any analysis to show whether any of the 2000 scenarios suffer from this problem.  Alternatively, CAISO has not developed a method to filter the stochastics for infeasible results.  As the solar and wind fleet grows, this issue becomes a bigger concern.  This issue should be addressed in the next revision.

Figure 1, showing the distribution of 2000 scenarios of July 2020 peak loads, has an unexplained number of occurrences as the load moves toward maximum load.  For example, there are about 10 occurrences when the load is 52,400 MW, but then almost 20 occurrences of load at 52,800 MW and 53,200 MW.  It does not make sense that when load increases toward the extreme events, the number of occurrences increases.

For resource inputs, the document says CAISO used the “shown RA fleet”, but it does not specify if the intent is to include RA resources’ hourly performance or if the intent is to include “shown RA values”.[2]  On page 12, it says “while the resource may be capable of producing more than NQC, the CAISO’s objective was to test the shown RA values.”  However, the CAISO deviated from this statement in an inconsistent manner without providing sufficient justification.

For wind and solar, CAISO did not use the NQC values based upon ELCC, but used historical profiles of hourly shapes in the model.  For hydro resources, it used historical hydro shapes, but then capped the output at the shown RA values.  For justification, it only offers that, based upon its review, using NQC seemed reasonable.  While using historical shapes for run-of-river hydro would be appropriate, using historical shapes for pondage hydro would limit that latter’s ability to provide capacity during shortage conditions.  The ideas of using stochastic analysis is to incorporate different outcomes, therefore it does not make sense to artificially limit the ability of hydro to meet load with a cap.  The justification of capping hydro at the NQC should be provided.

For demand response, CAISO utilized both shown RA values and RA Credit values, but then de-rated the latter to 75% of the shown value.  In the report, the justification is simply CAISO’s operational experience.  At the November 12 meeting, the CAISO clarified its experience as the Preliminary Root Cause Analysis Mid-August 2020 Heat Storm report.  The preliminary report is just that, preliminary, and the preliminary report noted that, for demand response, obtaining and reviewing the meter data will take additional time.  However, the conclusion regarding the performance of demand response in the preliminary report is faulty because it used inconsistent definitions for the benchmark and measurement.  The report measured response at the meter, but then included about a 25% gross-up for losses and planning reserve margin in the benchmark.  When measuring and benchmarking at the meter, the performance is close to 100%.  If CAISO seeks to gather performance data, the CPUC requires an annual report on the performance of the utility DR programs which would be a superior source for performance data to reliance on faulty data.[3]

There are different demand response programs with different characteristics.  Some are available during all hours of the day and week, such as the base interruptible program. Some have more variable response, such as A/C cycling and those offered by non-utility providers.  However, the CAISO fails to distinguish these differences in its reliability model.  Instead, it incorrectly, uniformly derated programs not included in supply plans.

Since CAISO is using an hourly reliability model, it should use hourly performance data for each resource.  Furthermore, it should take into account the weather conditions that may affect the output for each resource.  Therefore, the CAISO should make the following adjustments:

• For Wind and Solar, align the recorded shapes to match the closest weather profile conditions or ensure load shapes are not matched with infeasible solar or wind output.
• For hydro, remove the capping of output at NQC for run-of-river and model pondage hydro as a dispatchable unit that is energy- limited.
• Utilize profiles for demand response that is weather-sensitive.  During cooler periods, a lower amount, compared to its qualifying capacity based upon a 1 in 2 weather assumption, would be used.  During hot weather events greater than a 1 in 2 weather condition, the amount of expected weather-sensitive demand response would be greater than its qualifying capacity.
• For the thermal fleet using combustion turbines, include the necessary adjustments to reflect output increases during cooler temperatures and decreases during hotter temperatures.

[1] As temperature increases, the efficiency of solar panels declines.  When wide-spread heatwaves occur there is a lack of temperature differentials which cause wind output to decline.

[2] For example, a Shown RA Value would be a solar unit’s qualifying capacity based upon ELCC used as an input into the model, while a Shown RA Resource would be its hourly output as the input into the model.

[3] If CAISO disagrees with the application of adjustments for planning reserve margin or inclusion of line losses in the RA planning process for operational purposes, then it should be clear on this issue.  This is a separate issue of performance by DR customers’ ability to reduce load measured at the customer’s meter. 

6. Provide your organization’s feedback on the results topic as described in section 5:

Because of the modeling issues that need to be resolved, the results should not be utilized to draw any conclusions. 

            The following CAISO conclusion, on page 14, is factually incorrect:

The availability of DR resources in hours 19-21 in the model may also result in under-estimating the probability of shortfalls in the RA Showing Scenario. It is reasonable to assume that DR is available in the middle of the day for the Thermal Scenario, however, it likely a generous estimate is DR’s incremental load drop capabilities in the mid- to late-evening hours.

            CAISO said it used 75% of DR’s qualifying capacity in the model.  The Load Impact Protocols average the results from a 4 – 9 pm event.  With the arbitrary derate of DR’s performance, in reality, the model overestimated the probability of shortfall in the 4-9 pm period.  As for the middle of the day performance, DR response by programs with commercial customers (8 am- 5pm operation) will be understated because of the averaging on the performance during 4-9 pm.  However, this may be offset by reduced performance from residential programs (assuming no impact due to Covid-19 of increased residential load).  With Covid-19 impacts, the residential programs may have performed better than the Load Impact Protocol results for a 4-9 pm event. 

7. Provide your organization’s feedback on the interim needs topic as described in section 6:

            CLECA agrees that both the peak and net-peak periods should be examined for resource adequacy.

8. Provide your organization’s feedback on the proposed foundational framework as described in section 7:

            CLECA does not have comments on this section at this time.

9. Additional comments on the preliminary portfolio analysis:

1. Provide a summary of your organization's comments on the preliminary portfolio analysis:

Calpine appreciates the careful analysis reflected in the Preliminary Portfolio Analysis (PPA).  Calpine believes that it could be useful in developing ex ante reliability-based RA procurement requirements and accurate resource counting rules. 

Calpine does not agree, however, with the PPA’s conclusions on market design.  In particular, the CAISO continues to recommend using the analysis reflected in the PPA for an ex post validation of RA showings.  This sort of ex post validation exposes load to backstop costs that they cannot manage and fails to provide appropriate incentives to contract with the resources that are actually required to maintain reliability on a more forward basis.  In addition, this approach allows only the limited time afforded by the CAISO’s backstop mechanisms to cure any deficiencies identified in the ex post validation. 

Calpine also disagrees with the conclusion in the PPA that its analysis illustrates the need for new and distinct capacity requirements focused on net peak load.  Instead, it may illustrate that current RA requirements are too low to meet common reliability standards.  The fact that loss of load is concentrated in the evening may be fully consistent with achieving a reliability target, such as 1 event in 10 years, just as loss of load tends to be concentrated around gross load peaks at lower levels of renewable penetration.  Calpine believes that the current, ELCC methodology already captures the performance of resources when loss of load is likely to occur, which is now often the hours around the net peak, as well as their impact on shifting the peak later in the day when load is lower and hence capacity requirements are lower.  It may be appropriate to extend the methodology to other energy- and use-limited resources such as storage and DR.

Calpine strongly agrees with the following conclusion from a white paper that E3 developed for PJM:

As renewables and storage resources have gained market share, the PRM framework has been criticized as an antiquated, “peak-focused” requirement that is no longer relevant in a world where other hours may be more difficult for system operators to manage due to ramping events or lack of renewable energy production. However, these criticisms fail to recognize that a PRM requirement does not literally represent a requirement for capacity during the peak hour, but rather a requirement for capacity throughout the year that, for simplicity, is expressed in relation to a system’s expected peak demand. This convention originated in the era of firm resources: so long as a system had sufficient capacity to meet peak demand, those same resources would also be available to meet demand under all other conditions. The integration of increasing levels of renewables and storage does not render the PRM framework obsolete, but it does require more advanced techniques to measure the contribution of different types of resources towards that capacity requirement.[1]

Consequently, Calpine views efforts to develop new “net peak” or “all hours” capacity requirements as a distraction from the fundamental need to establish clear reliability-based capacity requirements, which could be framed in terms of a traditional planning reserve margin or something else, and accurate resource counting.

[1] https://www.ethree.com/wp-content/uploads/2020/08/E3-Practical-Application-of-ELCC.pdf

2. Provide your organization’s feedback on the Overview of the CAISO’s Production Simulation Model topic as described in section 4:

Calpine generally supports the CAISO’s stochastic production cost simulation approach to assessing reliability.

3. Provide your organization’s feedback on the defining “deficiency” topic as described in section 4.1:

Calpine generally agrees with the CAISO’s definition of deficiency (but acknowledges that there is a credible case to be made that a deficiency only occurs when load is shed involuntarily, not when load carries contingency reserves and faces the risk of being shed in the event of a contingency[1]).  Calpine also recognizes that the CAISO’s analysis reflects the ability of the RA fleet alone to serve load.  Calpine believes that this framing is reasonable.  CAISO should be able to serve load with the RA fleet and should not be counting on resources that have not been secured as RA to assure reliability.

[1] For example, see pp. 30-31 of http://www.caiso.com/Documents/Preliminary-Root-Cause-Analysis-Rotating-Outages-August-2020.pdf. 

4. Provide your organization’s feedback on the iterations and output topic as described in section 4.2:

Calpine has no comments on this topic at this time.

5. Provide your organization’s feedback on the model details topic as described in section 4.3 and all relevant subsections:

In light of the results, Calpine wonders whether the CAISO has compared the renewable profiles that it used in its analysis to the profiles that the CPUC uses to estimate ELCCs.  One possible explanation for the result that the Thermal Scenario seems to yield lower reliability is that the renewable profiles in the CAISO’s analysis are better from a reliability perspective than the profiles used by the CPUC to estimate ELCCs.  Consequently, solar and wind ELCC may understate the reliability contributions of those resources in the CAISO’s analysis and hence the amount of gas capacity with which they need to be replaced to yield equivalent reliability.  Another possibility, consistent with analysis that the CAISO has developed in support of its UCAP proposal, is that the average forced outage rates utilized in the analysis are significantly higher than typically assumed in reliability studies.  Consequently, it takes more imperfect gas generation to replace ELCC-based renewable generation and yield the same level of reliability than the CAISO assumed in developing the Thermal Scenario.

6. Provide your organization’s feedback on the results topic as described in section 5:

It appears that the 7.9 and 15.5 percent probabilities of a deficiency (or 2.6 and 4.1 percent excluding small deficiencies) are directly comparable to conventional reliability metrics such as 1 event in 10 years, where an event is treated as any day with loss of load.  If so, these reported probabilities of a deficiency seem high, i.e., almost as high or higher in a month than would be consistent with 1-in-10 for a full year.  This suggests that either the current planning reserve margin is too low to meet 1-in-10, at least without non-RA resources.

7. Provide your organization’s feedback on the interim needs topic as described in section 6:

Calpine disagrees with the conclusions in the Interim Needs section.  The analysis in the PPA suggests that the current PRM is too low.  It provides no obvious evidence that the fundamental construct of a single capacity requirement with ELCC counting for intermittent resources is flawed.  Even if a new net peak requirement were introduced, as the CAISO proposes, an appropriate level for the requirement analogous to a PRM would have to be developed.  California has a very complicated RA construct with many overlapping and arguably redundant requirements and products.  Before introducing yet additional requirements, the CAISO (and the CPUC) should ensure that the current requirements are actually specified correctly.

8. Provide your organization’s feedback on the proposed foundational framework as described in section 7:

Calpine agrees that foundational elements of a RA framework include establishing clear reliability requirements and how resources count towards those requirements and that to address those requirements it is necessary to determine the temporal granularity of the requirements including how annual loss of load standards might be applied to individual months under a program with monthly requirements.

9. Additional comments on the preliminary portfolio analysis:

1. Provide a summary of your organization's comments on the preliminary portfolio analysis:

EDF-Renewables appreciates the CAISO providing the Resource Adequacy (RA) Preliminary Portfolio Analysis as well as the CAISO’s commitment to provide further modeling using other months’ RA showings. Certainly, more information is needed to complete the picture and evaluate the effectiveness of the portfolio analysis as a tool in the framework proposed in the RA enhancements initiative. The portfolio analysis conclusion that the July 2020 RA showing is more reliable than the Thermal Scenario does not align with expected results and begs deeper investigation into the specific Thermal Scenarios that failed. With respect to near-term needs, EDF-Renewables is skeptical that increasing the CAISO’s backstop authority is the best use of limited time for new policy development between now and Summer 2021. EDF-Renewables requests the CAISO proactively collaborate in the framework established by the Joint Agencies’ (CAISO-CEC-CPUC) Preliminary Root Cause Analysis report  during the RA enhancement policy development effort and in other venues.

The August load shedding events were disquieting for everyone. The CAISO’s position that it “carries the ultimate obligation for maintaining the reliability of the bulk power system” is understandable and appreciated. The Governor and the legislature’s response to the August load shedding events and the Joint Agencies’ (CAISO-CEC-CPUC) August Load Shedding Root Cause Analysis report  make it clear that the way to ensure the CAISO can meet that obligation is in partnership with the Joint Agencies. From the August Event’s report (page 64, emphasis added):

“Preliminary Recommendations…
1) Near-term – by Summer 2021…
b) Resource Planning and Procurement

• Increase RA requirements for LSEs to more accurately reflect increasing risk of extreme weather events - The current planning targets were developed in 2004 and have not been updated since. The 1-in-2 load forecast plus a 15% reserve margin should be updated to better account for heat storms like the ones encountered in both August and September. The CPUC already has an open proceeding to consider changes in how the planning targets are set for the purposes of RA rules and this discussion should start before summer 2021. Once these changes are developed, the CPUC, CEC, and CAISO should ensure they are used consistently across all long- and short-term planning programs”

It is confusing then to see that the Resource Adequacy Preliminary Portfolio Analysis proposes to create net-load peak requirement and complimentary backstop procurement outside of coordination with the Joint Agencies. The challenges developers face navigating the CAISO and CPUC’s dueling RA processes are a matter of fact, and all RA policy enhancements must seek to explicitly synergize the disparate processes. A Joint Agency context is also the appropriate venue to answer the CAISO’s question, “Under what conditions is firm load drop acceptable?” as the answers to these questions have large societal impacts beyond the specific expertise of economists and market experts, and the state agencies bear the responsibility of assuring Californians' access to reliable utility services.

It is also noteworthy that during the August load shedding events no resources accepted the CAISO’s initial capacity procurement mechanism offer because of prior contracting commitments to other Balancing Authority Areas. EDF-Renewables requests clarification on how additional backstop procurement authority will solve this supply issue. The CAISO’s opinion seems to be that there is a significant probability that backstop procurement will happen frequently, but it is not clear what generation will be available to meet that ask. New generation rarely comes online as merchant generation. The RA Preliminary Portfolio Analysis state that “If the RA program was to guarantee a service level reliability level of near 100 percent, based on modeling results, it would need an additional 8,637 MW of capacity.” This solution should not be dismissed as unreasonable. Presumably, a reliability level of near 100 percent would also align closely with the CPUC’s 0.1 Loss of Load Expectation and whatever the equivalent CAISO acceptable service level reliability standard ends up being. A significant increase in up front procurement also seems to be the collective prediction of the development community, and they are gearing up to meet it. The CAISO’s Generation interconnection queue currently contains ~88,000 net-to-POI MW of interconnection requests. Of that 88,000 MW, ~21,000 net-to-POI MW have requested deliverability and have executed interconnection agreements, both measures are meaningful indicators for project viability. All ~21,000 net-to-POI MW have had to make significant financial posting to remain in the queue to this point, and all projects indicate planned Commercial Operation Dates 2025 or earlier.

2. Provide your organization’s feedback on the Overview of the CAISO’s Production Simulation Model topic as described in section 4:

3. Provide your organization’s feedback on the defining “deficiency” topic as described in section 4.1:

4. Provide your organization’s feedback on the iterations and output topic as described in section 4.2:

5. Provide your organization’s feedback on the model details topic as described in section 4.3 and all relevant subsections:

6. Provide your organization’s feedback on the results topic as described in section 5:

7. Provide your organization’s feedback on the interim needs topic as described in section 6:

8. Provide your organization’s feedback on the proposed foundational framework as described in section 7:

9. Additional comments on the preliminary portfolio analysis:

1. Provide a summary of your organization's comments on the preliminary portfolio analysis:

See attachment.

2. Provide your organization’s feedback on the Overview of the CAISO’s Production Simulation Model topic as described in section 4:

See attachment.

3. Provide your organization’s feedback on the defining “deficiency” topic as described in section 4.1:

See attachment.

4. Provide your organization’s feedback on the iterations and output topic as described in section 4.2:

See attachment.

5. Provide your organization’s feedback on the model details topic as described in section 4.3 and all relevant subsections:

See attachment.

6. Provide your organization’s feedback on the results topic as described in section 5:

See attachment.

7. Provide your organization’s feedback on the interim needs topic as described in section 6:

See attachment.

8. Provide your organization’s feedback on the proposed foundational framework as described in section 7:

See attachment.

9. Additional comments on the preliminary portfolio analysis:

See attachment.

1. Provide a summary of your organization's comments on the preliminary portfolio analysis:

MRP appreciates the opportunity to provide comments on the CAISO’s November 6, 2020 Resource Adequacy Enhancements: Supplemental Report – Preliminary Portfolio Assessment Supplement to the Fifth Revised Straw Proposal (“Portfolio Assessment”).

Within the current RA framework, the CAISO’s initial portfolio analysis is a step toward a more effective RA program.  The CAISO’s analysis focuses attention on several key Resource Adequacy (“RA”) issues, namely (1) the appropriate level of, and metric for, reliability; (2) how reliability is measured across different periods and what the appropriate temporal granularity for the RA program should be; and (3) how much, and what kind, of procurement is needed to cure deficiencies. 

MRP strongly supports a key foundational principle of the CAISO’s portfolio analysis, namely, that the analysis is conducted using the shown RA fleet only.  For too long, reliability analyses have relied on uncontracted imports as a “slack bus” that masks potentially problematic supply-demand imbalances.  While the CAISO’s portfolio analysis is not perfect (as MRP will note further below) it represents the most sensible step forward in the evolution of the RA program in a long time. 

The CAISO’s initial analysis was performed on the July RA showings.  In light of the issues raised by August heat wave events, it would have been more interesting and insightful if the assessment had been performed on the August RA portfolio; perhaps the CAISO can consider doing this in the future to provide some measure of how the metrics produced by the portfolio analysis stack up against a real-world stress case. 

MRP views the CAISO’s portfolio analysis as a more rigorous way of determining whether the RA fleet provides the desired level of reliability (whatever that may perceived to be) within a given month.  Additionally, the analysis, while not yet fully developed, surfaces some foundational RA program design issues (e.g., the right temporal granularity and curing deficiencies, and the benefits of assessing whether the RA program is doing what it is intended to do through techniques more sophisticated than simple stack analyses facilitated by simplistically assigning single RA values to intermittent resources).    

2. Provide your organization’s feedback on the Overview of the CAISO’s Production Simulation Model topic as described in section 4:

MRP notes that the CAISO does not include or assess compliance with local capacity requirements.  Ideally, an RA portfolio assessment would provide some insight into compliance with local capacity requirements, especially given expectations that use-limited resources such as demand response (“DR”) and battery energy storage systems (“BESS”) will increasingly be used to meet local capacity requirements.  This limitation notwithstanding, the CAISO’s portfolio assessment provides valuable insight on a system basis alone. 

3. Provide your organization’s feedback on the defining “deficiency” topic as described in section 4.1:

MRP largely supports the CAISO’s proposal to define a “deficiency” as any hour in which the production simulation shows the CAISO would have to call a Stage 2 Emergency (i.e., the CAISO does not have sufficient capacity to meet its load-serving, spin, non-spin and regulation.)   MRP respectfully encourages the CAISO to consider adding “load following” requirements to the list of capacity obligations above.  If the CAISO does not have sufficient capacity to follow, for example, the three-hour net load ramp, it may have to lean on the interconnection in hours in which it does not sufficient capacity resources.  Such leaning would be inconsistent with the “shown RA portfolio only” approach to the portfolio analysis.  Given that LSEs must satisfy flexible capacity requirements as part of their RA program obligations, any hour in which the CAISO has insufficient load following capacity to cover the net load ramp should also be considered a deficiency. 

4. Provide your organization’s feedback on the iterations and output topic as described in section 4.2:

The CAISO’s approach of using stochastic draws for load, wind solar and resource outages is reasonable.  As MRP remembers (either from the CAISO’s November 13 presentation or from the November 18 presentation during the Western Electricity Council Resource Adequacy Forum), these draws are assumed to be fully independent.  While these variables may not be fully independent and there may, in fact, be some correlations between the drawn variables, assuming the variables to be fully independent may be a preferable assumption.  For example, while there may be a correlation between solar output and load (i.e., high load days could be assumed to also be high solar days), in August 2020 the CAISO experienced diminished solar across the afternoon hours on a high load day (August 15).  Any correlations enforced among the stochastic variables should also allow for such outlier events. 

5. Provide your organization’s feedback on the model details topic as described in section 4.3 and all relevant subsections:

The CAISO notes (Portfolio Assessment, pages 12-13) that the model does not reflect limitations on the use of DR – it assumes DR is available regardless of the day of the week.  This is a weakness in the model.  Ideally, the model would not only reflect DR’s availability based on the day of the week, but also on the time of day.  Moreover, there likely is a strong correlation between some kinds of DR and load draws – weather-sensitive DR is not likely to be available on low-load days.  While DR may be assumed to be less needed on low-load days, it nevertheless could be needed on low-load days depending on other factors, like the generation outage draw. 

During the November 18 WECC RA Forum, John Fazio observed that using historical data to inform models may improperly bias the results if the future conditions are expected to be dissimilar to the historical conditions.  The example he cited was expected changes in climate turning the Pacific Northwest from a winter peaking system to a summer peaking system.  These kinds of “the past may not be the best indicator of the future” changes may be more impactful in long-range planning studies, but the CAISO should be cognizant that historical data, especially data taken from a long period of past history, might not be the best data to use to inform its portfolio assessment.  Given that California has experienced three what were deemed to be improbable “tail event” heat waves in the past 15 years (2006, 2017 and 2020), the stochastic variable should be configured to capture the increasing possibility of such heat waves going forward.  

The CAISO notes (at page 9) that off-peak imports were allowed to reach historical levels (11,666 MW).   How does that align with the representation that the portfolio analysis was done using RA shown resources only?   MRP requests clarification on this point. 

MRP agrees with using similar hydro year profiles.  MRP requests that the CAISO provide some indication as to how much these “similar” year hydro values can vary in the critical late summer months for the same snowpack and water content values in the spring. 

6. Provide your organization’s feedback on the results topic as described in section 5:

MRP shares the CAISO’s surprise that the “Thermal Scenario” performed slightly worse than the “RA Showing” portfolio.  MRP agrees that the likely cause of this is how solar was represented in the “RA Showing” portfolio and how the CAISO determined how much gas-fired generation was used to replace the intermittent resources.  The fact that that metrics for the Thermal Scenario and RA Showing are significantly different – for example, that the Thermal Scenario shows a much greater frequency of shortfalls than the RA Showing[1] – suggests that the amount of thermal generation used to replace the intermittent generation may not have been, in fact, a functional equivalent.  Further, as the CAISO notes, allowing DR to be available in hours in which it typically is not available (e.g., the late net load peak hours) may have also contributed to this surprising result.   If the CAISO intends to study thermal scenarios as part of its ongoing portfolios analyses, the topic of how the CAISO determines how much thermal generation is used to replace intermittent generation warrants additional discussion.  

[1] Portfolio Analysis at Figure 3. 

7. Provide your organization’s feedback on the interim needs topic as described in section 6:

MRP strongly supports the CAISO’s proposal to (1) develop and enforce a net load peak capacity requirement and (2) retain the gross load peak requirement.    The events of August 14 and 15 clearly demonstrate that an RA program focused exclusively on the gross load peak does not maintain reliability in all hours in light of (not intended as a sunset pun) the changing resource mix. 

8. Provide your organization’s feedback on the proposed foundational framework as described in section 7:

In this section, the CAISO identifies two core challenges:

(1) Establishing a defined reliability criteria or loss-of-load expectation that determines procurement targets and backstop procurement trigger,

(2) Determining the quantity and attributes of capacity needed to address a portfolio deficiency.

The CAISO identifies three questions arising from those two challenges:

• What is the correct granularity for the RA program: annual, seasonal or monthly? granularity?
• How do you apply an annualized planning standard to an RA program that is not implemented on an annual granularity?
• What is the desired service level reliability target?

MRP agrees that these are key questions to ask as the parties reconsider the RA program in Track 3.B of the CPUC’s RA proceeding (R.19-11-009).  MRP offers a few thoughts on these challenges and issues.

MRP suggests adding a fourth two-part question to the three questions asked by the CAISO: what is the appropriate margin to apply to procurement, and what uncertainties is that margin expressly intended to cover? 

With regards to granularity of the RA program, MRP strongly supports reconsideration of the RA program as an annual program, if for no other reason than an annual program would simplify the CAISO’s very complex outage coordination process. 

With regards to “spreading” the annualized standard across a program of less-than-annual granularity, MRP offers from painful experience that a loss-of-load event in August likely has radically different societal consequences than a loss of load event in March.  Shedding load in August carries different burdens – the loss of critical cooling and the attendant challenges that can raise for affected persons – that are not likely to be introduced by shedding load in March or December.[1]  On that basis, applying the annualized planning standard uniformly across all months may not be optimal from a societal perspective. 

MRP notes that RA is just one component of service reliability.   The author of these comments has had the great (mis)fortune to have had power intentionally turned off in 2020 both because of RA issues (the Stage 3 emergency on August 14) and because of network infrastructure issues (in additional to minor short-lived distribution-induced disruptions, a 22-hour Public Safety Power Shutoff event in October).  The fact that one event was due to distribution-level issues and one event was due to bulk supply inadequacy provided no special solace or relief; both events were unwelcome.  In fact, the event caused by bulk supply inadequacy was probably more affecting, because it happened when outside temperatures were over 100 degrees.  In an ideal world, the risks of the two different types of interruptions and the costs of remediating a particular type of interruption would be weighed against each other so that non-infinite capital – which always is secured at ratepayer expense – could be most effectively deployed to mitigate outages.  While, in an ideal world, all the factors leading to service level unreliability would be weighed to best deploy limited capital, for the purposes of this exercise, MRP agrees that the right metric of service reliability is loss of load related to bulk supply inadequacy. 

[1] MRP acknowledges that loss of load events in December or March can cause loss of heating, but, given that there are typically more options for heating than for cooling, stands by the comment that loss of load in August is likely to be more affecting than loss of load in March or December.

9. Additional comments on the preliminary portfolio analysis:

MRP has no additional comments.  

1. Provide a summary of your organization's comments on the preliminary portfolio analysis:

Repurposing the Summer Assessment stochastic model without utilizing non-RA resources makes sense. NCPA has no other comments at this time regarding the technical nuances such as iterations or model details, rather NCPA’s comments focus on higher level areas of interim needs and proposed foundational framework which should be addressed prior to further technical discussion regardless. NCPA understands the need for net load requirements but requires more justification from CAISO for dual gross and net load requirements. NCPA believes there could be some acceptable small degree of load shedding, say, 1 in 20 events. NCPA prefers maintaining monthly granularity for RA fillings. Lastly, due to the late introduction of "interim needs" including a significant new proposal to include a net load requirement, NCPA believes the current stakeholder process timeline is infeasible. 

2. Provide your organization’s feedback on the Overview of the CAISO’s Production Simulation Model topic as described in section 4:

No comment at this time.

3. Provide your organization’s feedback on the defining “deficiency” topic as described in section 4.1:

No comment at this time.

4. Provide your organization’s feedback on the iterations and output topic as described in section 4.2:

No comment at this time.

5. Provide your organization’s feedback on the model details topic as described in section 4.3 and all relevant subsections:

No comment at this time.

6. Provide your organization’s feedback on the results topic as described in section 5:

No comment at this time.

7. Provide your organization’s feedback on the interim needs topic as described in section 6:

NCPA requests that CAISO thoroughly explain why both gross and net load requirements are necessary. The net load (gross load less wind and solar) has been the focus for the last five years or so, so a transition from gross to net requirements makes sense but a requirement to meet both at once would be extremely excessive, expensive, and complex. NCPA could support a transition from planning for gross load requirements to planning for net load requirements, but not an obligation to meet simultaneous net and gross requirements. 

8. Provide your organization’s feedback on the proposed foundational framework as described in section 7:

CAISO and the PTOs now routinely shed load for Public Safety Power Shutoff (PSPS) events for hours or even days at a time. NCPA has worked with CAISO and PTOs to help reduce the frequency and duration of such events through enhanced modeling and coordination. In light of these circumstances, NCPA believes that CAISO is correct to ask what degree or probability of loss of load is acceptable. Since cost must be a significant consideration in addition to reliability, NCPA believes that the loss of load probability should not be zero.  A reasonable LOLP should help LSEs control costs, and those funds could be used to continue fortifying the grid. Stakeholders would expect CAISO, LRAs, and the CEC to perform an exhaustive coordinated analysis weighing procurement costs associated reducing LOLP versus the costs to the economy and quality of life associated with increasing the LOLP and provide recommendations. For example, procuring $100 million dollars in additional capacity to avoid a one hour outage in a 1 in 35 weather event may or may not be palatable to ratepayers.

NCPA prefers maintaining monthly granularity. The monthly showings offer LSEs needed flexibility to optimize portfolios. NCPA could be amenable to showing 90% of October system capacity in annual showings or otherwise adjusting Annual filings to support grid reliability by allowing CAISO to run a full RA assessment of summer months similar to how the summer assessment is run now.

9. Additional comments on the preliminary portfolio analysis:

Considering the significant implications that many of the concepts still being discussed may have on system reliability and planning standards, including ongoing discussion of migrating from a gross load to a net load requirement, NCPA believes that the overall project schedule and timeline associated with the Resource Adequacy Enhancements stakeholder initiative is overly ambitious (and should be reassessed). It is important that each of the complex concepts being discussed are fully considered to ensure any policies adopted are done rights, as compared to fast.

1. Provide a summary of your organization's comments on the preliminary portfolio analysis:

PG&E’s comments are summarized as follows:

1) PG&E requests CAISO enhance the portfolio analysis to meet the CAISO’s objectives: The CAISO’s stated goal is to establish the data needed to build the framework to determine: 1) The adequacy of a given portfolio and 2) How much additional capacity may be needed if the fleet is determined to be inadequate. The Portfolio analysis presented addresses the first goal (ex post assessment of a given portfolio).  However, CAISO needs to prioritize developing criteria so that LSEs and LRAs can determine the ex-ante requirements to ensure passage of the ex-post portfolio assessment test. Without such analysis the CAISO can neither define a line where backstop procurement will be needed nor propose how the UCAP requirements for system RA will be set up.  

2) PG&E believes the discussion and conclusions in the stakeholder meeting and MSC discussion go well beyond the initial objective, and CAISO should clearly lay out how this analysis will be used and how that use furthers the resource adequacy objectives for California: CAISO’s stated objective seems clear “to provide a framework to consider how to derive answers to these questions”: (1) Establishing a defined reliability criteria or loss-of-load expectation that determines procurement targets and backstop procurement triggers; and (2) Determining the quantity and attributes of capacity needed to address a portfolio deficiency. However, the CAISO concludes that “the study results and recent reliability events confirm the need to take interim measures” and suggests making changes to the foundational framework of RA program. This conclusion is well out of scope from the objective. Several of the CAISO's conclusions are not supported as a part of the analysis and require more analysis and policy discussion before advancing forward.

3) PG&E believes the CAISO should enhance the collaboration and coordination with the CPUC and other Local Regulatory Authorities, CEC and IOUs: The CAISO has not yet collaborated with any other party on this analysis and is only sharing it with other parties at this time. The CAISO should provide clarity on the next steps to complete the current analysis and conclusions as well as clarity on the process to work with other parties (IOUs, CPUC, etc.). 

4) PG&E also believes correlation and sensitivity analysis on load forecast, imports, batteries are needed, and the CAISO should provide further information on the correlations used between the variables used in the model. Assumptions around imports and storage in particular need additional discussion to judge the appropriateness of the modeling.

2. Provide your organization’s feedback on the Overview of the CAISO’s Production Simulation Model topic as described in section 4:

PG&E believes the analysis provided raises many significant questions but neither reaches clear conclusions nor achieves the Resource Adequacy Enhancements initiative objectives so far.

As commented earlier (cf. June Working Group meeting comments), PG&E supports the development of a portfolio assessment for CAISO to be more transparent in its decision-making processes (e.g. for the criteria CAISO may use to justify a decision to use CPM backstop or RMR). PG&E continues to request additional details about the model and its potential use and believes a larger policy discussion is required in this area. Currently, how the model could be used to determine how much additional capacity (e.g., additional procurement of storage) may be needed has not been addressed by the CAISO.

PG&E believes greater consistency between the objectives, assumptions logic and design of the reliability assessment studies is needed to clearly define reliability needs. Additionally, CAISO should have additional discussion on the actions LSE’s, LRA’s or potentially CAISO could take to meet the identified needs.

3. Provide your organization’s feedback on the defining “deficiency” topic as described in section 4.1:

CAISO defines a deficiency as follows: Any hour in which the production simulation shows the CAISO would have to call a Stage Two Emergency. This means the model shows the CAISO would have inadequate capacity to meet the aggregate of non-spin, spin, regulation, and load.

The restrictive definition of deficiency automatically impacts the model outcomes and the assessed hours during which the CAISO is most likely in need of capacity and the duration of deficiencies.

It’s not clear how deficiency is defined in current LOLE or PRM analysis and how CAISO will derive from the deficiency definition reliability criteria and LOLE. PG&E requests clarification on these two questions.

4. Provide your organization’s feedback on the iterations and output topic as described in section 4.2:

PG&E would appreciate the CAISO provides explicit documentation for 1) model inputs, 2) model calculations and data transformation, and 3) model outputs.

PG&E believes the following questions should be clarified to better understand the outcomes of the model:

• What is the nature of the stochastic variation being modeled?  Is it load?  Resource availability?  Fuel availability by use limited resources? 
• How is the variation of each of these factors being measured? 

PG&E offers detailed comments on the model inputs in section 4 of the Portfolio analysis paper (question 5) and the presentation of the model results in section 5 (question 6 below).

5. Provide your organization’s feedback on the model details topic as described in section 4.3 and all relevant subsections:

Load inputs

PG&E requests further details on how to read the load data presented and will appreciate some clarity on the accuracy of the figure 1: distribution of peak loads used for July production.

CAISO explained that the model uses the exact same load inputs in both the RA showing and Thermal scenarios and that the load inputs are based on the CAISO’s load forecast process used for the CAISO summer assessment (Portfolio analysis paper, section 4.3.2, pages 9 and 10).

Then CAISO states that the figure 1 shows “the distribution of the daily peak loads used in all 2000 iterations for the month of July”. However, the graph shows the monthly peak and the occurrence of CAISO annual peak.

It’s not clear whether it represents the outcome of a month of the model iteration or a year (see also comments on the presentation of the results – question 6).

Resource inputs

• Battery assumptions:
  It seems that the existing storage was largely ignored in 2020 summer assessment. While that may be an acceptable assumption for small amounts of storage, there is a significant amount of capacity coming online by Aug 1, 2021.  Storage will likely be a resource the CPUC and other LRAs look to for additional procurement if shortages are identified. 
  At the June working group meeting, stakeholders requested additional details about the model’s ability to model storage. This important topic was not considered in the Portfolio assessment the CAISO published on November 6.
   
  PG&E requests clarification on how the CAISO will assess the operation of storage particularly, the charging and discharging of the capacity and how this information will be incorporated in the model. PG&E acknowledges the challenge of modelling both charging and discharging capacity for storage. Should the alternative be to model storage only in net-peak load hours once batteries have been fully charged?
   
• Import assumptions:
  In the section 4.3.1 CAISO system, CAISO specifies that “net imports into the CAISO are limited to the amount of RA imports shown each month during on peak periods (hour-ending 16-21). Net imports during off peak hours (hour-ending 1-15, 22-24) are allowed to historical off-peak levels, currently 11,666 MW”. However, in the resource inputs (section 4.3.3), CAISO considers only the shown RA value.

The CAISO’s 2020 summer assessment assumed many more imports were available than were shown as RA.  CAISO has not provided any good justification for using only the amount of import RA in the model. During the August events, we could see little correlation between the amount of imports showing up over a given intertie and the amount of RA. While CAISO is trying to strengthen the requirements for RA import contracts as a part of this initiative, it is unclear yet how effective those provisions will be. Additionally, CAISO has not provided any reasonable justification for the assumption that no additional energy will show up from imports besides the amounts designated as RA.

6. Provide your organization’s feedback on the results topic as described in section 5:

Results interpretation

The CAISO’s goal is to establish the data needed to build the framework to determine:

• The adequacy of a given portfolio; and
• How much additional capacity may be needed if the fleet is determined to be inadequate.

The results presented could be considered an ex-post analysis measuring the adequacy of a given portfolio. As CAISO clearly explained the “stochastic production simulation provides a distribution of potential outcomes and probabilities, not yes no answer.

PG&E agrees with the CAISO that the portfolio analysis addresses the first goal by providing an approach for calculating the probability of shortfall of a given portfolio (ex post analysis). However, the analysis can’t determine how much capacity is needed in advance.

At this stage the CAISO has not addressed the ex-ante determination of the required capacity to meet the deficiency tests nor stated how they would set the requirements to ensure passage of the ex-post portfolio assessment test. 

PG&E urges the CAISO clarify how the “ex ante” analysis will used to define additional reliability needs and the criteria where the CAISO will backstop in case of capacity deficiency.

The ex-ante analysis should address the following goals: reliability of the California system and affordability for California customers. The CAISO should demonstrate how much additional capacity is needed and the cost associated to ensure the reliability of the system before taking any extra capacity procurement decision.

Based on the ISO’s study results, the July 2020 RA showing would provide for approximately a three percent loss of load expectation (LOLE). This probability translates to a 0.93 days LOLE in July. If July is representative of all 12 months, this would result in an equivalent of 10.95 days LOLE for the year.

PG&E believes this result should be interpreted with caution, the outcomes being obviously depend on the assumptions used and represent RA only resources for the month of July 2020. As stated above it’s not clear how the ISO will determine the quantity and attributes of capacity needed to address a portfolio deficiency.

Definition of UCAP requirements

PG&E also would like additional details on how the CAISO will derive potential UCAP requirements based on the resource inputs used in the model. In the resource inputs section, the CAISO specifies that the model uses resources NQCs value and the simulation model generates a “unique forced outage profile for each of the 2000 simulation scenarios based on historical forced outage rates from OMS”.

• What would be the model outcomes if the CAISO were using resources UCAP values as currently discussed in the RA Enhancements initiative instead of NQCs and forced outage rates?

Additional analysis: correlations used between the variables

PG&E would appreciate more transparency on the results of the presentation, particularly a discussion of the correlation across draws from the various stochastic variables used at this stage. 

CAISO said at the June working group meeting (slide 8) that the model can be run both stochastically and deterministically, allowing CAISO to develop robust statistical results while testing various sensitivities.

PG&E requests the CAISO provide supplemental analysis that will test sensitivities around the RA showings to help inform UCAP requirements.

• As stated in the Portfolio analysis paper, CAISO will conduct further modeling using other months’ RA showings. PG&E requests the CAISO provides, in the expected complementary study, analysis to test sensitivity of higher or lower imports; sensitivity to load forecasts and storage; and specify what correlations are used between the variables.
• The CAISO should provide in the results of the analysis more links between the summer assessment and this analysis and explain the differences in the conclusions of both studies.
• PG&E also requests to see more analysis on capacity in California that doesn’t fully qualify as RA (due to not meeting the RA deliverability requirements). Should such resources be modeled as zero for the RA portion or are different assumptions more appropriate given operational experience?  

Lastly, on the presentation of the results, instead of presenting the results as number of iterations PG&E believes all probabilities should be presented within a confidence interval.

7. Provide your organization’s feedback on the interim needs topic as described in section 6:

The CAISO concludes that “the study results and recent reliability events confirm the need to take interim measures” and considers that in addition to the current gross peak load requirements, a secondary net-load peak requirement is needed.

Based on the scenarios used (July RA 2020 showings and thermal scenario), the assumptions considered, the graph frequency of observed deficiencies by hour shows that more shortfalls may occur hour 16 to 19 in the Thermal scenario; the RA showing shortfall occurrences are observed mainly at hours 19-21.

To better understand this outcome, PG&E would welcome more sensitivity analysis on load, import, solar and battery assumptions, as requested in questions 4, 5 and 6.

Also, PG&E suggests these interim needs are re-assessed once CAISO has conducted further modelling using other months’ RA showings to complete the picture about how likely the RA fleet meets grid reliability needs across all months.

8. Provide your organization’s feedback on the proposed foundational framework as described in section 7:

As raised many times in the above sections, PG&E’s believes the analysis provided doesn’t address the two-fold objectives set by the CAISO. PG&E believes only once the analysis addresses the objectives should CAISO work with appropriate agencies to go beyond and discuss the RA foundational framework: Answering the following questions raised by CAISO requires better coordination with LRAs.

1. The correct granularity of the RA program: Annual, Seasonal, or Monthly? 

2. The application of an annualized planning standard 

3. The desired service level reliability target 

The CAISO has not collaborated with any other party on this analysis and is only sharing it with other parties at this time. The CAISO should provide clarity on the next steps to complete the current analysis and conclusions as well as clarity on the process to work with other parties (IOUs, CPUC, etc.).

9. Additional comments on the preliminary portfolio analysis:

None.

1. Provide a summary of your organization's comments on the preliminary portfolio analysis:

Please see Powerex's comments at https://powerex.com/sites/default/files/2020-11/2020-11-25%20Powerex_RA_Resource_Portfolio_Comments_0.pdf

2. Provide your organization’s feedback on the Overview of the CAISO’s Production Simulation Model topic as described in section 4:

Please see Powerex's comments at https://powerex.com/sites/default/files/2020-11/2020-11-25%20Powerex_RA_Resource_Portfolio_Comments_0.pdf

3. Provide your organization’s feedback on the defining “deficiency” topic as described in section 4.1:

Please see Powerex's comments at https://powerex.com/sites/default/files/2020-11/2020-11-25%20Powerex_RA_Resource_Portfolio_Comments_0.pdf

4. Provide your organization’s feedback on the iterations and output topic as described in section 4.2:

Please see Powerex's comments at https://powerex.com/sites/default/files/2020-11/2020-11-25%20Powerex_RA_Resource_Portfolio_Comments_0.pdf

5. Provide your organization’s feedback on the model details topic as described in section 4.3 and all relevant subsections:

Please see Powerex's comments at https://powerex.com/sites/default/files/2020-11/2020-11-25%20Powerex_RA_Resource_Portfolio_Comments_0.pdf

6. Provide your organization’s feedback on the results topic as described in section 5:

7. Provide your organization’s feedback on the interim needs topic as described in section 6:

8. Provide your organization’s feedback on the proposed foundational framework as described in section 7:

9. Additional comments on the preliminary portfolio analysis:

Please see Powerex's comments at https://powerex.com/sites/default/files/2020-11/2020-11-25%20Powerex_RA_Resource_Portfolio_Comments_0.pdf

1. Provide a summary of your organization's comments on the preliminary portfolio analysis:

Cal Advocates appreciates the opportunity to comment on the CAISO’s preliminary portfolio analysis and focuses on the following issues:

• The CAISO’s definition of a deficiency does not align with the definition used by the Western Electricity Coordinating Council (WECC), and the CAISO should instead define deficiency as a Stage Three Emergency to appropriately calibrate risks.
• Cal Advocates agrees that a net-load peak requirement could improve reliability as California continues to transition to a decarbonized grid.  However, the new net-load peak requirement would exist simultaneously with the current gross-load peak requirement.  The CAISO should establish the conditions for when the gross-load requirement is no longer necessary.  Likewise, Cal Advocates urges the CAISO to coordinate closely with the relevant California Public Utilities Commission (CPUC) proceedings to avoid acting at cross-purposes in the next few years.
• Finally, Cal Advocates supports maintaining the existing monthly planning standard of the resource adequacy program;  responds to the CAISO’s annualized reliability standard; and recommends that any reconsideration of the desired service level reliability target must involve other stakeholders.

2. Provide your organization’s feedback on the Overview of the CAISO’s Production Simulation Model topic as described in section 4:

Cal Advocates has no feedback on this issue at this time.

3. Provide your organization’s feedback on the defining “deficiency” topic as described in section 4.1:

The CAISO states that the dual purposes of this Preliminary Portfolio Analysis paper are:

(1) Establishing a defined reliability criteria or loss-of-load expectation that determines procurement targets and backstop procurement triggers; and

(2) Determining the quantity and attributes of capacity needed to address a portfolio deficiency.[1]

For the purposes of this analysis, the CAISO defines a deficiency as:

Any hour in which the production simulation shows the CAISO would have to call a Stage Two Emergency. This means the model shows the CAISO would have inadequate capacity to meet the aggregate of non-spin, spin, regulation, and load.[2] 

Just prior to this definition, the CAISO notes that, “At a Stage Two Emergency the CAISO begins using spinning reserves to serve load and sets up firm load to be shed as contingency reserve in its place” [emphasis added].[3]  When the CAISO declares a Stage Two Emergency, alternative emergency actions may still be executed, such as additional dispatch of available resources.[4]  An actual loss of load event due to load shedding does not occur unless the CAISO advances to a Stage Three Emergency.[5]

Since a Stage Two Emergency does not cause a loss of load event, the CAISO should modify its definition of a deficiency to be when a Stage Three Emergency is declared.  A Stage Two Emergency represents a milestone before any load loss occurs, unlike a Stage Three Emergency in which load shedding is ordered.  Additionally, using a Stage Two Emergency to define a loss of load event is not consistent with the WECC’s definition of Loss of Load Expectation.[6]  The CAISO should maintain its alignment with WECC requirements that guide the CAISO’s System Emergency Operating Procedure.[7] The goal of this stakeholder process should be to prevent load shedding, not to prevent a Stage Two Emergency.

Moreover, the CAISO should not ignore the costs that would be incurred by ratepayers to mitigate a Stage Two Emergency that itself may not lead to a loss of load event.  As the CAISO itself notes, the RA program is designed to tolerate a certain probability that load shed may ultimately occur.[8]  The CAISO should consider that its focus on   preventing a Stage Two Emergency could create significant unnecessary costs to ratepayers, and should instead define deficiency as a Stage Three Emergency.

[1] Resource Adequacy Enhancements: Supplemental Report – Preliminary Portfolio Assessment, November 6, 2020 (Supplemental Report), p. 3.

[2] Supplemental Report, p. 9. Metaphorically, a Stage Two emergency is having less money in a checking account than necessary to cover monthly expenses, while a Stage Three emergency is incurring an overdraft charge. Stage Two is manageable, but Stage Three is inarguably problematic.

[3] Ibid, p. 9.

[4] CAISO Operating Procedure 4420, Section 3.5 available at: https://www.caiso.com/Documents/4420.pdf

[5] Ibid.

[6] The CAISO stage emergencies comply with WECC operating standards (Supplemental Report, p, 9), but the a Stage Two emergency does not align with the WECC’s definition of loss of load expectation: “The expected number of days in the year when the daily peak demand exceeds the available generating capacity.”  WECC Glossary, p. 73 available at: https://www.wecc.org/_layouts/15/WopiFrame.aspx?sourcedoc=/Administrative/04%20Glossary.docx&action=default&DefaultItemOpen=1. 

[7] CAISO Operating Procedure 4420, Section 4.

[8] Supplemental Report, p. 18.

4. Provide your organization’s feedback on the iterations and output topic as described in section 4.2:

Cal Advocates has no feedback on this issue at this time.

5. Provide your organization’s feedback on the model details topic as described in section 4.3 and all relevant subsections:

Cal Advocates has no feedback on this issue at this time.

6. Provide your organization’s feedback on the results topic as described in section 5:

Cal Advocates has no feedback on this issue at this time.

7. Provide your organization’s feedback on the interim needs topic as described in section 6:

Cal Advocates affirms the CAISO’s assertion that, “a sole resource adequacy procurement target focused on the gross peak load misses the growing and more urgent reliability needs that occur during the net-load peak and the hours immediately following.”[1]  However, Cal Advocates is concerned with the CAISO’s assertion that, “a secondary resource adequacy reliability requirement must be promptly instituted… across the net-load peak hours.”[2] 

The CAISO states that, “[t]he two requirements would stay in place until there is additional information and vetting of these portfolio assessment results to inform and develop a more comprehensive measure for resource adequacy.”[3]  Cal Advocates urges caution when imposing new overlapping requirements, for two reasons.  First, in order to avoid creating another major open-ended program requirement, the CAISO should define the circumstances under which it will repeal the gross load peak requirement. The CAISO should specify the additional information and vetting that is needed to establish that the gross load peak requirement is superfluous. 

Second, other regulatory processes are working in parallel to address the needs that the CAISO has identified.  For example, Track 3.B of the RA proceeding at the CPUC is considering whether the RA program should be re-structured around energy requirements.[4]  Likewise, the CPUC instituted a new Rulemaking (R. 20-11-003) on November 19 to examine emergency measures to ensure reliability in the case of another extreme heat event in 2021.[5]  A reliability construct organized around energy requirements may address the same underlying problems that a net-load peak requirement seeks to solve.  Given the short-term measures under consideration in the CPUC’s new rulemaking and the potential for major long-term adjustments to California’s resource adequacy program, the CAISO should explain why adding a new capacity requirement is necessary.  

[1] Supplemental Report, p. 18.

[2] Supplemental Report, p. 18.

[3] Supplemental Report, p. 18.

[4] With regard to the timing of these structural refinements, CPUC Energy Division’s presentation during the Track 3.B Workshop – Day 1 stated that 2023 “may be more realistic” than 2022 and 2024 is “most realistic.” CPUC Track 3.B Workshop: RA Structural Changes, November 18, 2020,  https://www.cpuc.ca.gov/uploadedFiles/CPUC_Public_Website/Content/Utilities_and_Industries/Energy/Energy_Programs/Electric_Power_Procurement_and_Generation/Procurement_and_RA/RA/Track%203B%20Day%201%20presentation.pdf

[5] See CPUC Voting Meeting November 19, 2020 item 35.  The agenda is available at: https://docs.cpuc.ca.gov/PublishedDocs/Published/G000/M351/K624/351624237.pdf.  The voting results are available at: https://ia.cpuc.ca.gov/agendadocs/3475_results.pdf.

8. Provide your organization’s feedback on the proposed foundational framework as described in section 7:

The CAISO identifies the following three issues as critical to address the challenge of, “Establishing a defined reliability criteria or loss-of-load expectation that determines procurement targets and backstop procurement trigger.”[1]  Cal Advocates responds to each of three issues identified by the CAISO:

1. What is the correct granularity of the RA program: Annual, Seasonal, or Monthly?

The current monthly granularity of the RA program provides the correct level of precision for guiding reliability procurement targets.  Using monthly targets allows California to procure appropriate capacity to meet forecasted demand which varies significantly month to month.[2]  Implementing an annual or seasonal granularity standard may result in over-procurement during times of the year when there is less demand for energy.

2. Apply an annualized reliability standard

The CAISO suggests at least two methods for establishing an annualized reliability standard: “uniformly [or] shaped.”[3]  A uniform annual standard would evenly distribute the probability of a loss of load across all twelve months, much like the current planning reserve margin adds an additional 15 percent to each month’s RA requirements to prevent loss of load.  On the other hand, a shaped reliability standard would maintain the same level of aggregate risk but distribute it unevenly across the year, with on-peak months having higher risk of deficiency and off-peak months with lower risk.  In extremis, this would mean eliminating the probability of deficiencies in off-peak months and instead concentrate all deficiency risk into on-peak months.  However, a shaped reliability standard would appear to exacerbate deficiency risk during on-peak months and lead to over-procurement of capacity for off-peak months with no obvious reliability benefits. 

The CAISO should provide further explanation of the potential benefits associated with a shaped reliability standard.

3. What is the desired service level reliability target?

The service level reliability target is a foundational component of any RA program and should not be reassessed by the CAISO alone.  Rather, the CPUC, California Energy Commission, and other stakeholders must coordinate in any effort or process that would alter this fundamental piece of the RA program.  Cal Advocates recommends that the CAISO model reliability scenarios that include, at a minimum, the status quo RA reliability target structure.

[1] Supplemental Report, p. 19.

[2] CPUC 2018 Resource Adequacy Report, August 2019, Table 1 available at: https://www.cpuc.ca.gov/uploadedFiles/CPUC_Public_Website/Content/Utilities_and_Industries/Energy/Energy_Programs/Electric_Power_Procurement_and_Generation/Procurement_and_RA/RA/2018%20RA%20Report%20rev.pdf.

[3] Supplemental Report, p. 20.

9. Additional comments on the preliminary portfolio analysis:

Cal Advocates has no additional comments at this time.

1. Provide a summary of your organization's comments on the preliminary portfolio analysis:

SDG&E appreciates the opportunity to provide comments on the preliminary portfolio analysis.  At this time, SDG&E is unable derive any conclusions from the preliminary analysis.    The current analysis does not provide information that would indicate whether a different mix of resources could resolve the deficiency.  SDG&E encourages the CAISO to perform additional analysis for other historical RA months to better understand the overall cause of the identified deficiency. SDG&E believes the CAISO should include in its analysis the reliability contribution of all resources/imports, not just the shown RA resources, in order to understand the level of reliability that can be achieved.  This will allow stakeholders as well as the CAISO to understand the results of the July RA showing relative to that of the entire resource fleet.

2. Provide your organization’s feedback on the Overview of the CAISO’s Production Simulation Model topic as described in section 4:

SDG&E requests the CAISO to provide additional information describing the distribution of deficiencies relative to the load scenarios used.  Specifically, SDG&E is requesting that the CAISO provide a graph that shows the simulated monthly peak loads (MW) on the horizontal axis, and the corresponding maximum monthly system deficiency (MW) on the vertical axis for all deficiencies that occurred.  SDG&E understands that for the month of July there were 2000 different hourly load patterns tested, each pattern having its own monthly peak.  Such information would be helpful to understand when the deficiencies occurred relative to the 1-in-2 load forecast.  If the majority of deficiencies fall outside of the 1-in-2 load forecast, SDG&E believes that stakeholders should discuss whether the RA program should revise its planning standard to cover a higher load forecast scenario and the impacts to customers under such a standard before deciding that a higher planning standard must be adopted.

3. Provide your organization’s feedback on the defining “deficiency” topic as described in section 4.1:

SDG&E believes CAISO should clarify how the proposed definition of deficiency compares to the current 1-in-10 loss of load expectation standard.  As currently proposed, it is difficult to understand how much more stringent the CAISO’s proposed definition for deficiency is relative to the current planning standard.  As the CAISO noted during its workshop, the proposed definition does not mean that the CAISO will take action to shed load because there is additional non-RA capacity available on the grid.  Therefore, the event of actual deficiency may be unlikely to occur except in certain cases.  At this time, such information is unknown because the CAISO has not analyzed the reliability of the existing fleet of resources.  Once that information is provided, stakeholders would then be able to compare the CAISO proposed deficiency standard with that of the current planning standard.

4. Provide your organization’s feedback on the iterations and output topic as described in section 4.2:

See SDG&E’s response to question 2 above.  Also, as suggested in SDG&E’s response to question 1 above, the CAISO should perform additional studies that (i) account for the contribution of non-RA resources (including non-RA imports) towards system reliability, and (ii) consider all months of the year, not just July.  Non-RA resources would have their own unique distribution of hourly availability. 

5. Provide your organization’s feedback on the model details topic as described in section 4.3 and all relevant subsections:

SDG&E is concerned that the reliability contribution of non-RA resources outside the CAISO balancing authority is not being properly accounted for.  It appears that resources outside the CAISO balancing authority are being economically dispatched in each monthly iteration assuming (i) expected load levels in non-CAISO balancing authorities, (ii) expected resource availability, and (iii) power flows into the CAISO are limited to the amount of RA imports shown each month during on peak periods (hour-ending 16-21).  This modeling approach fails to provide a reasonable distribution of potential imports into the CAISO balancing authority, thereby biasing—in unknown ways--the results of the much more robust modeling of loads and resources within the CAISO balancing authority. 

6. Provide your organization’s feedback on the results topic as described in section 5:

SDG&E is unable to interpret the results of the analysis.  It is unclear how, or if, the proposed UCAP framework would reduce such deficiencies.  SDG&E requests the CAISO to clearly outline how the results in the preliminary analysis helps inform the UCAP framework in its next proposal.

7. Provide your organization’s feedback on the interim needs topic as described in section 6:

SDG&E believes it is premature to establish interim requirements because the underlying problems are unknown at this time.  As noted above, the analysis above does not support the CAISO’s proposal for the UCAP framework because it is 1) unclear if the deficiencies being analyzed are the same as a loss of load expectation, 2) it is unclear whether the month of July is representative of the rest of the year, and 3) it is unclear if a different mix of resources that had better UCAP values can resolve the deficiencies.  SDG&E believes the CAISO needs to perform additional analysis to identify the underlying issues before devising solutions.

8. Provide your organization’s feedback on the proposed foundational framework as described in section 7:

SDG&E believes the CPUC has the responsibility to set planning standards for its jurisdictional LSEs.  The CAISO should work collaboratively with the CPUC to provide the results of this analysis as well as other future analysis with the CPUC to ensure reliability.  In this instant analysis, while the CAISO states that additional capacity is needed to lower the probability of deficiency, as noted in our response to #7, it is 1) unclear if the deficiencies being analyzed are the same as a loss of load expectation, 2) it is unclear whether the month of July is representative of the rest of the year, and 3) it is unclear if a different mix of resources that had better UCAP values can resolve the deficiencies.  If in fact any type of capacity can resolve the deficiency, then it seems that the UCAP framework may be unnecessary.

9. Additional comments on the preliminary portfolio analysis:

As indicated in SDG&E’s response to question 1 above, a fundamental flaw in the CAISO’s reliability analysis is that it specifically excludes the reliability contribution of non-RA resources.   SDG&E recognizes that there are uncertainties associated with the reliability contribution of non-RA resources, but these uncertainties can be brought into the analysis such that a comprehensive assessment of CAISO balancing authority reliability can be performed.   SDG&E understands that non-RA resources “do not have the same availability obligation as the modeled RA resources.”[1] Nevertheless, non-RA resources have strong financial incentives to produce power when it is economic to do so and this output needs to be accounted for when determining the amount of dependable capacity that CAISO load serving entities need to contract for.

[1] CAISO’s November 6, 2020 Resource Adequacy Enhancements: Supplemental Report – Preliminary Portfolio Assessment, Supplement to the Fifth Revised Straw Proposal, p. 7.

1. Provide a summary of your organization's comments on the preliminary portfolio analysis:

The Six Cities appreciate the CAISO’s development and presentation of the Preliminary Portfolio Assessment and related identification of questions that require further analysis and discussion.  As a preliminary assessment, the analysis provides useful (but not comprehensive) information to guide further evaluation.  As discussed below, the Six Cities reiterate their view that the CAISO should supplement its analysis of RA portfolio sufficiency with additional analysis of overall fleet sufficiency, at least for months when it appears that RA showings may not be adequate.  The Six Cities support further evaluation of net-load peak RA requirements but disagree that a uniform planning standard must apply for all LRAs within the CAISO.

2. Provide your organization’s feedback on the Overview of the CAISO’s Production Simulation Model topic as described in section 4:

The Six Cities generally support the use of the production simulation tool used for the Summer Loads and Resources Assessment as the model for assessment of RA portfolio sufficiency.  However, as the Six Cities understand how the tool was applied for the Preliminary Portfolio Assessment, it appears that refinements are likely to be necessary to reflect features of the RA program that change as a result of the RA Enhancements initiative.

For example, the Six Cities understand that the Preliminary Portfolio Assessment was based on NQC values for the resources included in July 2020 RA showings.  It would be helpful if the CAISO explained what changes to the application of the model would be appropriate if future RA showings are based on UCAP values.  The Preliminary Portfolio Assessment indicates at page 12 that the model applied a forced outage profile for each of the simulation scenarios based on historical forced outage rates.  Because UCAP values already reflect forced outages, it would not seem appropriate to include a forced outage assessment in applying the model to UCAP resource values.

The Six Cities request that the CAISO explicitly identify and analyze interdependencies between elements of revised RA requirements and the sufficiency assessment model and explain how the model takes such interdependencies into account.

3. Provide your organization’s feedback on the defining “deficiency” topic as described in section 4.1:

The Six Cities take no position at this time with regard to the most appropriate definition of “deficiency” but agree that this is a threshold question that requires further evaluation.   However, by adopting this definition, this CAISO study is explicitly attempting to determine the (conservative) stochastic probabilities of Stage Two Emergency events.  In section 7, the CAISO equates this with “service level reliability”, which is in turn equated with controlled load shedding.  However, for reasons discussed below, the Six Cities disagree that these results provide unbiased estimates of the true probabilities associated with controlled load shedding events.

4. Provide your organization’s feedback on the iterations and output topic as described in section 4.2:

The Six Cities have no comments on this element of the Preliminary Portfolio Assessment as described in Section 4.2.

5. Provide your organization’s feedback on the model details topic as described in section 4.3 and all relevant subsections:

As noted in their previous comments in this initiative, the Six Cities support analysis of resource sufficiency based on RA showings as an initial and primary analysis but oppose the CAISO’s determination to limit application of the portfolio sufficiency model to RA resources under all circumstances.

The Six Cities are concerned that basing the portfolio sufficiency test solely on production modeling limited exclusively to shown RA resources is likely to be overly conservative and lead to unnecessary and/or unduly costly incremental procurement.  Because non-RA resources do not take on any forward obligation to make themselves available to the CAISO BAA during any specific periods, it makes sense to conduct a portfolio sufficiency test based solely on the production capability of shown RA resources as one metric of the portfolio sufficiency analysis.  But non-RA resources within the CAISO BAA do have obligations under the Participating Generator Agreement to comply with all applicable provisions of the CAISO Tariff (Participating Generator Agreement Section 4.2), including, inter alia, (1) responding to Exceptional Dispatch instructions when they are able to do so (Tariff Section 34.11.1), (2) informing the CAISO of changes in operational status (Tariff Section 4.6.1.1), and (3) complying with outage management requirements under Tariff Section 9, including obtaining CAISO approval for planned outages (Tariff Section 9.3.2.).  Further, all Participating Generators are subject to CAISO control “to prevent an imminent or threatened System Emergency.”  (Tariff Section 7.7.2(c)(1).)  In the planning horizon, non-RA resources are subject to RMR designation by the CAISO (Tariff Section 41).  In light of the foregoing obligations of all Participating Generators, it seems unreasonable to completely ignore the capabilities of non-RA resources in evaluating portfolio sufficiency.  Such resources are not merely economic energy substitutes, as the Summer Assessment studies implicitly have recognized.

The Six Cities, therefore, recommend that the CAISO conduct the portfolio sufficiency tests based on both shown RA resources (both internal and external to the CAISO BAA) and on shown RA resources plus all non-RA resources within the CAISO BAA.  The broader test (i.e., all shown RA resources plus all non-RA resources within the CAISO BAA) would provide useful supplemental information on the severity of reliability risks arising from any deficiency identified by the narrower sufficiency test (based only on shown RA resources) as well as potential guidance on the most efficient approach for any supplemental RA procurement considered necessary.  Given an eventual run time of approximately three days as indicated during the November 12, 2020 stakeholder conference, it should be feasible to conduct the portfolio sufficiency tests based on both sets of resource inputs for every month.  At a minimum, however, the CAISO could perform the broader-based test in months for which the narrower test indicates a deficiency.  Along with the frequency distributions produced by the RA-only sufficiency analysis, the broader-based, all resources test would provide useful information to assist in determining the most cost-effective approach for addressing any deficiencies identified by the RA-only analysis.

6. Provide your organization’s feedback on the results topic as described in section 5:

The Six Cities support the inclusion of frequency distribution analyses in the results of the portfolio sufficiency assessment as providing useful information regarding the timing and duration of potential deficiencies to inform consideration of any incremental procurement to address deficiencies.

7. Provide your organization’s feedback on the interim needs topic as described in section 6:

The Six Cities generally agree that consideration of a net-load peak RA requirement is appropriate at this time but require further information on the details of any such proposed requirement before expressing a definitive position.

8. Provide your organization’s feedback on the proposed foundational framework as described in section 7:

The Six Cities agree that further analysis and evaluation are necessary with regard to the development of reliability targets.  Of particular importance here is the ability to accurately relate these conservative probabilities of capacity shortfalls to unbiased probabilities of actual load shedding events.

The Six Cities also do not agree with the CAISO’s premise, expressed at page 19 of the Preliminary Portfolio Assessment, that the reliability “standards must be made uniformly across all entities within the CAISO footprint.”  Contrary to the CAISO’s assertion, adoption of different standards by different LRAs does not necessarily result in “leaning.”  As the CAISO acknowledges, “[w]hen the CAISO system is stressed . . . grid operators . . . rely on the entire pool of resources within its footprint and across the WECC for help.”  There is no uniform planning standard applicable on a WECC-wide basis, and the proposed framework for the Extended Day-Ahead Market expressly disclaims any intention of imposing an EDAM-wide planning standard.  Given the coexistence of varying planning standards throughout the WECC, there is no justification for proposing to usurp the authority of LRAs by imposing a uniform planning standard within the CAISO Balancing Authority Area.

9. Additional comments on the preliminary portfolio analysis:

The Six Cities have no additional comments at this time.

1. Provide a summary of your organization's comments on the preliminary portfolio analysis:

SCE appreciates the CAISO’s effort in performing the portfolio analysis and supports the CAISO’s plan to perform the analysis for additional months. These analyses, when structured appropriately, can provide useful information, and inform discussions around reliability challenges the grid faces today. Based on the preliminary results from this analysis, SCE believes additional conversation and discussion on key aspects of the CAISO study approach are necessary before moving forward. These include:

• Jurisdictional consideration. The proper level of reliability for the resource adequacy (RA) program should be established by Local Regulatory Authority (LRA) including the CPUC. Any changes to the current RA reliability level should be evaluated under LRA/CPUC proceedings.
• The CAISO approach can lead to significant deviation from the CPUC’s RA program. How will the CAISO approach ensure RA needs are addressed upfront through LSE procurements?
• Various modeling assumptions should be closely examined, including whether those assumptions should be consistent among different but related efforts, including IRP, RA and the Summer Assessment.

SCE looks forward to further discussion on these topics as part of the stakeholder process.

2. Provide your organization’s feedback on the Overview of the CAISO’s Production Simulation Model topic as described in section 4:

While SCE is not necessarily opposed to the approach of evaluating RA based on a production simulation model, the approach should be thoroughly examined to ensure its consistency with the RA program. If there are any changes deemed necessary based on simulation results and related findings, those changes should first be submitted to LRA and CPUC for adoption. Without including those changes in the LRA/CPUC RA program, and instead setting RA requirements based on production simulation results could cause a significant deviation from the LRA/CPUC RA program.  Fundamentally, the RA programs of the LRAs have been established with rules that are set a priori and as such can be fully complied without after-the-fact review of the resources chosen.  The only element that this differs within is the local RA requirements in which effectiveness of the resources is determined after showings are made.  The paper on the CAISO portfolio assessment appears to apply a similar after-the-fact analysis for system and potentially flex RA resources.  As the CAISO works through this issue with stakeholders and the LRAs, particular attention should be provided to how such analysis can help to inform the RA requirements such that parties are able to comply with the RA needs with certainty rather than make best efforts only to be faced with the uncertainty of cure periods and backstop procurement.  In addition, if such learnings are integrated into the RA procurement requirements, the process will be better able to assign costs to those that cause them rather than perform backstop in a “cumulative deficiency” manner.

SCE is concerned that the CAISO’s study approach is not consistent with the RA program design. Specifically, due to various modeling assumptions, as discussed below, the study results capacity requirements and the reliability level that the RA program may not be designed to achieve. For instance, the preliminary simulation results show that the July 2020 RA showing would provide for approximately a 3% loss of load expectation (LOLE). The study seems to conclude that if July is representative of all months, then the 3% LOLE would be equivalent to 10.95 days LOLE for the year.[1] In other words, the CAISO study seems to conclude that the July RA showing is 100 times worse than the 1-in-10 LOLE standard[2]. The study also suggests that the CAISO would need to backstop 5,800MW of capacity to bring July to 1-in-10 LOLE[3]. SCE is concerned that the study includes several input assumptions that may not be intended by the RA program, as discussed below.  SCE notes that within the CPUC Track 3B process, the topic of planning reserve margin and more generally the level of reliability expected from the RA program is under discussion.  SCE believes that the study data put forth by the CAISO is helpful in examining the range of potential outcomes.

The definition of “deficiency”

The CAISO approach defines deficiency as any hour in which the production simulation shows the CAISO would have to call a Stage Two Emergency[4]. The CAISO should clarify if this definition is consistent with a LOLE definition that is commonly applied in production simulations. Since the CAISO models only the RA fleet, would this definition result in more deficiencies than the approach of applying a common LOLE definition to a system including both RA and non-RA resources?

Load forecast

The CAISO approach models load forecast levels that are significantly higher than the 1-in-2 load forecast. Specifically, the CAISO study models 1-in-10 load forecast, 1-in-20 load forecast (or 10% higher than the 1-in-2 load forecast), and a load forecast that is 9,000MW (or 20%) higher than the 1-in-2 load forecast[5].

The RA requirements today are based on 1-in-2 load forecast and a 15% planning reserving margin (PRM). It appears that the CAISO study models forced outages and reserves explicitly; therefore, to accommodate the 15% PRM, the CAISO study would only need to model load forecast error, which can be less than 2.5% (or equivalent to 1,100MW)[6]. This seems to imply that the CAISO study models scenarios where the peak load was up to 8,000MW higher than the peak load plus the 15% PRM that the RA program requires.

If the understanding of the load forecasts used in the CAISO study is correct, then the CAISO study essentially assumes that the RA program addresses the need to meet peak load forecast that is well beyond the 1-in-2 load forecast, which can lead to “deficiencies” tied to extreme level of load forecasts that the current RA program is not intended to address. If the CAISO believes that the current RA 1-in-2 load forecast should elevate to a different level, then the CAISO should seek a change under the CPUC RA proceeding for adoption.

The load forecast assumption, as well as other assumptions, as discussed below, may have led to significant “deficiencies” and the modeled need for backstop procurement. Under the CAISO approach, the CAISO is essentially requiring all LSEs to procure RA capacity to meet load forecasts that are well beyond the current RA requirements established by LRAs and CPUC. As mentioned above, if the CAISO believes that the current 1-in-2 load forecast standard or the 15% PRM should be elevated, then this should be addressed in the CPUC proceeding, since the RA program is in the LRA/CPUC’s jurisdiction.  

Resource input and energy requirement for RA

 The CAISO approach models resources’ production in meeting hourly load under various scenarios. For wind and solar, effective load carrying capability (ELCC) values as currently calculated by the CPUC are replaced with energy production of the resource. For other resources, individual resource constraints are modeled to the greatest extent possible[7]. External areas are also modeled, including 35 WECC zones and path limits between zones, while net imports into the CAISO are limited to the amount of RA imports.

 SCE agrees with the approach of using production data for wind and solar in the simulation. The approach of using production data for wind and solar resources instead of ELCC values likely will more accurately represent the contribution from those resources. SCE understands the CAISO’s motivation of including individual resources’ constraints in meeting hourly load. SCE believes that this information can be very helpful in evaluating the various models that have been proposed in the CPUC’s track 3B RA OIR to ensure that not only is the load/net load peak met but so are the energy needs of the system. 

Reliability metric and framework

Regarding discussions of reliability framework and appropriate reliability service level for RA, SCE believes that the discussion should also take place under the CPUC RA proceeding. There the discussion should address key issues including: the appropriate reliability service level that the RA program should be designed to achieve and key considerations for determining this reliability level and how to ensure this reliability level can be achieved through upfront RA procurement. While the preliminary assessment provides useful information, further discussion on these topics is essential.     

Other observations and issues

• The approach that CAISO adopted in this study might not be consistent with industry’s common practice. The probabilistic LOLE analysis is typically performed to determine the amount of capacity that needs to be acquired to meet a desired reliability target (industry practice of 0.1 days/year) as compared to the study’s inclusion of high/extreme load forecasts and Stage 2 Emergency in deriving a capacity shortfall curve in meeting the load forecasts.
• As an after-the-fact analysis, the study results indicate capacity shortage for July 2020 in order to avoid unserved load or unmet ancillary service requirements.  However, there was no stage-2 emergency issued by CAISO in July 2020. The study results may not be consistent with actual market conditions. The CAISO should provide information on which load scenarios the simulated deficiencies occurred.
• The CAISO should provide information on whether, and to what extent, deficiencies may be addressed by energy storage resources, since it appears most of the deficiencies are small and over 90% of the days with deficiencies, those deficiencies were less than four hours in duration. To the extent possible, the CAISO should consider simulating the effects of potential energy storage resources in addressing those deficiencies.[8]  In addition, instead of capping net imports at RA imports, the simulation may consider modeling those RA imports as a self-schedule or bid with negative bid prices per the latest rules for RA imports. The simulation should examine whether there are any exports from the CAISO to WECC areas under a deficiency.

[1] “However, based on the results from the CAISO’s study, the July 2020 RA showing would provide for approximately a three percent LOLE…. This would result in an equivalent 10.95 days LOLE for the year.” Report, at 20, available at http://www.caiso.com/InitiativeDocuments/PreliminaryPortfolioAnalysis-ResourceAdequacyEnhancements.pdf. 

[2] A 1-in-10 LOLE standard would be equivalent to a 0.1 day for the year. 10.95 days would be 100 times worse than 0.1 day for the day.

[3] Report, at 21.

[4] This means the model shows the CAISO would have inadequate capacity to meet the aggregate of non-spin, spin, regulation, and load.

[5] Figure 1 of the Report.

[6] CAISO Presentation shows a less than 2.5% of peak load forecast for July 2020, available at http://www.caiso.com/Documents/Presentation-MarketPerformance-PlanningForum-Sep9-2020.pdf, at page 55. Although this percentage may or may not be aligned with the 15% PRM when it was established, the information provides a useful reference.

[7] Resource confidential Masterfile parameters for each resource are modeled in the simulation, including minimum run time, minimum down and ramp rates. Presentation, at 23, available at http://www.caiso.com/InitiativeDocuments/Presentation-ResourceAdequacyEnhancements-Nov122020.pdf.

[8] It was also discussed during the November 13, 2020 MSC meeting that energy storage resources may reduce the frequency of deficiencies but increase their magnitude when deficiency does occur. Such effects should be quantified through simulation if possible.

3. Provide your organization’s feedback on the defining “deficiency” topic as described in section 4.1:

Please see comments in Section 2.

4. Provide your organization’s feedback on the iterations and output topic as described in section 4.2:

Please see comments in Section 2.

5. Provide your organization’s feedback on the model details topic as described in section 4.3 and all relevant subsections:

Please see comments in Section 2.

6. Provide your organization’s feedback on the results topic as described in section 5:

Please see comments in Section 2.

7. Provide your organization’s feedback on the interim needs topic as described in section 6:

SCE concurs with the CAISO on the need to address net load peak hours. The CAISO should provide additional information on how net load requirement will be set as an interim measure suggested in the analysis report. SCE/CalCCA has submitted a joint proposal under the CPUC’s Track 3B of the RA OIR. Within that proposal, there will be an explicit requirement to meet net load peak. The CAISO should clarify whether the CAISO is referring to SCE/CalCCA joint proposal or similar proposals as a method to set an explicit net load requirement.

8. Provide your organization’s feedback on the proposed foundational framework as described in section 7:

Please see comments in Section 2.

9. Additional comments on the preliminary portfolio analysis:

1. Provide a summary of your organization's comments on the preliminary portfolio analysis:

Overview of RA Enhancements Position

The Western Power Trading Forum (WTPF) again respectfully asks the CAISO to consider rescoping this initiative to respond to California’s capacity crisis. The summer blackouts have highlighted gaps in California’s dual Resource Adequacy (RA) programs, and WPTF believes the RA Enhancements proposal overall is a step backward not forward for reliability and market efficiency. The proposals main features – UCAP, local RA, the monthly portfolio assessment, and the planned outage process will reduce transparency, increase transactions costs, and, most concerningly, move away from coordination with the California Public Utilities Commission (CPUC).

WPTF believes CAISO should devote more resources to ensuring a robust RA program that is transparent, efficient, and most importantly, coordinated with the CPUC’s Integrated Resource Planning (IRP) and RA programs. The RA Enhancements initiative scope, which include the issues CAISO identified as driving the need for the RA Enhancements initiative, were made public in October 2018 – over two years ago. A lot has changed in the last two years from a reliability and markets perspective. WPTF urges CAISO and stakeholders alike to step back and revisit the Issue Paper (page 3) to ensure the overall driving needs for this stakeholder effort are vetted to reflect the current environment and updated as appropriate. WPTF further urges the CAISO to vet the proposed solutions with stakeholders based on the needs identified in any updated problem statement.

Overview of Portfolio Assessment Position

WPTF appreciates the CAISO’s hard work to test the portfolio study approach in advance of the draft final proposal. The findings are interesting and demonstrate the significant time and effort that it takes to assess the reliability of the grid. WPTF believes the results also demonstrate how impactful different input assumptions will be on the likelihood of the backstop triggering additional procurement.

WPTF continues to have significant concerns with the portfolio assessment. While there is stakeholder consensus that the CAISO and CPUC should coordinate across forward and backstop mechanisms to meet reliability needs while balancing costs; it is not clear whether the portfolio assessment achieves this outcome. WPTF believes the CAISO still needs to demonstrate a need for this new backstop capacity authority to incrementally improve reliability in a manner that balances costs and reliability benefits. Notably WPTF still has specific questions as to the (1) alignment with LRA’s and CPUC’s forward procurement processes and (2) study transparency and analytical rigor to demonstrate reliability need. 

2. Provide your organization’s feedback on the Overview of the CAISO’s Production Simulation Model topic as described in section 4:

3. Provide your organization’s feedback on the defining “deficiency” topic as described in section 4.1:

4. Provide your organization’s feedback on the iterations and output topic as described in section 4.2:

5. Provide your organization’s feedback on the model details topic as described in section 4.3 and all relevant subsections:

6. Provide your organization’s feedback on the results topic as described in section 5:

7. Provide your organization’s feedback on the interim needs topic as described in section 6:

8. Provide your organization’s feedback on the proposed foundational framework as described in section 7:

9. Additional comments on the preliminary portfolio analysis:

Portfolio Assessment Design Comments

Lack of alignment with forward procurement

WPTF does not believe the proposed monthly portfolio assessment is appropriate to implement as designed because of the lack of coordination with forward procurement. The portfolio assessment, in its current design, is inconsistent with California’s primary long-term planning process, the CPUC’s Integrated Resource Plan, and its short-term planning process – the system RA requirement. The CAISO proposes to use only shown RA resources in the production simulation tool to assess how likely the shown monthly RA fleet supports grid reliability, which inherently is different that the physical, existing RA fleet and forward capacity study methodologies. This creates a situation where the CAISO will very like require additional last-minute procurement based on a study that has different input assumptions and reliability targets than occurs in the forward procurement processes.

For example, the CAISO’s own Summer Assessment and the CPUC’s IRP proceeding both utilize a portfolio which includes deliverable RA resources and energy only resources to meet system reliability and energy needs. Thus it seems inherently the portfolio assessment will require a higher level of RA capacity. If so, the CAISO should articulate what the goal of the portfolio assessment study is compared to forward procurement. At the time of the study, RA capacity will either exist in the system or not – so it is unclear to WPTF why the CAISO is proposing a higher level of RA than forward processes. 

If the study does yield frequent backstop events, this will have a direct impact on costs. By nature, last minute procurement is riskier from a reliability standpoint (what if the capacity has already been sold external to the CAISO or doesn’t exist), but also more costly from a market standpoint. The portfolio assessment proposal brings additional cost risk to LSEs depending on the expected frequency of backstop use. 

An LSE could be found to meet their month ahead RA requirement through their RA showing, but if the monthly portfolio assessment determines some collective deficiency in reliability based on the RA committed resources, a compliant LSE could end up with additional capacity cost allocated to cure the assessed system deficiency - regardless of whether their individual shown RA resources aligned with the CAISO’s portfolio assessment needs. This scenario illustrates the efficiency loss that occurs by using a backstop mechanism rather than forward procurement to procure additional RA needs. LSEs are unable to mitigate their own costs or likelihood of the backstop because there is no way for one LSE to know what another LSE may show in their RA plan.

Ultimately, WPTF believes that in order for the portfolio analysis to be successful the CAISO must demonstrate it can run timely assessments, clearly define the trigger for its CPM backstop procurement, and provide the market insights on the results that drive forward procurement. WPTF urges the CAISO to establish any additional rules for backstop procurement with a degree of consistency with the CPUC’s RA program by aligning the metric used to assess if there is sufficient capacity procured to meet reliability.

Finally, while ideally the CAISO would demonstrate that the backstop improves reliability while balancing consumer interests, if the CAISO is determined to add an additional backstop process that is more stringent than the forward procurement process, the CAISO must also ensure that sufficient capacity exists in the market to cure its identified incremental reliability need.

Transparency and data concerns

The current design of the portfolio assessment lacks transparency, particularly around the drivers of the study results.  CAISO must provide more information on study results and especially which scenarios failed or were found deficient, both data and analysis, for this tool to be useful as a market signal. CAISO also must provide specific details for how the results of the portfolio assessment, which produces a range of probabilities of a system deficiency given size of the shortfall, would be operationalized into a trigger for Capacity Procurement Mechanism (CPM) and should not allow CAISO flexibility to apply a different methodology month over month to trigger backstop procurement. If the mechanism and its trigger for backstop procurement is poorly understood it is impossible for the market to predict the mix of resources and production profiles to maintain an acceptable level of reliability of the system. This is especially troubling to WPTF because the portfolio assessment is proposed to be done right before the month capacity is needed. Thus, if the market cannot predict the results, LSEs will be left scrambling to procure RA at last minute (high) prices and there is nothing the CAISO can do to impact the actual amount of available resources in that time horizon. Yes, they may be able to procure additional RA if excess capacity is available, but the mechanism fails to be an upfront investment signal on system need.

The lack of transparency is most immediately apparent by the fact the ‘RA Showing’ scenario resulted in a more reliable/lower probability of shortfall than the ‘Thermal’ scenario which replaces all renewable resources with natural gas resources.  This outcome is counter-intuitive and was not clearly explained on the workshop call or in the paper. It is unclear whether this odd outcome is indicative of a study flaw or study inputs. Our general understanding is that a driver of this outcome is due to the fact the ‘RA Showing’ and ‘Thermal’ scenarios are not an apples-to-apples comparison.  For example, some solar is counted at the ELCC value while other is counted at the average maximum output.  When doing studies of this nature and looking to derive themes and market needs, consistency is important.

Additionally, we appreciate the data that is provided in the Supplement Paper but found multiple errors. The Table 1 individual RA values do not add up to the Total RA Value for either scenario. We appreciate the correction in the presentation, but it would be helpful to have the numbers right from the beginning. The same goes for Figures 5 and 6 in the paper.  How can there be more observations for Hour 1 in the Thermal scenario in Figure 6 when it is a subset of Figure 5? Data errors make it challenging to dig into a proposal. The numbers end up confusing as opposed to illuminating.  WPTF would like to request additional data and analysis rigor if the CAISO continues with this proposal.

Thank you for consideration of our comments.