Draft 12/18/98 - Southwest Power Pool



Event Analysis Report – Sample

1. THE EVENT ANALYSIS REPORT (EAR) BEING PROVIDED TO YOU AS A SAMPLE REPORT WAS PREPARED BY THE SOUTHWEST POWER POOL REGIONAL ENTITY (SPP RE) FROM THE EVENT ANALYSIS PROCESS DOCUMENTS SUBMITTED BY THE INDIVIDUAL SPP REGISTERED ENTITIES INVOLVED IN THE FEBRUARY 2011 SOUTHWEST COLD WEATHER EVENT..

2. The EAR reflects an actual weather event that impacted multiple regions across the United States and involved several entities in the SPP Region. Company-specific identifiers have been redacted.

3. The EAR is meant to be a sample report and the level of detail necessary to fully explain your event will vary depending on the category and details of your specific event.

4. The EAR contains notices concerning Confidential/CEII Protected information even though the February 2011 Southwest Cold Weather Event in the SPP region did not involve CEII protected information. If your event involves CEII protected information, leave the notice on your report, otherwise remove it.

5. Your EAR should include your company specific identifiers, not those of the SPP RE. The Acknowledgements section of the EAR includes a description of the SPP RE that is provided as an example and should be replaced with your company-specific information.

TABLE OF CONTENTS

TABLE OF CONTENTS…………………………………………………………………………………I

Acknowledgements 1

Introduction 2

Quick Facts 3

Overview 5

Event Analysis Report 6

Pre-Event Condition 7

Event 7

Restoration Observations 9

Root Cause Analysis............................................................................................................,.........10

Recommendations & Corrective Actions. 11

Conclusion.............................................................................................................................,...... 12

Appendices.............................................................................................................................,..... 15

Acknowledgements

THE NORTH AMERICAN ELECTRIC RELIABILITY CORPORATION’S (NERC) MISSION IS TO ENSURE THE RELIABILITY OF THE BULK POWER SYSTEM IN NORTH AMERICA. TO ACHIEVE THAT, NERC DEVELOPS AND ENFORCES RELIABILITY STANDARDS; ASSESSES ADEQUACY ANNUALLY VIA A 10-YEAR FORECAST AND WINTER AND SUMMER FORECASTS; MONITORS THE BULK POWER SYSTEM; AUDITS OWNERS, OPERATORS, AND USERS FOR PREPAREDNESS; AND EDUCATES, TRAINS, AND CERTIFIES INDUSTRY PERSONNEL. NERC IS A SELF-REGULATORY ORGANIZATION, SUBJECT TO OVERSIGHT BY THE U.S. FEDERAL ENERGY REGULATORY COMMISSION AND GOVERNMENTAL AUTHORITIES IN CANADA. LEARN MORE AT WWW..

The Southwest Power Pool Regional Entity (SPP RE), an independent and functionally separate division of SPP, was created to fulfill the duties specified in the FERC-approved Regional Entity Delegation Agreement between SPP and NERC. As a NERC Regional Entity, SPP RE promotes and works to improve bulk power system (BPS) reliability. Specifically, SPP RE is responsible for overseeing the development of regional reliability standards; monitoring and enforcing registered entities' compliance with reliability standards; assessing and evaluating the reliability of the BPS; and providing technical expertise and assistance to the owners, operators and users of the BPS, in particular to the registered entities located within SPP RE's footprint - an eight-state area that includes all or parts of Arkansas, Kansas, Louisiana, Mississippi, Missouri, New Mexico, Oklahoma, and Texas.

Introduction

MAINTAINING THE RELIABILITY OF THE NORTH AMERICAN BULK ELECTRIC SYSTEM DEPENDS ON THE COMPLICATED AND TECHNICALLY SOPHISTICATED ACTIVITIES OF BALANCING ELECTRICITY SUPPLY AND DEMAND AND MANAGING THE FLOW OF ELECTRICITY THROUGHOUT NORTH AMERICA’S INTERCONNECTED NETWORKS.

Every system disturbance is an opportunity to learn more about how the electric system responds to abnormal or extreme conditions. By detailed analysis of disturbances, the industry can learn key issues about the behavior of the system that can be used to improve the bulk power system’s response to future disturbances.

This report addresses the extreme winter weather event that occurred January 31 – February 4, 2011 across multiple reliability regions including Region I, Region II, and SPP RE. The SPP region only had three utilities, Company A, Company B and Company C that either experienced difficulties with their generators or thought that they may experience impacts due to events in other regions during the winter weather event.

After SPP RE became aware of the situation with these three utilities, SPP RE polled the other entities in the region to see if their area had any impacts to the BES due to the weather situation. While some Balancing Authorities (BA) experienced load and customer loss, these losses were limited to the distribution system. No other SPP entities experienced any impact to the reliability of the BES.

Purpose

The purpose of this report is to present the findings of the analyses of this system disturbance, which include:

• Analyzing this bulk power system event

• Identifying the root causes that may be precursors of potentially more serious events

• Assessing the reliability performance for lessons learned

• Disseminating the findings and lessons learned to the electric industry to improve reliability performance

The purpose falls in line with the Reliability Assessments and Performance Analysis Program’s objectives that are part of the scope of the Electric Reliability Organization, and which are detailed within NERC’s Rules of Procedure.

Methodology

The process used in this analysis follows NERC’s Blackout and Disturbance Response Procedures and uses root cause analysis methods, which include:

• Reviewing the sequence of events

• Defining and analyze causal factors

• Analyzing each factor’s root causes

• Developing and evaluating corrective actions

Quick Facts

|TABLE 1 — QUICK FACTS COMPANY A |

|DATE & TIME |FEBRUARY 1,2 AND 3, 2011 |

|LOWEST FREQUENCY |59.941 HZ RANGING TO 60.061 HZ, WITH AN AVERAGE OF APPROX. 60.000 HZ |

|GENERATION LOSS |X,XXX MWS THROUGHOUT THE EVENT TIMEFRAME, BUT NOT SIMULTANEOUSLY |

|LOAD LOSS |NONE |

|TRANSMISSION INVOLVED |NONE |

|ENTITIES INVOLVED |COMPANY A |

|SYSTEM RECOVERY |2/1 AT 1103 CST TO 2/3 AT 1513 CST |

|ROOT CAUSES AND MAJOR CONTRIBUTORY |THE EXTREMELY COLD WEATHER CAUSED COMPONENT FREEZING AT SEVERAL OF COMPANY A’S GENERATING UNITS. |

|FACTORS | |

|Table 1 — Quick Facts Company B |

|Date & Time |2/2/2011 at 17:58 PM CST |

|Lowest Frequency |60.009 HZ ranging to 60.021 just prior to the disturbance |

|Generation Loss |Company B, Unit 1, xxx MWs |

|Load Loss |None |

|Transmission Involved |None |

|Entities Involved |Company B |

|System Recovery |2/2/11 at 1956 CST to 2/3/11 at 1512 CST |

|Root Causes and Major Contributory |Company B had multiple events happen on different generating units during February 2 -3, 2011. Human |

|Factors |error, failed equipment exacerbated by the extremely cold temperatures, and declining gas pressure due |

| |to high demand all contributed to Company B’s situation during the winter weather event. |

|Table 1 — Quick Facts Company C |

|Date & Time |2/2/2011 at 5 PM CST |

|Lowest Frequency |N/A |

|Generation Loss |None |

|Load Loss |None |

|Transmission Involved |None |

|Entities Involved |Region I utilities and gas suppliers and processors, Company C |

|System Recovery |Company C did not lose any generation or load as a result of the gas supply situation during the event. |

|Root Causes and Major Contributory |Region I experienced high loads and loss of some generation that resulted in the implementation of |

|Factors |rolling blackouts. Region I notified gas suppliers and processors that they may experience blackouts |

| |and they in turn notified Company C that they may not be able to meet scheduled gas deliveries. Company|

| |C began planning for potential re-dispatch and for the use of alternative fuel supplies in the event of |

| |a disruption. |

Overview

COMPANY A

Company A is an Investor Owned vertically integrated electric utility. Company A headquarters are located in City, State.

Company A serves more than xxx,xxx retail customers in States X and Y. Company A has a generation capacity of approximately x,xxx MW.

Company A owns and maintains transmission lines of the following voltages:   xxx kV; xx miles, yyy kV; yyy miles, zzz kV; zzz miles.   The Company A network has several interconnection points— (list)

Company B

Company B is an electric cooperative that provides power to customers in State Z. That power is delivered across a high-voltage transmission network consisting of xx substations and more than xxxx miles of xxx kV, xxx-kV, and xxx-kV lines. Company B provides distribution services to more than xxx,xxx metering points. Generation Facilities include several natural gas fired units and a coal unit.

Company C

Company C serves approximately xxx,xxx customers in a xx,xxx square mile service area in several states. Company C is a summer peaking utility and its all time summer peak BA demand was x,xxx MWs. Company C owns approximately x,xxx megawatts of regulated generating capacity and has an additional x,xxx megawatts of IPP capacity in its Balancing Authority Area. 

Company C’s system includes about xxx miles of xxx kV xxx miles of xxx kV, and xxxx miles of xxx kV transmission lines.  Company C has BES AC transmission interconnection points with neighboring utilities: (list) Company C has a total of xxx interconnection points with neighboring Balancing Authorities and Load Serving Entities.

Event Analysis Report

WHAT WAS DONE TO PRODUCE THIS REPORT? HOW DATA WAS GATHERED ETC?

SPP RE staff initially became aware of the event through the companies’ OE-417 reports. SPP RE staff issued a data hold on 2/11/11 to Companies A, B and C. A subsequent data hold request was sent to all SPP entities on 2/18/11. SPP RE staff also requested that the three utilities that did experience impacts to the BES during the event complete an Events Analysis Process Docment, Appendix A on 2/11/11, followed by a request on 2/25/11 for an Appendix D and Appendix G. SPP RE staff reviewed the company information, requested follow-up information, and prepared this report.

The information that formed the basis of the report came directly from the utilities via written communication or through follow-up verbal discussions. Furthermore, SPP RE polled the other SPP member utilities to determine weather-related impacts across the footprint. No other SPP entities experienced any BES disturbances, although some BAs did experience load and/or customer loss on the distribution system as shown in the table below:

Company Load Loss Customers

D 4 MW N/A

E 12 MW 0

F 2 MW 500

G 12 MW 4383

H 45 MW 10,420

I 31.5 MW 8,222

J 104.2 MW 36,424

K 3 MW 92

Pre-Event Conditions

SYSTEM CONDITIONS BEFORE EVENT. ACTIONS THAT LED TO EVENT.

Prior to the deterioration of the weather on the evening of January 31, 2011 system conditions were normal throughout the region and for all three of the utilities that are the subject of this report. A major winter storm moved across the region the night of January 31 and continued through February 1, 2011.[1] The initial precipitation was mostly freezing rain or sleet, but as the cold air continued to spill south, a quick changeover to snow occurred. The snow continued, heavy at times, into the afternoon of Feb. 1 before tapering off. Total snowfall amounts of 10-15 inches were common across northeast Oklahoma.

The “strong storm system strengthened rapidly in Texas on Tuesday February 1. Heavy snow of 12-18 inches (some amounts over 20 inches) and winds of 30-40 mph made for blizzard conditions from Oklahoma to Milwaukee. The snow and ice extended northeast all the way to parts of New England, although amounts were not as great as in the Midwest.”[2]

The weather system moving across Region 1 and SPP created different situations for the three SPP utilities during the event timeframe, as will be discussed in further detail in the next section. Company A experienced component freezing on many of their generators. Company B’s events involved generation outages that resulted from numerous causes including human error, equipment failures exacerbated by the extremely cold temperatures, and declining gas pressure to two units due to high natural gas demand. Company C, on the other hand, took precautions to avoid generator outages due to the situation the weather caused in Region 1 and for a Region 1 utility. Since Company C received advanced notice of potential gas supply problems, Company C was able to avoid firm load shed by issuing public appeals for conservation and burning oil in place of natural gas at some units to get through a limited fuel situation that resulted from rolling blackouts in Region 1.

Event

DETAILED DESCRIPTION OF EVENT

Company A

The cold weather conditions caused generator control component freezing at several of Company A’s power plants. Throughout the entire event, Company had a total of s,xxx MWs of generation offline, but not simultaneously. Equipment problems included a broken fuel supply component ; frozen components such as intermediate drum level transmitters and tubing at the Company A Plant 1; and freezing of the feedwater flow transmitter sensing lines at Company A Plant 2. Company A moved into EEA-2 status on Tuesday February 1, 2011 at 11:30 CST. Once the units tripped, the situation was exacerbated by the lack of heat previously generated by each unit. This made recovering each unit’s capacity more difficult. The power plant crews worked to thaw components and return units to service and Company A was removed from EEA-2 status on Thursday, Feb 3 at 15:13 CST. At no time during the event was Company A in an EEA-3 status.

Company B

Company B’s experience during the winter weather began with a Control Room Operator entering the wrong set point for Company B, Unit , the largest generating unit in the Balancing Authority (BA). With the trip of B1 at 1758, Company B attempted to bring on two combustion turbines, B4 and B6, but they failed to start which created a capacity deficit. When B5 was brought on line, gas limitations in the area did not allow for both B4 and B5 to remain online for very long before gas pressure began to decay. B1 startup and generating capability was further exacerbated by a faulty speed changer on an induced draft fan which ultimately caused a second trip. Further compounding the situation was the transmission limitations for importing generation into the BA. These circumstances resulted in an EEA-3 situation for Company B that lasted from 1931 2/2/11 to 1512 on 2/3/11.

Company C

As extremely cold temperatures and increased loads began occurring across State X, both Region 1 and the Region 1 Utility began to implement rolling black-outs. As a result, some natural gas compressor stations and gas processing plants located in those areas (that supply natural gas to some generating stations in the Company C area) were affected by these regional blackouts. These blackouts, and issues resulting in restarting the gas compressor and processing facilities after the blackouts, combined with regional supply freeze offs, caused the interstate and intrastate natural gas pipeline transportation suppliers to notify Company C that they might not be able to meet scheduled firm deliveries of natural gas, which in turn could have affected the amount of generation available to serve load in the Company C area.

Due to this notice, Company C made plans for potential re-dispatch to maximize available generation and for potential fuel switching, and began implementing steps of its Emergency Operations Plan. These steps included:

• Loading all available generating capacity,

• Determining status of adjacent BA’s for potential assistance,

• Informing the Reliability Coordinator (RC) and adjacent Balancing Authorities (BA’s) of system status,

• Reducing load through public appeals and curtailment of interruptible loads,

• Declaration of Energy Emergency through its RC.

Through these steps, as well as the use of alternative fuels, Company C was able to carry all firm load and the required level of operating reserves. At no time during the event did Company C issue an EEA.

Company C filed an initial form OE-417 with the DOE, NERC and SPP RE on February 2, 2011, detailing the generation fuel issue described above. Company C filed an updated form OE-417 on February 3, 2011, as the fuel situation was continuing and high loads on the Company C system caused Company C to issue public appeals for the conservation of energy. Company C filed a final form OE-417 on February 4, 2011.

Restoration Observations

HOW WAS RESTORATION PERFORMED?

Company A

Company A restoration work was performed on the individual generating units starting on February 1, 2011 and continued through February 9th. See table in Appendices section of the report for detailed restoration information.

Company B

With the trip of B1 at 1758 on February 2, Company B requested xxx MWs from SPP’s Reserve Sharing Group and notified its neighboring BAs and TOPs that B1 had tripped. Within 10 minutes of the B1 trip, Company B attempted to start the gas turbines, but B6 (xx MWs) did not start due to a heater tripping in the start-up torque converter. B4 was then called to start, but it too failed to start due to gas valve controller issues. B4 was then called on to start. At this point, (1915), Company B had problems securing their schedule due to the unavailability of transmission paths into the BA. A neighboring utility was called to start wholesale customer-owned generation available for emergencies. At 1931, SPP RC issued an EEA-3 due to the lack of transmission into the BA and the inability to start xxx MWs of gas turbines and a xxx MW deficit. B1 was brought back online at 1956, but was limited to xx MWs during ramp. B4 was brought on line at 2124. B6 continued to have start up problems after heating up the hydraulic fluid due to problems with a gas valve that was not associated with the weather.

February 3 began with Company B continuing in an EEA-3 due to the B1 limitation, B6 failing to start and transmission unavailability restricting the ability to bring in firm schedules. At 30 minutes after midnight, Company B was asked by a gas transportation company about gas usage at City, State and at 0104, B5 was taken off line to relieve the gas pressure situation. B1 remained de-rated due to a problem with the motor on an ID fan speed changer. As the load began ramping up that morning, Company B sent an email to its members asking for a public appeal to conserve energy and all Company B cities were notified to generate per requirements to bring on all generation. At about 9 am, the B6 repairs were complete and the unit was brought online. The B1 restoration was further hampered by another human error when an operator sent a control signal to the in-service fan instead of the fan being repaired, resulting in the good fan tripping. With the loss of all ID fans, the unit tripped at 1019. Due to recovering gas pressure at City, State, B5 was brought online at about 10:30 a.m. At 1335, B1 was brought back online and began ramping to full load. After testing the ID fan repair at half load, B1 was released for full load operation at 1504 and the EEA3 was terminated at 1512.

Company C

Company C did not lose any generators, transmission facilities, load or customers during the event, so no restoration work was necessary.

Root Cause Analysis

WHAT WAS THE CAUSE OF THE EVENT?

While two of the SPP utilities experienced generation loss during the winter weather event, the causes were varied. Company A had problems with several of their generators, primarily due to freezing components, as shown in the table above. Company B’s generation issues began with a human error that caused the B1 ougate, the Balancing Authority’s largest generating unit, to trip. This outage then created a series of generator events that were exacerbated by the extreme cold weather conditions including equipment failures and decreasing gas supply pressure. These events and causes will be discussed in more detail below.

Company A

Company A does make winter weather preparations on both a seasonal and event-driven basis. Company A performs winterization of plant components and inspections freeze protection equipment each year. As severe weather conditions are forecast, Company A once again inspects vital freeze protection systems, schedules additional staff and checks and restocks supplies such as fuel for heaters and deicing materials. Despite these preparations, the extreme temperature and wind conditions that began in Company A’s area the night of January 31, 2011 caused various components to freeze at many of Company A’s generating units.

Company B

The root cause of both of the B1 trips was human error. The first trip was caused by a Control Room Operator incorrectly entering a set point for the excess boiler O2. With the boiler O2 control loop running in manual at an output of 73%, the operator entered a set point of 2.6% in the controller. The boiler O2 level at the time was about 2.6%. The operator inadvertently entered the actual O2 percentage instead of the controller output percentage. The very low set point caused all ten secondary air dampers to drive in the closed direction. Secondary air duct pressure went high as a result, and the forced draft fans could not respond quickly enough to prevent a unit trip due to the high secondary air duct pressure.

During B1 restoration, a faulty motor on the charging spring assembly associated with the speed changer on one of the induced draft fans limited B1’s capability to less than half load because the fan could not be switched to high speed. During attempts to replace the faulty motor, a second trip occurred on B1 due to human error. One of the control technicians assisting with the repair initiated a control signal to the “good” fan, causing this fan to trip and immediately tripping the unit on loss of both ID fans. This situation was exacerbated due to the training activities being conducted by the Controls Coordinator.

B6 failed to start because the hydraulic oil in the startup package torque converter was too cold to allow the unit to spool enough to satisfy control system speed permissive. Subsequent investigation revealed that a heater breaker had tripped causing the fluid to quickly cool below acceptable temperatures in the very cold weather. B6 unit remained unavailable after heating up the hydraulic fuel fluid due to the gas valve positioner’s failure to respond to control signals. This CT is located remotely at a site that is not manned around the clock which contributed to the restoration time.

The B4 combustion turbine failed to start due to a problem with the control system that controls the gas valves during a unit startup. Once B4 and B5 were both online, very high demand on the natural gas supply system resulted in declining gas pressure on the supply line into the City Station. As a result, B5 was taken offline to stabilize the gas pressure.

Transmission paths into the BA were also limited during the event due to the unavailability outside the BA which limited Company B’s options for replacement power. Company B called on wholesale customer-owned generation, but failed to call on all the cities in the BA for emergency generation in a timely manner.

In summary, the root causes of Company B’s events during the winter weather period were varied. Human error caused the initial event on Company B’s system and subsequent human error, either directly or indirectly, contributed to Company B remaining in an EEA3 condition for several days. The extreme weather conditions also caused problems for Company B with components freezing and the demand for natural gas causing Company B to shutdown a generator. Further compounding the situation for Company B was the lack of transmission capability into the BA which limited Company B’s options for replacement power and the failure to call on all emergency generation.

Company C

Company C did not experience any generation loss. Due to the notification that Company C may have gas supply issues, Company C was able to implement contingency plans that were effective in maintaining the operation of their generators.

Recommendations & Corrective Actions

WHAT WILL BE DONE TO PREVENT SIMILAR OCCURRENCE

Company A

Company A is in the process of reviewing the details of the event and creating an incident Command System to prepare for and execute during weather events of this type. Many of Company A’s corrective actions are specific to individual generators, but some overall preventative actions include:

1. Ensuring that freeze protection for vital systems is working prior to weather events so that the protected equipment doesn’t fail;

2. Schedule additional staff on site ahead of the winter weather events;

3. Keep critical paths clear of snow and ice throughout the winter weather event;

4. Ensuring that critical supplies such as fuel for heaters, deicing materials, and personal protective gear are available on site before the winter event

Company B

Company B experienced generator outages during the winter weather due to multiple factors including human error and equipment failures. The corrective actions related to the human error occurrences during the winter weather event include possible disciplinary actions, additional training for control room personnel, bringing an additional NERC-certified System Operator on duty during capacity and energy emergencies, and suspending training activities during events. Equipment that failed to operate during the event such as a gas valve and positioner and a control system will be replaced. Company B also intends to update their emergency operating plans and reinforce the importance of completing the plan with the System Operators. Company B provided an extensive list of corrective actions and lessons learned -see Appendices section of this report.

Company C

Company C believes that the interdependency between electric emergency operating procedures should be considered in the emergency plans of Region 1 and other BAs. It may be necessary for BAs to identify gas suppliers for all BES generators as critical loads in their emergency operation plans, even if the suppliers serve generators outside the BA area. Generation plants should also review weatherization procedures to assure adequacy and may need to take additional steps when extremely cold temperatures are expected.

Conclusion

THESE UTILITIES IN THE SPP REGION OPERATE IN WINTER WEATHER CONDITIONS EVERY YEAR AND AS A RESULT MAKE PREPARATIONS FOR SUCH CONDITIONS AS DESCRIBED BELOW:

Company A

Company A addresses weatherization issues at its power plants typically in two ways, seasonally and event driven. Seasonally, Company A has standing orders to complete tasks such as winterizing the deareator & drum, pump houses, cooling towers, and intake; inspecting freeze protection equipment such as heat tracers and freeze protection panels; and performing dew point checks.

As severe winter weather approaches, plants inspect the freeze protection for vital systems to ensure it is working prior to the weather event so that the protected equipment does not fail; schedules additional staff on site ahead of the weather event; maintains critical ingress/egress paths clear of snow and ice throughout the weather event; inventories and restocks critical supplies such as fuel for heaters, deicing materials, and personal protective gear to ensure all supplies are available on site before the weather event.

Company B

Company B utilizes different winterization procedures at its various generating facilities to protect against problems caused by cold weather.

• B1 is the primary baseload unit in Company B’s generation fleet. Its boiler is enclosed, and the unit is well-equipped with heat tracing and insulation on systems that are exposed to ambient temperatures. Compressed air systems are also equipped with dryers to keep moisture out of air lines and reduce the risk of freezing. There are defined procedures for cycling cells in out and out of service to help reduce icing at the cooling tower. There is also cold weather operating procedures in place for the coal handling system. Cold weather preparation begins in the fall with verification of heat trace and equipment heater functionality and preventative maintenance on HVAC equipment. Company B also contacts its coal supplier to ensure adequate coal fuel supplies are available in case of heavy snow or frozen coal.

• B7 typically runs year round in a load following mode. The unit is well-equipped with heat tracing and insulation on systems that are exposed to ambient temperatures. There are defined procedures for cycling cells in out and out of service to help reduce icing at the cooling tower. Compressed air systems are also equipped with dryers to keep moisture out of air lines and reduce the risk of freezing. Similar to B1, cold weather preparation begins in the fall with verification of heat trace and equipment heater functionality and preventative maintenance on HVAC equipment.

• B2, B8, and B9 are intermediate steam units that typically only run during the summer months and during shoulder months as required to cover for outages on other units. When not in service these units are drained to prevent freezing of exposed systems. Each of these seasonal units has layup procedures to protect equipment during the winter months when these units are typically offline. These layup procedures include placement of insulation blankets in duct work and air heaters, as well as keeping heat sources in service. When these units are shut down following their summer run, the boilers are flash drained to rid boiler circuits of water and further reduce the risk of freezing and corrossion. These units also have cold weather procedures for cooling tower operation and other outside systems, but as mentioned they are not typically called upon to operate during the winter months.

• B3, B4, B5, B6 and B10 are all combustion turbines. These units are equipped with lube oil heaters and electric or gas compartment heaters.

o B6: Winter preparation includes draining and blowing down evaporative coolers and isolating the water source to these coolers, installing wind break panels to block cold winds from blowing under the turbine compartments, and verifying oil (lube and hydraulic starting package) heaters and enclosure heaters are energized and functional. Periodic rounds include reading temperature gauges and checking electrical heater breakers to verify they are properly energized.

o B3, B4 and B5: Winter preparation includes draining evaporative cooler sumps and isolating the water source and verifying lube oil and enclosure heaters are energized and functional. Periodic rounds include reading temperature gauges and checking electrical heater breakers to verify they are properly energized.

o B10: Winter preparation includes placing gas and electric heaters in service and confirming they operate correctly and confirming lube oil heaters are energized and functional. Periodic rounds include reading temperature gauges and checking heating equipment to verify it is functioning correctly.

Company C

In the October/November time frame of each year, the Company C’s work management system will automatically generate cold weather/winterization preventive maintenance work orders for Company C generating plants. Plant planners will also review the work order backlog to identify any other cold weather related work orders that need to be completed. Operations personnel will have been trained on their respective Standard Operating Procedures, which include the cold weather operating procedures. This informal process has not changed in the previous five winters.

Beginning with the first cold weather alert issued January 31, 2011 by Company C’s meteorological forecasting function and continuing through the period of the cold weather event ending on February 6, 2011, actions taken by Company C’s generating plants to respond to the cold weather event included, but were not limited to:

• Increasing Operator staffing levels

• Increased frequency of Operator rounds for equipment susceptible to issues in cold weather;

• Faster response to cold weather related maintenance work orders;

• Review of cold weather operating procedures;

• Completion of “yellow” and “orange” alert checklists;

• Addressing a Chemistry Lab sulfuric acid freeze warning;

• Starting auxiliary boilers at certain plants to supplement plant steam space heaters as needed;

• Closing all exterior building doors, dampers and louvers;

• Rechecking and adding heat trace circuits;

• Plugging exterior wall openings;

• Adding wind breaks;

• Adding additional insulation;

• Utilizing heat lamps and portable space heaters.

Even though the utilities in the SPP region do prepare for winter weather, the experiences across multiple regions during this event highlight the need to prepare even better for extreme winter weather conditions. Company A and Company B both experienced problems with their generators due to the extreme temperatures and have plans to improve their winter weather procedures. Due to advance notice and preparation by Company C, they did not experience any impacts to their generating units or system reliability.

This extreme winter weather event caused significantly more problems in Region 1 and Region 2 than it did in SPP’s area. Nevertheless, one area in particular where there may be a need for improvement is highlighted in Company C’s Recommendations. It may be beneficial for RTOs to identify natural gas supply or processing facilities plans as critical loads in their emergency operation to avoid taking them off-line during rolling blackouts as this may cause cascading events back to the electric system.

APPENDICES

DRAWINGS, DATA, MAPS, ETC WARNINGS AND ADVISORIES

Winter Weather Warnings

The map below shows the range of warnings and advisories by the National Weather Services as of 9:46 a.m. February 1, 2011.

[pic]



Company A Generation Outages and Deratings; Restoration Efforts

|Company A Generation Outages and Deratings Week of January 31, 2011 |

|Generator |Outage Start Date |Outage Stop Date |

Company B - Corrective Actions and Lessons Learned

|Primary Interest Groups1 |Problem Statement |Details |Corrective Actions |Lessons Learned |

|Power Plant Operators |B1 1trip #1 caused by operator |The Control Room Operator incorrectly entered a set point for |Meetings were held with all operating |Situational awareness is very important. During |

| |error ‐incorrect set point |excess boiler O2. With the boiler O2 control loop running in manual|crews to discuss the situation and |periods when the System Status is critical, extra|

| |entered in plant control system |at an output of 73%, the operator entered a set point of 2.6%. The |reinforce the importance of paying |care should be taken during operations and |

| | |boiler O2 level at the time was about 2.6% , and the operator |attention to operator actions being |consideration should be given to the value of an |

| | |inadvertently used the engineering units for the set point as |performed. A high fidelity simulator is |operator action compared to the risk of that |

| | |opposed to the output percentage. |also being installed at the facility to |action. |

| | | |provide enhanced training opportunities.| |

| | | |Disciplinary action under review. | |

|Power Plant Operators |B1 trip #2 caused by operator |While repairing a faulty motor on a speed changer on one of two |The problem was discussed with the |At the time of the event, the Controls |

| |error ‐control signal sent to |induced draft fans, a Controls Coordinator was assisting with the |Controls Coordinators to stress the |Coordinator was training two other employees, |

| |incorrect fan |repairs by sending control signals from the plant Distributed |importance of staying focused on the job|which caused him to be distracted from his |

| | |Control System to the problematic fan as directed by maintenance |being performed. |primary duties of assisting with the ID fan |

| | |staff at the fan. The Coordinator accidentally called up the | |repair. In the future, distractions such as this |

| | |control screen for the good ID fan and sent a control signal to | |will be eliminated when assisting with critical |

| | |that fan that caused it to stop and ultimately trip the unit due to| |plant repairs or performing other operations |

| | |loss of both ID fans. | |which could impact the availability of the unit. |

|Power Plant Operators |B3 combustion turbine fail to |While attempting to start this CT, the unit failed to start due to |A control card was pulled and reseated, | |

| |start ‐gas valve controller |a problem with the control system that controls the gas valve |but the operator still had to drive the | |

| |problems |during a unit startup. |unit manually from 95% to 100% speed | |

| | | |before placing the unit online. A | |

| | | |complete controls replacement has been | |

| | | |budgeted for this unit due to an | |

| | | |increasing number of similar problems. | |

|Power Plant Operators |B4 combustion turbine fail to |While attempting to start this CT, the unit failed to start because|The status of the breaker in question is|This particular CT is located remotely at a site |

| |start ‐heater breaker on |the hydraulic oil in the startup package torque converter was too |being evaluated for inclusion in the |that is not manned around the clock. The unit |

| |hydraulic oil reservoir tripped |cold to allow the unit to spool up quickly enough to satisfy |plant control system so that its status |control system is linked to other generation |

| | |control system speed permissive. Subsequent investigation revealed |can be monitored and alarms can be |sites that are manned around the clock for alarm |

| | |that a heater breaker had tripped causing the fluid to quickly cool|established if the breaker is tripped. |monitoring purposes. It is important to make sure|

| | |below acceptable temperatures in the very cold weather. | |the alarms sent to the control system are |

| | | | |adequate. Consideration should also be given to |

| | | | |dispatching personnel to the site any time B1 |

| | | | |goes down as well as calling in additional staff |

| | | | |for around the clock, onsite monitoring when |

| | | | |system status is critical. |

|Power Plant Operators |B4 combustion turbine fail to |The unit failed to start when the gas valve positioner stopped |The gas valve and positioner were | |

| |start ‐gas valve control problems|responding to control signals. |disassembled and cleaned. This helped | |

| | | |the problem, but a new gas valve and | |

| | | |position have been ordered to improve | |

| | | |reliability. | |

|Balancing Authority and |All municipal units not called in|Per Company B's Capacity and Energy Step Plan, which is based on |Company B has reinforced with the System|At the time the System Operators reached the |

|Transmission Operator |timely manner to generate |the requirements outlined in Attachment 1‐EOP‐002‐2.1, all |Operations staff the importance of |point on the Capacity and Energy Emergency Step |

| | |available generation should be brought online when energy and |completing all actions identified in the|Plan where they were to call the Company B |

| | |capacity are deficient and an EEA is anticipated. Company B has |Capacity and Energy Emergency Step Plan |municipal generating stations, B1 had just been |

| | |contractual rights to the capacity and associated energy from |in a timely manner. Company B will also |returned to service and it appeared that the EEA3|

| | |several municipalities. Company B did not call several of these |pursue emergency contact information for|was about ready to be lifted. In addition, at the|

| | |cities until the morning after an EEA3 was declared the previous |those facilities that aren't staffed 24 |time it was unlikely that most of the Company B |

| | |evening. |hours a day those facilities that aren't|municipal generating stations would have been |

| | | |staffed 24 hours a day. |staffed as these stations are not staffed 24 |

| | | | |hours a day and it was past 7:00 p.m. As it |

| | | | |turned out, B1 had additional issues and the EEA |

| | | | |remained in effect until the following afternoon.|

| | | | |Moreover, at least one of the municipal |

| | | | |generation stations was staffed until the |

| | | | |municipal generation stations was staffed until |

| | | | |midnight and might have responded. The lesson |

| | | | |learned is that the step process should continue |

| | | | |to be followed until all items have been |

| | | | |completed or until the EEA is over. We are also |

| | | | |considering the need to have the cities with |

| | | | |generation keep employees at their units around |

| | | | |the clock during times when B1 is offline or the |

| | | | |system status is critical. |

| | | | | |

Company B - Corrective Actions and Lessons Learned, Continued

|Primary Interest Groups1 |Problem Statement |Details |Corrective Actions |Lessons Learned |

|Balancing Authority and |Message to municipal |When contacting municipal generators, System Operators initially |When contacting municipal generators System |There might have been a few MWs left on |

|Transmission Operator |generators to generate all |asked that these facilities generate to cover their city load. |Operators will ask them to generate at maximum |the table by not asking all municipal |

| |they can during an EEA |However, some of these generators have reverse power relay |capacity. However, as Company B reviewed SPP's |generators to generate beyond their own |

| |should be clear |settings, or have the ability to pull reverse power relays, so |information, it concluded that SPP may not be |city load when capable of doing so. |

| | |that they would have been able to generate more than their own |metering this excess generation correctly. | |

| | |city load. |Company B will advise SPP of this possible issue | |

| | |[pic] |and pursue any needed corrections. | |

|Transmission Operator |Protocols for communication|The System Operators on duty at the time of the B1 trip correctly|The wording in the BES Study Procedure will be |It is important that operating |

| |of BES study results needs |followed procedures (and met compliance requirements) associated |clarified so that it is clear what is to be |procedures are clearly written so that |

| |to be clarified in the |with conducting a study following a unit trip. The Transmission |communicated when. |they follow the requirements of the |

| |Company B BES Study |Operations Planner was called out, he completed the study, no | |applicable NERC Reliability Standard |

| |Procedure |changes to System Operating Limits were noted, and the | |without leaving room for alternate |

| | |Transmission Operations Planner communicated this to the System | |interpretations. |

| | |Operators. During the compliance review, an ambiguity was | | |

| | |discovered in the procedure that could confuse the Operations | | |

| | |Planner as to his obligation top p g communicate study results. | | |

|Balancing Authority and |Staffing levels during |During the system emergency that started when the BA's largest |It will be standard operating practice going |Company B examined whether the |

|Transmission Operator |system emergencies should |unit tripped, the System Operators on duty were faced with many |forward to call in an additional NERC‐certified |procedures were, in fact, workable or |

| |be increased |responsibilities. Because of the numerous operational and |System Operator during capacity and energy |needed to be improved. Although the |

| | |communication requirements that were faced, they did not make it |emergencies. |System Operators on duty at the time of |

| | |all the way through the Capacity and Energy Emergency Step | |the event reported that they believed |

| | |Procedure in advance of the EEA3 being declared by the RC. | |they could handle the situation without |

| | | | |additional help, in fact, they failed to|

| | | | |follow all of the Step procedures. While|

| | | | |automation might be a future option, the|

| | | | |review team concluded that processes |

| | | | |should be improved in the near term by |

| | | | |bringing on an extra operator when there|

| | | | |is an unplanned H1 outage. That operator|

| | | | |will be tasked with working through the |

| | | | |Capacity and Energy Emergency Step Plan |

| | | | |while the other System Operators handle |

| | | | |any other operational issues associated |

| | | | |with the event. |

| | | | | |

|Transmission Operator |Need to discuss the need |Almost every time B1 trips, the Company B system is immediately |A note has been added in the Capacity and Energy |The declaration of an EEA1 shortly after|

| |for an EEA1 with the RC |put in a capacity deficient situation. Because of SPP |Emergency Step Procedure to ask the RC to declare|a B1 trip, (or any other time the loss |

| |early in the process |transmission constraints, firm transmission paths are rarely, if |an EEA1 early in the process. Additionally, the |of a generator creates a capacity |

| | |ever, available for replacement power. Thus, once B1 trips, an |Resource Tool utilized by System Operations to |deficient situation), would provide a |

| | |EEA2 or EEA3 is likely to be needed unless B1can be returned to |ensure adequate energy resources are available to|little more advance notice to other TOPs|

| | |service quickly. This is because there is not sufficient |serve load on an hourly basis will be modified to|and BAs and the RC. |

| | |quick‐start generation inside the Company B BA to replace the |include a capacity section. This is to ensure the| |

| | |loss of B1. Given the lack of adequate transmission and the |System Operators and the RC are aware when firm | |

| | |limited quick‐start generation in the BA, Company B should be |capacity is not available to serve load plus | |

| | |asking the RC to declare an EEA1 as soon as B1 trips. |reserve requirements. If a capacity deficiency is| |

| | | |noted at any time, the need for an EEA will be | |

| | | |evaluated. | |

|Balancing Authority and |Minimum System Status |System Status levels are defined by the System Operators based on|Minimum System Status levels will be defined for |A formalized process tying System Status|

|Transmission Operator |levels should be tied to |various factors. There is currently no requirement to increase |each EEA level. |levels to EEA levels will ensure that |

| |the various EEA levels |System Status levels during EEA's to defined minimum levels. | |the System Status is accurately set |

| | |System Operators typically do upgrade System Status in these | |during emergencies. |

| | |situations, but there is no process in place to ensure that this | | |

| | |happens. | | |

Company B - Corrective Actions and Lessons Learned, Continued

|Primary Interest Groups1 |Problem Statement |Details |Corrective Actions |Lessons Learned |

|Balancing Authority and Transmission |There is no plan for |Attachment 1-EOP-002-2.1 includes implementation of |Company B transmission engineering will |Company B System Operators do not know how to |

|Operator |implementing voltage |voltage reduction programs as a step to take prior to |complete a study to determine the |implement a voltage reduction program or if such |

| |reduction programs during an|declaring an EEA if such programs are available. |effectiveness of a voltage reduction |a program would be effective. |

| |EEA |Company B has not studied how effective a voltage |program on the Company B system. The | |

| | |reduction program would be for its system. |study will also identify how to implement| |

| | | |a voltage reduction program if it is | |

| | | |determined such a program would be | |

| | | |effective. | |

|Balancing Authority and Transmission |There is no plan for |Attachment 1-EOP-002-2.1 includes implementation of |Each facility will develop an energy |Company B does not have a formal plan to conserve|

|Operator |reducing energy consumption |utility conservation programs as a step to take prior |conservation plan to be implemented when |energy at its own facilities during an energy |

| |at Company B's facilities |to declaring an EEA if such programs are available. |directed to do so by System Operations |emergency. |

| |during an EEA |Company B has not developed formal plans for conserving|during an energy emergency. | |

| | |energy at its facilities during these events. | | |

|Balancing Authority and Transmission |Public appeals should |Attachment 1-EOP-001-0 provides elements for |A process for including educational |While most consumers know steps they can take to |

|Operator |include information on how |consideration in developing emergency plans. When |messages in public appeals to conserve |conserve energy, educational messages included in|

| |the public can conserve |issuing public appeals to conserve energy, this |energy will be developed. In the future, |public appeals may offer additional ideas that |

| |energy and should be timed |attachment suggests that these appeals should include |the timing of public appeals will be |increase the amount of load reduction achieved as|

| |to reduce load during the |educational messages on how to accomplish load and |coordinated more closely with load |a result of the amount of load reduction achieved|

| |morning ramp if possible. |energy reductions. There was not guidance in Company |profiles. |as a result of the public appeal. The public |

| | |B's appeal. Moreover, the appeal was issued when the | |appeal that was issued occurred at the end of the|

| | |morning ramp was almost over. | |morning ramp. An earlier appeal might have |

| | | | |resulted in a more effective reduction in load. |

|Balancing Authority and Transmission |When bringing all available |Although wind generators are not dispatchable |Wording in the Capacity and Energy Plan |The wording in the emergency procedures |

|Operator |generation online during an |resources, they are potentially "available generation" |will be added to clarify that black start|associated with bringing black start units online|

| |energy emergency, black |that can provide energy during emergencies if the wind |units should not be brought online during|during a capacity and energy emergency needs to |

| |start units should be |is blowing. The owners of these facilities should be |a capacity and energy emergency during a |be clarified and energy emergency needs to be |

| |exempted start units should |informed of EEAs to ensure that if the wind is blowing |capacity and energy emergency. |clarified. |

| |be exempted |they are operating rather than going off line for if | | |

| | |the wind is blowing they are operating rather than | | |

| | |going off‐line for maintenance or electing not to | | |

| | |generate for economic reasons. | | |

|Balancing Authority and Transmission |Wind generators should be |Although wind generators are not dispatchable |Wording will be added to the Capacity and|Although wind generators aren't dispatchable, |

|Operator |contacted during an EEA |resources, they do provide energy during emergencies if|Energy Emergency Step Plan associated |they can't provide energy if they're in an outage|

| | |the wind is blowing. The owners of these facilities |with contacting the operators of wind |or if in the future there are times they choose |

| | |should be informed of EEAs so that they can postpone |generating facilities that Company B has |not to operate for economic reasons. If planned |

| | |any planned maintenance to maximize generator |PPA's with to ensure that unit |maintenance at these facilities can be deferred |

| | |availability. |availability is maximized during an EEA. |during an EEA it might allow for additional |

| | | |Currently, approximately xxx MW of |energy to be provided. As wind resources increase|

| | | |additional wind generation is |and market rules change there may be other |

| | | |interconnected to the Company B BA but |reasons to advise wind generators of an EE |

| | | |pseudo tied to and interconnected to the | |

| | | |Company B BA but pseudo | |

| | | |[pic]tied to and controlled by other BAs | |

| | | |and sold under PPAs to other utilities. | |

| | | |It seems unlikely under current market | |

| | | |rules that any of these wind resources | |

| | | |would be off‐line if the wind is blowing,| |

| | | |but rules and economics might change in | |

| | | |the future. In particular, wind energy | |

| | | |being dispatched to loads to the east | |

| | | |could relieve east to west congestion | |

| | | |limiting Company B imports. Company B | |

| | | |will look at whether additional | |

| | | |procedures applicable to pseudo-tied wind| |

| | | |generation might be useful now or in the | |

| | | |future, particularly given that their | |

| | | |output may relieve east to west | |

| | | |congestion limiting Company B imports. | |

| | | | | |

|Balancing Authority and Transmission |Identify who should be |Company B does not have a formal list of government |Company B will work with the Member |When discussing the rolling blackouts that |

|Operator |contacted if rolling |agencies and others who should be contacted in the |Owners to review their list of contacts |occurred in other regions, Company B realized |

| |blackouts need to be |event rolling blackouts are required. Contacts such as |that will be utilized during rolling |that it could take additional measures to ensure |

| |implemented |these are typically the responsibility of the Member |blackouts and determine if improvements |it would be more prepared if rolling blackouts |

| | |Owners. |are needed. |were necessary in its service territory. |

Company B - Corrective Actions and Lessons Learned, Continued

|Primary Interest Groups1 |Problem Statement |Details |Corrective Actions |Lessons Learned |

|Balancing Authority and Transmission |Operating reserve requirements |The need for filing Other Extreme Conditions reports |Wording will be added to the Capacity and|During an EEA we should file OEC reports to relieve |

|Operator |during EEAs need to be clarified |during EEAs to relieve operating reserve requirements |Energy Emergency Step Procedure to |our operating reserve requirements. The Ancillary |

| |EEAs need to be clarified |needs to be clarified in the Capacity and Energy |identify the need to file hourly Other |Service plan should be updated accordingly, as well |

| | |Emergency Plan. The requirements associated with |Extreme Conditions reports to relieve the|as the plan should be updated accordingly, as well as|

| | |adjusting reserve allocations on Company B Ancillary |requirement to carry operating reserves |the operating reserve calculation on SCADA. |

| | |Service plans also needs to be clarified as well as |during an EEA. Wording will also be added| |

| | |required on Company B Ancillary Service plans also |to define the requirements for adjusting | |

| | |needs to be clarified |reserve added to define the requirements | |

| | |[pic]as well as required adjustments to the Operating |for adjusting reserve allocations on the | |

| | |Reserve calculation on SCADA also need to be clarified.|Ancillary Service plan. Training will be | |

| | | |conducted with System Operators | |

| | | |associated with required adjustments to | |

| | | |the Operating Reserve calculation on | |

| | | |SCADA. | |

|Balancing Authority |The process for keeping the RC |There are several ways Company B submits load and |SPP has been contacted and asked to |Company B submits load and capability information |

| |informed of current load and |capability information to the RC, but the preferred |provide guidance on the preferred method |through resource plans, daily load and capability |

| |capability information needs to be |process for keeping the RC informed of this information|of communicating up-to-date load and |reports, real‐time ICCP data, and OPS1 outage |

| |clarified |during an EEA needs to be clarified. |capability information. |reporting. However, the preferred method for making |

| | | | |sure the RC is are of current load and capability |

| | | | |data needs to aware of current load and capability |

| | | | |data needs to be clarified with the RC. |

|Balancing Authority and Transmission |Initiation of the Company B Load |All System Operators are trained on the Capacity and |Wording will be added to the Capacity and|Although Company B did not have to implement its |

|Operator |Shedding Plan needs to be included |Energy Emergency Plan which requires that the System |Energy Emergency Step Procedure to |manual load shedding plan for this event, formalizing|

| |as the last step in the Capacity |Operator initiate Company B's manual load shedding plan|clearly identify the need to implement |this action in the Capacity and Energy Emergency Step|

| |and Energy Emergency Step |without delay in the event all required actions have |the manual load shedding plan after |Procedure will ensure that all applicable NERC |

| |Procedure. |been exhausted or if they cannot be completed in |exhaustion of all required steps or if |Reliability Standards are followed in the event of a |

| | |sufficient time to resolve the energy condition. |all steps cannot be completed in |capacity and energy emergency. |

| | |However, the Step procedure, which was created as a |sufficient time to resolve the energy | |

| | |checklist to ensure all required actions are taken |emergency. | |

| | |during a capacity and energy emergency, does not | | |

| | |include initiation of the manual load shedding plan as | | |

| | |the last step. | | |

-----------------------

[1]

[2]

-----------------------

August 2011

Event Analysis and Information Exchange Program

This document is Protected Critical Infrastructure Information (PCII) as designated by the Department of Homeland Security (DHS) under the Critical Infrastructure Information Act of 2002 (CII Act) and 6 Code of Federal Regulations (CFR) Part 29

zÉ[pic]?É[pic]ùÉ[pic]¼Ê[pic]FË[pic]Î[pic]Î[pic]öêêêê9±kdàp[3]$[pic]$[pic]If[pic][4]–lÖÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ

PROTECTED CRITICAL INFRASTRUCTURE INFORMATION — DO NOT RELEASE

February 2011 Southwest Cold Weather Event in the SPP Region

................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download