Home NRC Library Document Collections Events, Reports Associated with Part 21 Reports 1997

Part 21 Report - 1997-600

ACCESSION #: 9508080294 LICENSEE EVENT REPORT (LER) FACILITY NAME: TURKEY POINT UNITS 3 AND 4 PAGE: 1 OF 23 DOCKET NUMBER: 05000250 TITLE: DESIGN DEFECT IN SAFEGUARDS BUS SEQUENCER TEST LOGIC PLACES BOTH UNITS OUTSIDE THE DESIGN BASIS EVENT DATE: 11/03/94 LER #: 94-005-02 REPORT DATE: 07/17/95 OTHER FACILITIES INVOLVED: TURKEY POINT UNIT 4 DOCKET NO: 05000251 OPERATING MODE: 1/5 POWER LEVEL: 100/0 THIS REPORT IS SUBMITTED PURSUANT TO THE REQUIREMENTS OF 10 CFR SECTION: 10 CFR 50.73(a)(2)(ii), (a)(2)(v), (a)(2)(vii), 10 CFR 21 LICENSEE CONTACT FOR THIS LER: NAME: C. L. Mowrey, Licensing OEF TELEPHONE: (305) 246-6204 Engineer/Analyst COMPONENT FAILURE DESCRIPTION: CAUSE: B SYSTEM: JE COMPONENT: 34 MANUFACTURER: A160 REPORTABLE NPRDS: Y SUPPLEMENTAL REPORT EXPECTED: NO ABSTRACT: On November 3, 1994, Turkey Point Unit 3 was in Mode 1 at 100% power, and Unit 4 was in Mode 5 during a refueling outage. During the Unit 4 Integrated Safeguards Test, the 3A sequencer failed to respond to the Unit 4 Safety Injection signal. A defect was found in the sequencer software logic which, for a limited period of time, could inhibit any or all of the four sequences from responding to specific valid signals. The defect only affects the sequences during manual or automatic testing. The sequences were installed in late 1991. Monthly manual testing of the sequencer has been resumed. Front panel visual examinations are being performed every 8 hours, and internal visual examinations are being performed every 24 hours. A permanent repair to the software logic is being evaluated. Independent consultants performed an assessment of the existing sequencer design, software design, and the Validation and Verification process. Two other conditions were discovered, one involving Containment Spray (CS) pump autostart, and one involving the 480 Volt Load Center feeder breaker autoclosure. Both were determined to have minimal safety significance. The CS system and the 480 Volt Load Centers remain operable. END OF ABSTRACT TEXT PAGE 2 OF 23 Supplement 2 of this LER reports on (1) the original test logic defect discovered last November, and reported in the original LER; (2) the Containment Spray Autostart issue discovered in January as part of the design review described in corrective action #9, and reported in Supplement #1; and (3) an additional condition involving 480 Volt Load Center feeder breaker autoclosure, discovered in June while preparing the software modifications to fix the earlier two issues. Where applicable, each major section of the LER contains subsections appropriate to each of the three issues listed. I. DESCRIPTION OF THE EVENT I.a ORIGINAL TEST LOGIC DEFECT On November 3, 1994, Turkey Point Unit 3 was operating in Mode 1 at 100% power, and Unit 4 was in Mode 5 during a refueling outage. During the Unit 4 Integrated Safeguards Test, a failure of the 3A sequencer [JE:34] to respond to the opposite unit's Safety Injection (SI) signal occurred. Troubleshooting resulted in the discovery of a defect in the sequencer software logic which, under certain conditions, could inhibit the sequencer from responding to a valid emergency signal. The defect manifested itself in the failure of the 3A High Head Safety Injection (HHSI) pump [BQ:p] to start. Turkey Point has four HHSI pumps; one per train, per unit. Each HHSI pump is capable of providing 50 percent of system requirements, therefore two of the four are required to mitigate the consequences of accidents analyzed in the Updated Final Safety Analysis Report (UFSAR). In order to meet single failure criteria, each sequencer signals its associated HHSI pump to start, and the opposite unit's sequences signal their associated HHSI pumps to start. For example, an SI signal on Unit 3, Train A, signals the 3A sequencer and both of the Unit 4 sequences. With no equipment failures, all four HHSI pumps will respond to an SI signal on either unit. The software logic defect is limited to the test function, but the defect is common to all four sequences (one sequencer per train, per unit). The design intent of the sequences is such that should a "real" emergency signal occur while the sequencer is being tested, the test signal clears, allowing actuation of the Engineered Safety Features controlled by the sequencer. Because the sequences would not have responded properly to an SI signal as designed, Turkey Point Units 3 and 4 have been operating outside their design basis. This condition was reported to the NRCOC at 1609 on November 3, 1994, in accordance with 10CFR50.72(b)(ii)(B). I.b CONTAINMENT SPRAY PUMP AUTOSTART ISSUE The detailed review of the sequencer software, described in Corrective Action #6, resulted in the discovery of one other error in the software, which is independent of the test mode. A potential condition was identified which, for a remote set of circumstances, would preclude the automatic start of the Containment Spray (CS) pumps [BE:p]. The condition identified occurs when the Hi-Hi Containment Pressure (HHCP) signal is received by the sequencer during an approximate 60 millisecond (ms) time window just prior to the end of sequencer load block 3 for Loss of Coolant Accident (LOCA) or Loss of Offsite Power coincident with LOCA (LOOP/LOCA) events. The sequencer is designed to autostart the CS pumps 11 to 13 seconds after an SI TEXT PAGE 3 OF 23 signal (without LOOP) if the HHCP signal is present or at or after 44 seconds under conditions where the HHCP signal occurs more than 13 seconds after receipt of the SI signal. For a LOOP/LOCA, these times are shifted by the bus stripping and Emergency Diesel Generator (EDG) [EK:dg] start delay of approximately 16 seconds. Thus the 60 ms window occurs 12.886 to 12.945 seconds after receipt of an LOCA signal, or 28.886 to 28.945 seconds after receipt of a LOOP/LOCA signal. Although Turkey Point is licensed to accommodate a LOCA with or without a concurrent LOOP, the sequencer was designed to accommodate non-concurrent LOOP/LOCA sequences as well. As a result, for certain non-concurrent events, a Main Steam Line Break or a Small Break LOCA (but large enough to cause a HHCP signal) can also create conditions under which this error may manifest itself. Automatic CS pump start actually involves two HHCP signals; one via the sequencer logic as described above, and one directly from Engineered Safety Features Actuation System (ESFAS) relay [JE:44]. Because of the minimum pulse required to assure CS pump breaker [BE:bkr] closure, and a potential relay race with a CS pump start permissive from ESFAS, the CS pump breaker may not receive a close signal of sufficient duration to assure breaker closure. The identified condition is unique to the start of the CS pump because the CS pump start signal duration decreases as the postulated receipt of a HHCP signal approaches the end of load block 3. All other sequenced equipment receives a start pulse of fixed duration, either 2 or 5 seconds. This condition was determined to be not significant, in part because the manual start capability of the CS pump is not affected (and is adequately proceduralized), and in part because the probability of occurrence of the condition is lower than the probability of a common-mode failure of both trains of containment spray. The significance of the condition is discussed further in Section III. I.c 480 VOLT LOAD CENTER FEEDER BREAKER AUTOCLOSURE CONDITION During verification and validation of modifications associated with the corrective actions for the original defect, an additional condition was discovered wherein the 4.16 KV breakers [EA:52] which feed the safety- related 480 Volt Load Centers [ED] may fail to automatically close during certain unique events. If an SI signal is received between 15.5 and 16 seconds after a LOOP, the sequencer will provide a breaker close signal to the 4.16 KV breakers which feed the 480 Volt Load Centers at the same time that a breaker strip signal is present. This will result in the Load Centers not being automatically reloaded onto the bus. The failure of the Load Centers to automatically load can occur on all four sequences if a simultaneous undervoltage signal occurs for both units (complete LOOP) and signals from both trains of SI are present. When a LOOP event takes place, the sequencer initiates bus stripping approximately 1 second after the undervoltage input. Approximately 8 seconds later the EDG output breaker closes. The first LOOP load block is sequenced on at 16.5 seconds after the loss of offsite power. The design intended that if an SI signal is received prior to 16 seconds after the LOOP, the LOOP/LOCA loads are automatically sequenced on without re-stripping the bus. If an SI signal is received after the EDG output breaker is closed, then the LOOP loads are re-stripped from the bus, the sequencer timers would reset and after a time delay, the LOOP/LOCA loads would be sequenced on. The sequencer pulse to strip the bus is 1 second in duration. TEXT PAGE 4 OF 23 If the Load Center breakers receive a close pulse when a strip pulse is present, the breaker control logic prevents the breaker from responding to the close signal. This breaker control logic, commonly known as an "anti-pumping" circuit, is designed as a protective feature for the breaker. The breaker will not respond unless the strip and close signals are both removed and the close pulse is reapplied. Since, under the identified condition, the sequencer can provide a strip signal concurrent with a close signal, followed by removal of the close signal (which is not reapplied), the load center breakers would not automatically close. This condition was determined to be one which alone could have prevented the fulfillment of the safety function of a system needed to mitigate the consequences of an accident. It was reported to the NRCOC on June 19, 1995, in accordance with 10 CFR 50.72(b)(2)(iii). I.d SEQUENCER DESIGN BASIS AND FUNCTIONAL REQUIREMENTS Each of the four sequences, 3C23A-1, 3C23B-1, 4C23A-1, and 4C23B-1, is associated with a given train (3A, 3B, 4A, and 4B, respectively). They are designated Class 1E, Seismic Category I, since their operation is required for safe shutdown of the reactor in the event of a LOOP and to mitigate the consequences of a design basis accident. The sequences are Programmable Logic Controller (PLC)-based cabinets using a PLC for bus stripping and load logic and control. The signal path structure of the PLC uses dedicated input modules, control logic, and dedicated output modules. LOOP Signal Only On a LOOP in a given unit, both sequences associated with that unit will respond accordingly to clear their associated buses, stripping all 4.16 KV loads and specified 480 Volt loads within one second after the LOOP signal is generated. The Emergency Diesel Generators (EDGs) [EK:dg] will start, and within 15 seconds the EDG output breakers [EK:bkr] close, then loads required for safe reactor shutdown are sequentially connected to the corresponding bus; the first load block output signal is generated 16.5 seconds after the onset of the LOOP. The first load block output signal closes the 480 Volt Load Center feeder breakers. LOCA Signal Only If either unit experiences a LOCA, and preferred (offsite) power is available, bus stripping signals and EDG breaker closure permissive signals will not be initiated by the sequences. Vital loads will be sequentially connected to the buses by the sequences (including the opposite unit's HHSI pumps). If an EDG is already operating and parallelled to offsite power, and either unit experiences a LOCA, the EDG breaker will trip. The EDG will continue to run in a standby condition. On the LOCA unit, Engineered Safety Features (ESF) equipment will be sequentially loaded onto the bus by the sequencer. Following a LOCA, if any given train experiences undervoltage, bus stripping, EDG breaker closure, and sequential loading will be directed. TEXT PAGE 5 OF 23 LOOP/LOCA After a LOOP on both units, if one unit experiences a LOCA, the buses associated with the LOCA unit will be stripped and ESF loads will be loaded onto the bus. On the non-LOCA unit, both buses are stripped again, and reloaded with essential equipment; both HHSI pumps will also start. Sequencer Testing Each sequencer is provided with Manual test and Automatic Self-test capability. The test mode is determined by a three-position Test Selector switch. The three positions are AUTO (self-tests 15 steps or scenarios in the automatic test sequence), MAN (each test is manually initiated), and OFF (no test signals are generated). In the automatic test mode, the sequencer continuously tests the input cards, output cards, and output relay coils, and exercises the program logic. The sequencer is designed to abort the manual and automatic test modes in response to a valid input. The automatic self-test function is normally in operation, however it is not required to be in service for the sequencer to perform its safety function. The manual test, in addition to testing all the conditions covered by the automatic test, actuates the output relays. However, blocking relays energize before the output relays energize, and the output relays de-energize before the blocking relays de-energize. Placing the Test Selector switch in MAN stops automatic self-testing. Manual testing involves five stripping/clearing scenarios (bus clearing, 480 Volt undervoltage with SI present, 480 Volt degraded voltage, 4.16 KV undervoltage, and safety injection [LOCA] on an isolated bus). Upon completion of the stripping tests, sequencing scenarios are tested manually by rotation of a Sequencing Mode Test Selector switch through eleven steps or loading scenarios (LOOP; LOOP/LOCA same train; LOOP/LOCA other unit; LOCA same train; LOCA other unit; LOOP/LOCA same train with concurrent HHCP; LOOP/LOCA same train with HHCP before 13 seconds; LOOP/LOCA same train with HHCP after 13 seconds; LOCA same train with concurrent HHCP; LOCA same train with HHCP before 13 seconds; LOCA same train with HHCP after 13 seconds). Automatic self-testing cycles through 15 of the 16 test steps in the same order (the bus clearing scenario is not tested in AUTO). The test steps start roughly an hour apart, and there is one hour in the automatic test sequence in which no testing takes place, so a full cycle of automatic self-testing takes approximately sixteen hours. Then the cycle begins again. Should a valid process input signal be received during manual or automatic testing, the testing stops, the test signal clears, and the inhibit signal is supposed to clear if present, allowing the valid signal to sequentially energize the output relays and their associated ESF equipment. TEXT PAGE 6 OF 23 II. CAUSE OF THE EVENT II.a TEST LOGIC DEFECT The 3A sequencer failed to respond as expected to an opposite unit SI signal. The 3A sequencer had dropped out of the Automatic Self-Test without alarming, indicating that it had received a valid input signal. During troubleshooting, the input LED for a 4A SI signal was found to be lit, indicating the signal was still present. The 3A sequencer response should have been to start the 3A HHSI pump after a 3 second delay. However, the pump failed to start because it did not receive a start signal from the sequencer. Following the failure of 3A HHSI pump to start in response to a 4A SI input signal as described above, an analysis of the sequencer software logic was performed to determine the root cause of the failure. A software design defect was discovered whereby the start signal for the 3A HHSI pump remained inhibited during sequencer automatic test step 3 (LOOP/LOCA Other Unit) even though a valid process input was present. In parallel with the above analysis, this particular fault was duplicated on the sequencer simulator which is identical to the 3C23A-1 (3A) sequencer. This is in contrast to the original design bases of the sequencer Automatic Self-Test and Manual Test functions. The review was then expanded to include additional test modes, process inputs, and required outputs. It was found that the problem exists during both manual and automatic testing, during sequencer test steps 2, 3, 6, 8, and 10. These steps correspond to the following scenarios: Step 2 LOOP/LOCA Step 3 LOOP/LOCA Other Unit Step 6 LOOP/LOCA with concurrent High High Containment Pressure Step 8 LOOP/LOCA with High High Containment Pressure less than 13 seconds later Step 10 LOOP/LOCA with High High Containment Pressure more than 13 seconds later Note that these are tested scenarios, not potential plant events. Note too that all five of the affected test step scenarios involve LOOP and LOCA. If a valid SI signal is received 15 seconds or later into one of the above tests, the test signal clears as intended, but the inhibit signal is maintained by means of latching logic. This latching logic is originally established by the test signal, but may be maintained by the process input signal if it arrives prior to removal of the test signal. Since the above condition is applicable to both the automatic self-test and manual testing, the sequencer must be considered inoperable during both testing modes. Note, however, that this defect will not cause a sequencer operating malfunction with the Test Selector switch in any position for any design basis scenario which involves a loss of offsite power. TEXT PAGE 7 OF 23 This software logic defect was introduced during the detailed logic design phase of the software development. The detailed logic designer and the independent verifier failed to recognize the interaction between some process logic inhibits and the test logic. The defect in the software logic was not detected during the Validation and Verification process (V&V) because the response to valid inputs was not tested during all stripping and loading sequences of the automatic and manual testing logic. FPL has evaluated the V&V for the sequences and concluded that the existing V&V adequately addresses operation of the sequences with the Test Selector switch in OFF. This logic defect can occur when the sequencer is in either the manual or automatic test mode, and the test sequence currently being executed is loading sequence test 2, 3, 6, 8, or 10. This was determined based on a review of the sequencer logic drawings for the 15 steps in the automatic test sequence, and design basis event signals. The sequencer simulator was used to confirm the results of the review. The defect cannot affect sequencer operation with the Test Selector switch OFF. In loading sequence tests 2, 6, 8, or 10, the sequencer may be inhibited from responding to a valid SI signal on the same train. In loading sequence test 3, the sequencer may be inhibited from responding to a valid SI signal on the opposite unit. II.b 480 VOLT LOAD CENTER CONDITION For the 480 Volt Load Center feeder breaker autoclosure failure, the design implementation failed to account for the duration of time between closing of the EDG breaker, which is the permissive for re-stripping, and the beginning of LOOP load sequencing at 16 seconds. During this window of time, if an SI signal is received an unnecessary strip pulse occurs. For the most part, the unnecessary pulse is inconsequential since the bus is already strapped. However, if the strip pulse occurs between 1 second and 0.5 seconds prior to the first LOOP load block (between 15.5 and 16 seconds after offsite power loss), the strip pulse overlaps the breaker close pulse. After 16 seconds, receipt of an SI signal resets the load sequence timers and the overlap does not occur. A test was conducted on the sequencer in the Training Building to confirm that both stripping and sequencing signals to the Load Center feeder breakers could be simultaneously generated by a LOOP followed by a LOCA during an approximate half second window of Load Block 1. The test was performed using the EDG 3A sequencer logic in the training sequencer. The PLC software was modified to simulate the timing for a valid SI same train signal upon receipt of a valid LOOP (4.16 KV undervoltage input) over the 15.5 to 16 second window. The testing confirmed that the overlapping strip/load signals could occur as described. III. ANALYSIS OF THE EVENT III.a TEST LOGIC DEFECT As a result of the erroneous inhibit signals, the potential exists for any sequencer output to be prevented from being generated when required. Exactly which output or outputs is(are) determined by a combination of factors, i.e., which test scenario is in progress, how long since the test scenario was initiated, and which process input or inputs are received. In TEXT PAGE 8 OF 23 general, for the approximate one-hour duration of each of the above test steps (with the Test Selector switch in AUTO), the sequencer will not respond correctly to a valid process input signal. With the sequencer Test Selector switch in AUTO, the sequencer steps sequentially through sixteen steps as described above; first the five bus stripping/clearing steps, followed by the eleven LOOP and/or LOCA scenarios. Note that the five test steps affected by the software defect are all in the loading sequence test steps, so the first affected step is the seventh step in the total testing sequence. During each of these affected test steps, fifteen seconds after the initiation of the step, the sequencer would not have responded properly to a valid process input signal. So the sequencer was inoperable for about five hours out of each sixteen hour period as long as its Test Selector switch was in AUTO. The sequencer was also inoperable for the duration of any Manual test of the five test steps listed above. A complete manual test on one sequencer takes about one hour. The review of the sequencer logic determined that improper operation of the sequencer could occur for only certain sequencer stripping/loading scenarios in which an SI signal without LOOP occurs. The sequencer logic software defect does not affect any scenarios where a LOOP also occurs, whether before, after, or concurrent with an SI signal. A failure modes and effects matrix identified the following four potential plant events where the logic software defect could affect the operation of the sequencer, depending upon which of the five affected test steps (discussed above in CAUSE OF THE EVENT II.a) are being performed when the SI signal is received by the sequencer: #1 LOCA Same Train #2 LOCA on other Unit #3 LOCA w/High High Containment Pressure (HHCP) < 13 seconds #4 LOCA w/HHCP > 13 seconds Note that these are potential plant events, not test step scenarios. Note too that in contrast to the list of affected test step scenarios presented earlier, none of the potential plant events affected involve a LOOP. For each of these events, the sequencer could receive a valid SI signal but the logic defect could inhibit the sequencer from starting equipment. Events #1, #3, and #4 above each have four logic test steps out of a total of sixteen which would inhibit the sequencer from providing a start signal to the equipment it controls while event #2 is affected by only one of the sixteen logic test steps. The probability that an individual sequencer would not respond to a valid same train SI signal is 4 hours/16 hours = 2.5E-1. The probability that an individual sequencer would not respond to a valid opposite unit SI signal is 1 hour/16 hours = 6.25E-2. The equipment affected due to the failure of a sequencer was identified from plant drawings. The equipment listed below is specific to the 3A sequencer. The equipment lists would be similar for the other three sequences. TEXT PAGE 9 OF 23 For event #1, the following equipment would not be automatically loaded by the sequencer: Residual Heat Removal Pump 3A [BP:p] HHSI Pump 3A Intake Cooling Water Pumps 3A (1) and 3C (1) [BI:p] Emergency Containment Cooler Fan 3B and 3C [BK:fan] Component Cooling Water Pumps 3A (1) and 3C (1) [CC:p] Emergency Containment Filter Fans 3B and 3C {BK:fan] Note (1): The equipment identified may already be in operation and may not require manual action to start. For events #3 and #4 (LOCA w/HHCP < 13 sec; LOCA w/HHCP > 13 sec), Containment Spray Pump 3A would be affected in addition to the equipment identified above for event #1. For event #2 (LOCA Other Unit), only the 3A HHSI Pump would not be automatically started. It should be noted that one of the initiating signals for Auxiliary Feedwater (AFW) system [BA:p] is bus stripping, which is controlled by the sequencer. No credit is taken, however, for bus stripping in the accident analyses for initiating AFW. AFW is also initiated on low-low steam generator level, SI, manual initiation and trip of all Main Feedwater pumps [SJ:p]. Using the above information, the defect in the sequencer test logic represents a potential concern for events where SI is required for mitigation and no LOOP is experienced. III.b CONTAINMENT SPRAY AUTOSTART ISSUE Using the Turkey Point baseline Probabilistic Safety Assessment (PSA) model, the probability of dual train failure of the CS system if called on to operate has been estimated to be approximately 2.6E-3. This estimate reflects CS system and support system component failure probabilities not including either of the software errors reported here. The failure to automatically start a CS pump due to this software error can only occur under a very remote set of circumstances. The 60 ms window is on the same order as the tolerance on relay pick-up times and the sequencer processing and timing tolerances. Even with sophisticated timing equipment, it is unlikely that the failure mode could be demonstrated repeatedly. The probability of receipt of a HHCP signal during a 60 ms window of vulnerability compared to the range of timing conditions for which the sequencer is designed is considerably smaller than the overall system reliability identified above. If it is assumed that HHCP can occur at any time within approximately two minutes after the SI signal (the earliest time at which SI is postulated to be reset), then the probability of the evaluated condition occurring on one train is: 0.060 sec/(2 min x 60 sec/min) = 5.0E-4 The estimate of the probability of a CS pump not starting automatically in a LOCA or LOOP/LOCA due to the reported software error is therefore approximately a factor of five below the estimated probability of both CS trains failing during a design basis event. TEXT PAGE 10 OF 23 The probability of the software error affecting both trains is considerably lower, since it would require: 1) the initiating SI signals to be at the sequencer inputs within 60 ms of each other; 2) the two trains of HHCP both occurring within the 60 ms window of vulnerability; 3) the sequencer input processing times to be identical; and 4) the timing of the two sequences in synchronization. The difference in the cumulative delay time for relay actuations on the two trains of ESFAS and differences in sequencer processing, in all likelihood would be sufficient to preclude the condition on both trains. This conclusion is supported by a review of previous Integrated Safeguards Test data. The difference between the train A and B CS pump recorded start times during a simulated LOOP/LOCA has been between 90 and 500 ms. Since some timing differences between the trains can be expected, and timing differences greater than 60 ms have been recorded during previous safeguards tests, the probability that the specific error could affect both trains of Containment Spray is therefore considerably less than the single train probability. III.c 480 VOLT LOAD CENTER CONDITION The limiting components affected by loss of the load centers are the Unit 4 EDGs auxiliary equipment, including cooling fans. The loss of the Unit 4 EDG auxiliaries means that the EDG will start to heat up in a short period of time. At full EDG loading, the EDG would exceed its temperature ratings after about 8 minutes. However, because the load centers were not energized, the EDG would only be partly loaded. Considering lower pump flows, Unit 4 EDG loading is estimated below: LOAD KW SI Pump 220 KW RHR Pump 140 KW CS Pump 110 KW ICW Pump 265 KW CCW Pump 380 KW ______________________ Total 1115 KW Under these lower loading conditions, overheating of the Unit 4 EDGs is not expected to occur for approximately 14 minutes. Other loads lost as a result of the loss of the load centers include motor operated valves which must open to allow high and low head safety injection flow. These valves are significant primarily to analyzed accidents, specifically Small Break LOCAs, as discussed below. Effect on Analyzed Accidents III.d TEST LOGIC DEFECT A review of the Turkey Point UFSAR Chapter 14 Accident Analyses was performed to determine which accidents would be potentially affected by the sequencer test software logic defect. This review identified 7 of the 22 accidents which may be affected. Two of the seven, "Loss of External Load" and "Loss of A.C. Power" were determined to be dependent on the sequencer but not affected, since the inhibited sequencer failure mode applies to LOCA scenarios only, i.e., no LOOP. TEXT PAGE 11 OF 23 The five accidents which both require SI, and are affected by the sequencer test software logic defect, are the following: 1. Large Break Loss-of-Coolant Accident (LBLOCA) 2. Small Break LOCA (SBLOCA) 3. Rupture of a Steam Pipe (Main Steam Line Break, or MSLB) 4. Steam Generator Tube Rupture (SGTR) 5. Rupture of a Control Rod Mechanism Housing The effects of the sequencer test logic defect will be discussed below for each of the five accidents. In each case, the transient is described and equipment necessary for mitigation of accidents is identified. Each transient is then evaluated assuming all four sequences fail to operate properly. Credit is assumed for operator action to start HHSI pumps as well as other ESF equipment within 10 minutes as described below. LARGE BREAK LOSS OF COOLANT ACCIDENT A LOCA would result from a rupture of the Reactor Coolant System (RCS) or any line connected to that system up to the first closed valve. For a postulated LBLOCA, a reactor trip is initiated by pressurizer low pressure (1790 psig) while the SI signal is actuated by pressurizer low pressure at 1636 psig. The consequences of the LBLOCA are limited in two ways: 1. Reactor trip and borated water injection supplement void formation in causing rapid reduction of nuclear power to a residual level corresponding to fission product decay. 2. Injection of borated water ensures sufficient flooding of the core to prevent excessive temperatures and provide long term cooling. The reactor is designed to withstand the thermal effects caused by a LBLOCA including the double ended severance of the largest RCS pipe. The reactor core and internals, together with the Emergency Core Cooling System (ECCS), are designed so that the reactor can be safely shutdown and the essential heat transfer geometry of the core will be preserved following an accident. The LBLOCA analysis presented in Section 14.3 of the UFSAR assumes that 2 of 4 HHSI pumps and 1 of 2 RHR pumps are automatically actuated during the accident. If all four sequences were inoperable because of the simultaneous presence of the test logic defect, SI actuation would not occur automatically. The LBLOCA is a design basis event whose probability of occurrence is extremely small. A LBLOCA is considered to be a break with a total cross-sectional area equal or greater than 1.0 ft**2. LBLOCA sensitivity studies, performed in 1988 to assess the impact of delaying SI, indicate that the maximum permissible SI delay is about 1 minute in order not to exceed the Peak Clad Temperature criteria of 10 CFR 50.46, and about 5 minutes to avoid exceeding fuel melt temperature, for a generic Westinghouse four-loop PWR. As a result of the test logic defect, Turkey Point tested operator reaction times to manually start SI in the absence of an automatic start (described below under MITIGATION OF SEQUENCER FAILURE MODES). The maximum time did not exceed 4 minutes. This information was provided to Westinghouse, who then determined that if SI is delayed 3 minutes and 15 seconds, the peak clad temperature for the hot rod will not exceed 1922 degrees Fahrenheit. If a conservative adiabatic heat TEXT PAGE 12 OF 23 up rate of six degrees per second is assumed for the fuel, SI may be delayed until four minutes into the LOCA without exceeding 10 CFR 50.46 PCT criteria. Therefore,if reasonable operator action is credited, no core damage would be expected. Containment Response to a LBLOCA A LBLOCA results in a significant mass and energy release into containment that results in pressurization of the containment structure. The UFSAR indicates that the pressurization event is limited by the size of containment, by containment heat sinks, and by the operation of containment cooling equipment (containment sprays and emergency containment coolers). The containment analysis for the LBLOCA was assessed using better estimate techniques in 1989 by Westinghouse. This analysis showed that peak containment pressure for a Double Ended Pump Suction (DEPS) to be on the order of 42 to 45 psig. Using the mass and energy release values developed for the design basis reconstitution work, Westinghouse re-performed the Turkey Point containment analysis assuming no operation of the containment spray pumps or the emergency containment coolers, for ten minutes. This reanalysis shows that the peak pressure of the DEPS LOCA to be approximately 44.3 psig. Accordingly, since this peak pressure is less than the design pressure of 55 psig and less than the originally analyzed peak pressure of 49.9 psig, the results are acceptable. The ultimate strength of the Turkey Point containments is estimated to be approximately 140 psig based on the Individual Plant Examination (IPE) analysis work. Dose Consequences for a LBLOCA The UFSAR contains an offsite dose evaluation that assumes a total core release (100% noble gas, 50% halogens) occurring at time t = 0 with results that remain within 10 CFR Part 100 guidelines. The event under review, however, is different than that evaluated in the UFSAR in that engineered safety features are assumed to be delayed. Using knowledge learned from observation of accident phenomena and advanced light water reactor development programs, it has been concluded that an instantaneous core melt and release of fission products to containment is not credible. Rather, significant release to the containment would not be expected to occur during the first 10 minutes of an accident. During this time, credit is taken for operator action to start SI, containment sprays, etc. Manual actuation of the containment sprays and emergency filters would provide for fission product cleanup within containment. While a calculation has not been performed, it is expected that the offsite dose consequences for this event will not exceed those stated in the UFSAR. Operation of sprays and filters will provide radioactive material cleanup prior to any significant fission product release from the containment. SMALL BREAK LOSS OF COOLANT ACCIDENT (SBLOCA) SBLOCAs are slow transients which take longer to initiate SI and therefore are less sensitive to delays in the actuation of the HHSI pumps. Containment response and dose consequences for the SBLOCA event, for the original software defect involving Autotest, are bounded by LBLOCA discussions above. TEXT PAGE 13 OF 23 The 480 Volt Load Center condition involves the SBLOCA analyses, since a specific size of small break would be required to generate the specific event timing which leads to the condition (SI signal 15.5-16 seconds after a LOOP). The effect of that condition on the SBLOCA analyses is discussed later. MAIN STEAM LINE BREAK The UFSAR analyzes two separate steam line break events; opening a relief or safety valve, and main steam piping failure. The piping failure bounds the opening of the relief or safety valve. Since the sequencer issue is only a concern for the offsite power available case, only a main steam piping failure with offsite power available will be addressed. The most limiting cooldown event occurs at zero power with no decay heat. As indicated in the UFSAR, credit is taken for a single HHSI pump to provide borated water to return the core to a subcritical state. Westinghouse re-performed the limiting MSLB accident with offsite power available assuming SI was not available for 10 minutes. The results of this analysis indicate that the event can be accommodated without SI for 10 minutes with acceptable results. A Main Steam Line Break inside containment also results in a containment pressurization transient. This event was rerun by Westinghouse assuming no active containment pressure mitigating features (i.e. no containment sprays or containment coolers). Assuming no safeguards actuation, peak containment pressure for the MSLB was 48.8 psig occurring approximately 300 seconds (5 minutes) into the transient. This is within the containment design pressure of 55 psig and is therefore acceptable. STEAM GENERATOR TUBE RUPTURE The event examined in the UFSAR is a complete tube break adjacent to the tube sheet. Each steam generator tube has a nominal diameter of 0.875 inches with a wall thickness of 0.050 inches. Accordingly, the cross- sectional break area of a double ended tube rupture is less than 1.0 square inches. This very small break area shows that this event is bounded by the SBLOCA in terms of assessing the potential for core damage resulting from this event, and that dose releases for this event will not increase as a result of delayed SI. RCCA EJECTION - RUPTURE OF A CONTROL ROD MECHANISM HOUSING The event examined in the UFSAR is a failure of a control rod mechanism pressure housing such that RCS pressure would eject the control rod and drive shaft to a fully withdrawn position. The consequence of this mechanical failure is a rapid positive reactivity insertion together with an adverse core power distribution. The reactivity transient is terminated by the Doppler reactivity effects of the increased fuel temperature, and by subsequent reactor trip before conditions are reached that can result in fuel melt. Actions are included in the Emergency Operating Procedures (EOPs) to address a SBLOCA that could be caused by a failed control rod mechanism pressure housing. Accident consequences of a SBLOCA in the reactor vessel upper head are bounded by the design-basis SBLOCA. TEXT PAGE 14 OF 23 Summary of Potential Accident Consequences Of the five UFSAR accidents affected, four are bounded by the LBLOCA. Consequences of a LBLOCA are acceptable if operator action to start ESF equipment takes place within four minutes of the start of the accident. Consequences of the SBLOCA, SGTR, and RCCA ejection are acceptable even if no operator action is taken for 10 minutes. The consequences of a MSLB are acceptable without operator action for 10 minutes, since containment pressure peaks, below the design pressure, 5 minutes into the accident. III.e 480 VOLT LOAD CENTER CONDITION The UFSAR analyzed a spectrum of SBLOCAs, as provided below: BREAK SIZE 1.5-inch 2.0-inch 3.0-inch Break Initiation, sec. 0.0 0.0 0.0 Reactor Trip Signal, sec. 67.3 35.2 15.0 Safety Injection Signal, sec. 107.5 56.2 25.8 Top of Core Uncovered, sec. approximately 3500 1562 700 Accumulator Injection Begins N/A N/A approximately 1200 Peak Clad Temp. Occurs, sec. 5034 2692 1305 Top of Core Recovered, sec. >5050 >4000 approximately 3000 A LOOP is assumed to occur concurrent with a reactor trip. Assuming an instantaneous LOOP at reactor trip and extrapolating from the above table, a break size of about 2.5 inch equivalent diameter would result in the timing sequence of concern. The credibility of the occurrence of the UFSAR analyzed scenario creating the conditions necessary to cause the loss of the 480 Volt Load Center feeder breaker automatic function has been evaluated. The two major mechanistic possibilities for such a loss would be either; (1) a failure of the transfer of both buses from the auxiliary to start-up transformers (fast bus transfer); or (2) the loss of the switchyard or transmission system due to the loss of a Turkey Point unit. For a LOOP caused by the failure of the fast bus transfer, it is improbable that it would occur simultaneously with the reactor trip because of the delay time designed into the reactor trip/turbine trip/generator lockout logic sequences. A reactor trip caused by low pressurizer pressure initiates a turbine trip. Provided there are no other events and/or failures which would cause a direct generator lockout signal, there is a designed 30 second time delay between turbine trip and generator lockout. At the end of the 30 second time delay, a generator lockout signal will be generated. The generator lockout signal will trip the generator field breaker, the generator mid and TEXT PAGE 15 OF 23 east switchyard breakers, the auxiliary transformer breakers and will close the startup transformer breakers (fast bus transfer). Failure of the fast bus transfer would cause a LOOP, pickup of the undervoltage relays, strip the 4.16 KV buses, start the EDGs and sequence the emergency loads on the EDGs. Therefore, on the failure of both the A bus and B bus fast transfer, the LOOP would be expected to be initiated 30+ seconds after a reactor trip without an SI. If at any time during this 30 second time delay an SI signal is generated, the auxiliary transformer breaker will open and fast bus transfer to the start-up transformer will be initiated. Should the fast bus transfer fail, a LOOP would be generated and the sequence above would not occur. The resultant event is essentially a "simultaneous" LOOP/LOCA and the sequencer would operate as designed. Therefore, the break would have to be of a specific size which would generate the conditions necessary to initiate SI at approximately 46 seconds after reactor/turbine trip occurs and the operators do not manually initiate the SI signal. The other mechanistic scenario, loss of the transmission system due to loss of a single unit is also very unlikely since the grid is operated in such a manner as to remain stable in any single contingency situation such as loss of a unit or a large transmission line. As part of the original safety review performed in November 1994 for the sequencer auto-test issue, FPL evaluated the impact of delaying safety injection for 10 minutes for a spectrum of SBLOCAs. Using the EPRI MAAP code, small breaks of 2 and 6 inch equivalent diameter were examined. For the 6 inch break, the accumulator would not deplete for more than 20 minutes and core melt would not be expected for more than 50 minutes. For the 2 inch break, the accumulator did not deplete and core melt was not expected to occur. It was judged that provided the accumulator had not depleted and SI was restored, core damage would not occur (i.e. peak clad temperature would not exceed 2200 degrees F). Westinghouse has subsequently performed a SBLOCA analysis using NOTRUMP, which is an NRC approved code, for a 2.3 inch equivalent diameter break (the break size was iterated on to obtain the proper delay between LOOP and SI) assuming safety injection is restored 10 minutes into the event. For this event, a peak clad temperature of 954 degrees F occurred at 1818 seconds (approx. 30 minutes) into the event. Based on this event sequence, additional time would be available to the operator beyond 10 minutes to restore safety injection. MITIGATION OF SEQUENCER FAILURE MODES Because the presence of an SI signal during sequencer testing (automatic or manual mode) may render the sequencer inoperative, the dependence on SI was the primary consideration for determining the five affected accidents. For each of the affected accidents, the EOPs were reviewed to determine what mitigating actions would be taken by the operator. The effectiveness of the mitigating actions was also assessed based on its sequence within the procedures. Upon initiation of any of the five affected accidents discussed above, the reactor would trip placing the operators in procedure 3/4-EOP-E-0, "Reactor Trip or Safety Injection." At Step 4 in EOP-E-0, the operator verifies whether SI is actuated or is required. If an SI is required, the operator verifies that HHSI and RHR pumps have started, or he is required to manually start these pumps in Step 8. These two steps are part of the immediate actions to be taken by an operator following a reactor trip. TEXT PAGE 16 OF 23 In addition, the foldout pages for EOP-E-0 contains specific reactor trip and SI actuation criteria which require operators to start the HHSI pumps. Therefore FPL concludes that for these five accidents, there is a high probability that timely mitigating actions would have been taken by the operators to activate safeguards equipment even if the sequencer had failed. To assess the operators' ability to accommodate sequencer test software logic defects, the Turkey Point Training Department constructed three different scenarios involving design basis accidents with failed sequences. The failure mode modeled was a failure of the sequencer to load safeguards equipment. These scenario runs were completed on November 5, 1994. The three scenarios were: 1. A LOOP/LBLOCA with Unit 3 sequences failed. 2. A LBLOCA with no LOOP, with Unit 3 sequences failed. 3. A SBLOCA with no LOOP, with Unit 3 sequences failed, Unit 4 HHSI pump breakers racked out, and the Unit 3 HHSI pump control switches in PULL TO LOCK on the Unit 4 control board. Six control room crews ran each of the three scenarios, for a total of 18 simulator exercises. The Training Department was primarily interested in determining how long it took the control room crew to successfully energize all available safeguards equipment. A summary of the control room crew response times follows: RESPONSE TIMES FOR FULL SAFEGUARDS INITIATION (IN MIN.SEC) CREW LOOP/LOCA LBLOCA SBLOCA SCENARIO SCENARIO SCENARIO A 2:40 2:30 2:45 B 2:00 2:10 1:40 C 2:50 1:30 1:30 D 8:00 1:30 1:55 E 4:40 3:15 1:05 F 2:50 1:32 1:20 The simulator training coordinator stated that the longest time required to initiate SI flow was during Crew D's 8 minute LOOP/LOCA scenario; it took them approximately 4 minutes. However, the sequencer defect is not present for LOOP scenarios. The longest non-LOOP response time was 3 minutes and 15 seconds. The longest time to energize all available ESF loads, even with a LOOP, was 8 minutes, which applies to the Containment Spray issue and the 480 Volt Load Center issue. An assumed operator response time of 10 minutes is therefore conservative. TEXT PAGE 17 OF 23 In addition to the scenario exercises described above, a review of earlier observations of operating crews in simulator training during July and August 1994 was made. These observations illustrated that it took each crew 4 to 5 minutes from event initiation to complete alignment of the required safeguards equipment associated with a full sequencer failure. Operator verification of SI, and HHSI pump flow, is performed within the immediate action steps (Steps 4 and 8 respectively) of EOP-E-0. The first 14 steps are memorized by the control room crew. In addition, immediate action steps are required to be re-verified by the operators. Therefore FPL concludes that the control room crew would be successful in timely initiation of HHSI pump flow in the event of a sequencer malfunction. PROBABILISTIC SAFETY ASSESSMENTS III.f TEST LOGIC DEFECT A probabilistic safety assessment was performed to estimate the safety impact of inhibited emergency sequencer operation due to a logic error in the software associated with the test feature. The assessment is based on the Turkey Point IPE Submittal and subsequent updates, and includes the effect of the failure of all four sequences. The recovery actions are added to the model for different scenarios, e.g., recovery for LBLOCA vs. SBLOCA. These operator actions are calculated based on the time available to do the actions (NUREG/CR-4550, Vol. 3, Rev. 1, Part 1), and the time it takes the operators to perform the actions obtained from a review of 3/4-EOPs-0 and from simulator scenario runs. The probabalistic safety assessment determined that the estimated change in the Core Damage Frequency (CDF) under the above conditions, with all four sequences inoperable, is 6.3E-6/yr. However, all four sequences were not inoperable at all times. Each sequencer is inoperable during 5 of the 16 tests. In order for all sequences to fail simultaneously, all sequences would have to be in an affected test. This would happen most often if all four sequencer test cycles were synchronized. Even if all four sequences were synchronized on the same test cycle, the sequences would all be inoperable during only 5 of the 16 tests. Therefore, all four sequences would be inoperable approximately one-third of the time. This results in an estimated change in CDF of 2.1E-6/yr. This change in core damage frequency increases the baseline CDF by 3.2%. The PSA calculation considers an average probability over a one year period. The 3.2% increase in the CDF is a conservative estimate for this situation. This increase in CDF is not safety significant, based on the acceptance criteria stipulated in the draft EPRI PSA Application Guide. The estimated risk impact of loss of sequences for LBLOCAs is relatively low due to the low initiating event frequency of LBLOCAs, and recovery actions described in the early steps of the EOP E-0 for reactor trip and SI. Although SBLOCAs have a higher initiating event frequency the risk is relatively low because the operator has more time available to perform recovery actions. TEXT PAGE 18 OF 23 III.g CONTAINMENT SPRAY AUTOSTART ISSUE An estimate of the potential risk impact of the failure of the CS pumps to automatically start was performed. The scenario is assumed to occur for a certain size LOCA or MSLB such that the HHCP signal is generated at the 12.9 to 13.0 second window during which the sequences may not actuate CS pumps automatically. A further assumption is that failure of all containment spray with a medium LOCA leads directly to core damage. The core damage frequency increase is thus estimated to be: CDF = (frequency of event [medium and small LOCAs, MSLBJ) times (probability of "right size" break to cause the event) times (probability of failure of manual starting of CS pumps) = (1.0E-4 + 1.0E-3 + 1.0E-4) x (5.0E-4) x (6.0E-3) = 3.6 E-9/year Note that the frequency of the event is conservatively estimated to be that of the medium LOCA (6-13.5 inches), the small LOCA (2-6 inches) or a MSLB. Since a specifically-timed LOOP would be required for either the small LOCA or the MSLB to be of concern, the CDF is actually lower. A estimated increase in the CDF of 3.6E-9/yr is insignificant compared to the baseline CDF of 6.63E-5/yr. III.h 480 VOLT LOAD CENTER ISSUE To provide a bounding estimate of the probability of such an event over the half second interval of interest, the following expression is used: Probability of a certain size LOCA * Conditional probability of induced LOOP which coincides with the LOCA that satisfies the certain half second interval. 1.26E-3/Yr * 1.0E-3 * .5/60 = 1.05E-8/Yr where: 1.26E-3/Yr = the frequency of a small-small LOCA, plus stuck open Power Operated Relief Valve (PORV) (not recoverable) 1.OE-3 = the probability of LOOP given a reactor trip .5/60 = the exact timing fraction for the certain break size that results in scenario of interest (see below) This calculation includes all the LOCA scenarios that can generate an SI signal between 15.5 and 75.5 seconds after the reactor trip (the LOOP is postulated to occur from 0 to 60 seconds after the reactor trip). Considering the timing between the trip and SI actuation, FPL concluded that there are two scenarios that can cause such events; (1) a small-small LOCA (between 3/8" and 2.5" breaks); or (2) a PORV sticks open and hardware or operators fail to isolate the leak. For the case of a small-small LOCA, the initiating event frequency is 1.0E-3/Yr . For the case of one PORV sticking open, the initiating event frequency is 2.6E-2/Yr. Normally the PORVs close if the pressure drops below 2000 psi; if they fail to close, the operator can block the leak by using block valves. A recovery action failure probability of .01 can be conservatively assumed for controlling the PORV opening. Therefore, the overall non-recoverable PORV sticking open initiating event frequency may be estimated as 2.6E-4/Yr (2.6E-2 * .01 = 2.6E-4). TEXT PAGE 19 OF 23 Based on a study performed by the Federal Power Commission, the probability that offsite power would be lost as a result of the generator trip caused by a LOCA is estimated to be 1.0E-3. For the specific break size to coincide in a certain half second interval the fraction of LOOP then SI event timing may be estimated as .5/60. This estimate assumes a uniform distribution for the coincident LOOP and SI signals within a 60 second interval. Note that the above estimates are conservative in several aspects. First, all SI events may not present the same degree of challenge to the plant safety systems. Secondly, no operator actions are credited for mitigating the core damage scenarios. Thus the scenarios initiated by a LOCA followed by a LOOP and subsequent actuation of SI contribute at most 1.05E-8/Yr to the CDF. These scenarios are not considered safety significant. if operator recovery action is considered, the core damage frequency would be expected to drop one to two orders of magnitude. Comparing this event to NRC stated safety criteria for the industry shows that this event is several orders of magnitude less severe than that which would require NRC action. Comparison to industry developed PSA criteria shows that this scenario is several orders of magnitude below that which would require action. SAFETY SIGNIFICANCE AND OPERABILITY III.i TEST LOGIC DEFECT The periodic inoperability of all four sequences, as described above, has existed since the sequences were installed during the dual unit outage in 1990/1991. The sequences were accepted as operational in September and October, 1991, for Units 3 and 4, respectively. From early December, 1991, until November, 1992 (Unit 3) and May, 1993 (Unit 4) the sequences' Test Selector switches were in OFF except for monthly manual tests, as described in LER 251/91-007. Since then, there have been four challenges to the bus sequences (between the two units). LER 251/92-004 reported an inadvertent Safety Injection on Unit 4; all plant equipment responded as designed, including the Unit 3 HHSI pumps. LERs 250/92-009 and 250/92-013 reported a LOOP (due to hurricane Andrew), and an inadvertent 3A bus stripping. In these three instances the sequences Test Selector switches were not in AUTO, and they performed as designed. LER 250/94-002 reported an inadvertent ESF actuation on Unit 3, in which all equipment responded as design, except the 4A HHSI pump. At that time the failure of the 4A HHSI pump was attributed to an intermittent failure, which could not be reproduced. As a result of the discovery of the defect reported herein, that earlier event can now be reproduced at will on the sequencer simulator. FPL believes that the 4A HHSI pump failed to start because of the same defect that caused the 3A HHSI pump failure to start, reported in this LER. Since there have been no actual events requiring Engineered Safety Features actuation to protect the plant, the health and safety of the public has not been affected by the periodic inoperability of the sequences. This event is reportable under the requirements of 10 CFR 50.73 (a)(2)(i)(B), (a)(ii)(A), (a)(ii)(B), (a)(v), (a)(vii), and 10 CFR 21. TEXT PAGE 20 OF 20 III.j CONTAINMENT SPRAY AUTOSTART ISSUE Regarding the second software error involving the CS pump autostart, FPL has concluded that the CS system remains OPERABLE because, in the highly unlikely event that the condition were to occur, simple operator action to start the CS pumps, in accordance with the plant's emergency operating procedures, would ensure compliance with the system specified functions. The ability to manually start the CS pumps as much as 10 minutes into the event and maintain required cooling is supported by analysis, procedures, and training. In addition the safety significance of the evaluated condition is extremely low because the probability of the evaluated condition is lower than the probability of a common mode failure of both CS trains, as discussed earlier under Possible Accident Consequences for Sequencer Failure Modes. In any case, the contribution to CDF of this software error is negligible. III.k 480 VOLT LOAD CENTER ISSUE Similar arguments obtain for the condition involving the 480 Volt Load Center feeder breaker autoclosure failure. The probability of occurrence of the specific scenario is very low. The contribution to CDF is similarly very low, neglecting any mitigating operator action. Nevertheless, should the scenario occur, simple operator action, again as much as 10 minutes into the event, drops the CDF by one to two orders of magnitude. In accordance with Generic Letter 91-18, a licensee cannot replace automatic action with manual action if the automatic action is needed to avoid exceeding a "Safety Limit." Safety limit is defined in 10CFR50.36, as is limiting safety system setting. "Where a limiting safety system setting is specified for a variable on which a safety limit has been placed, the setting must be so chosen that automatic protective action will correct the abnormal situation before a safety limit is exceeded." Turkey Point's safety limits and limiting safety system settings are defined in Technical Specifications. The limiting safety system settings are reactor trip setpoints. There is no reactor trip setpoint on 480 Volt Load Centers. The only two safety limits are reactor pressure, and a graph combining pressure, Ta sub avg, and reactor power. Neither of these are challenged by the loss of the automatic re-energization of the 480 Volt Load Centers. Therefore manual action can be credited to determine the operability of the 480 Volt Load Centers, if it can be shown that such action (1) is proceduralized, and (2) is not heroic. Emergency Operating Procedures 3/4-EOP-E-0, Reactor Trip or Safety injection, direct the reactor operators to verify ECCS flow, and provide guidance to get flow in the Response Not Obtained column. The actions are taken from the control board, are part of the standard training of reactor operators, and involve no hazard. Indeed, when the scenario in question was imposed on several crews on the simulator, their delay in re-energizing the load centers was in waiting for the sequencer to finish sequencing. Thus the actions (1) are proceduralized, and (2) are not heroic. Therefore, FPL concludes that the 480 Volt Load Centers remain operable. TEXT PAGE 21 OF 23 IV. CORRECTIVE ACTIONS 1. The Test Selector switches on all four sequences were placed in OFF. Tags have been hung on each switch to require specific permission from the Nuclear Plant Supervisor to change the position of the switch. With the sequencer test mode switch in the OFF position, the automatic test logic is disabled. The sequencer is fully functional and will respond properly to input signals. The automatic test function is not a requirement for periodic surveillance of the sequencer. 2. With the Test Selector switch in OFF, additional visual inspections are being performed on a eight hour basis as described below: a. The local reflash annunciators points are verified not in alarm. b. The I/O power, PLC Power, and ANN Power switches are verified in the ON position and the Processor Power white indicating light is verified illuminated. c. The Test Selector switch is verified in the OFF position; the Stripping Clearing Test Selector and Sequencing Mode Test Selector Switches are verified in the OFF position. d. The 2 green test reset indicating lights and the sequencing reset green indicating lights are verified illuminated. e. The other indicating lights are verified not to be illuminated (except the ground fault indicating lights are supposed to be dimly lit). f. Every 24 hours, the sequencer door is opened, the Processor Indicator LED is verified to be a solid green and the 9 indicator I/O cards "ACTIVE" LED are verified to be a solid green. 3. A detailed review of the original Validation and Verification process was performed; it has been concluded that an oversight occurred because not all sequencer functions were validated during all modes of automatic and manual testing. The existing verification and validation sufficiently covers the sequencer safety functions if the Test Selector switch remains OFF. 4. Functional testing on the sequencer simulator of design basis inputs has been repeated with the Test Selector switch OFF, with acceptable results. 5. A safety evaluation has been issued demonstrating sequencer operability with the test selector switch in the OFF position. This safety evaluation was approved by the Plant Nuclear Safety Committee on November 4, 1994. 6. Independent consultants were retained to perform an assessment of the existing sequencer design, software design and V&V. This "Independent Assessment Team" (IAT) concluded that operation of the sequences with the Test Selector switch in OFF represented a safe condition and that FPL's evaluation of the condition was appropriate. The second phase of the IAT's assignment was to provide a detailed review of the software documentation. Some drawing discrepancies were identified and have been evaluated. In general the discrepancies TEXT PAGE 22 OF 23 dealt with the inclusion of additional information on the logic diagrams not reflected in the ladder diagrams, to aid in understanding the logic diagrams. One other software error was identified involving autostart of the CS pumps, and has been discussed earlier in the LER. The drawing discrepancies will be corrected when the software is modified (see Corrective Action #9 below). The IAT confirmed that the V&V was not comprehensive enough to test certain aspects of the logic. "The plan was weak in that it relied almost completely on testing as the V&V methodology. More emphasis on the analysis of the requirements and design would have increased the likelihood of discovering the design flaw." A revision to the V&V documentation will be made coincident with the design modifications described on Corrective Action #9 below. 7. The original software vendor, United Controls, Inc. has been notified of this defect and its significance. 8. In order to eliminate issues related to the use of one-of-a-kind or first-of-a-kind equipment, FPL implemented Nuclear Policy NP-905, Equipment Selection, in October of 1991. This policy states in part that, "FPL's nuclear engineering department shall select only specific models of equipment with proven records of reliable performance for use in FPL nuclear facilities. Verification of the equipment reliability must be established through contact with NPRDS, nuclear station managers, or other appropriate sources. If no prior operating experience is available, appropriate prototype testing, under equivalent plant operating conditions, must be undertaken to establish its reliability before it is placed in service at FPL nuclear facilities." The Engineering Quality Instructions contain the Nuclear Policy requirements for design outputs. 9. Design modifications to eliminate the identified problems will be implemented during the next refueling outages of each unit. 10. Other safety-related process computer suppliers were notified of the event on November 14, 1994. These suppliers responded that similar software errors do not exist in other safety-related process computers. 11. An FPL Nuclear Engineering standard will be developed on the use of PLCs, prior to the procurement of any additional PLC-based equipment. 12. Manual testing of the sequences was resumed on January 11, 1995. 13. Emergency Operating Procedures 3/4-EOP-E-0, Reactor Trip or Safety Injection, have been revised to require the operator to verify that the Load Centers associated with the energized 4.16 KV bus(es) are energized. V. ADDITIONAL INFORMATION EIIS Codes are shown in the format [EIIS SYSTEM: IEEE component function identifier, second component function identifier (if appropriate)]. The Programmable Logic Controllers used in the sequences are made by Allen-Bradley; the sequences are assembled by United Controls, Inc. (UCI). According to UCI, Florida Power & Light Company is the only utility to which UCI supplied this sequencer. TEXT PAGE 23 OF 23 The condition wherein the 480 Volt Load Center feeder breakers may not close automatically may have generic implications not associated with digital load sequences. It appears that any time such a breaker is presented with conflicting simultaneous close and trip signals, if that breaker has an "anti-pumping" circuit like the one described in this report, that breaker will not close. FPL is not able to determine if such conflicting signals may be generated by an analog or "relay-based" load sequencer system. *** END OF DOCUMENT ***

Page Last Reviewed/Updated Tuesday, March 09, 2021