Insights

Flixborough - the change that changed everything

The first day of June 2024 was the 50th anniversary of the Flixborough disaster. Whilst not the first serious accident to occur in the process industries, it represented a watershed moment in the development of Process Safety and brought about improvements and practices which are still relevant 50 years on. Perhaps the practice most closely associated with the Flixborough disaster is Management of Change (MoC).

The hydrocarbon release which caused the massive explosion was the result of the failure of some temporary piping connecting Reactors 4 and 6 in the Caprolactam plant. Famously, no engineering drawing of the pipe was done, just a chalk outline on the workshop floor to help the fabricators. Because this action is now dismissed as grossly inconceivable - and it would have been inconceivable in most chemical plants even in 1974 - it has always been difficult to inspire people to improve their MoC systems using Flixborough as an example. They simply cannot believe such a thing could happen at their plant.

But there was another major MoC failure that led up to the Flixborough disaster, a more subtle one, the type that Risk Engineers and other Process Safety professionals encounter in many facilities all too often.

In November 1973, several months prior to the explosion, a leak developed in the top-mounted agitator gland on Reactor 5, causing hot cyclohexane vapour to leak out. A temporary deluge ring was mounted above the reactor to spray plant cooling water to condense the vapours, and this water soaked the reactor insulation. This water contained sufficient levels of nitrates to induce stress corrosion cracking (SCC) on the hot external surface of the reactor. A crack propagated vertically to more than 2 m in length and eventually penetrated the internal reactor cladding, leading to a small leak of cyclohexane in March 1974.

The leak was detected, the reactor removed from its plinth, and the rest is history. The lack of MoC meant that no one realised that the water-soaked reactor insulation introduced a corrosion hazard. Writing about this water spray situation some years after, Trevor Kletz noted that “the more innocuous a modification appears to be, the further its influence seems to extend”.

Recognising subtle change will always be a challenge, but with the right Loss Prevention mindset and systems it is perfectly possible to manage every change, every time.

Ignorance is certainly not bliss

What one company's crisis can teach us about more effective control of risk and dissemination of knowledge

What connects a 15th century German theologian, the Danish pharmaceutical company Novo Nordisk and the concept of High Reliability Organizations (HROs)? It's the idea of Learned Ignorance, a concept which provides us with a different perspective on improving Process Safety Management in the process industries.

Novo Nordisk’s brush with disaster

In 1990 the Food & Drug Administration (FDA) standards for ‘current good manufacturing practice’ (cGMP) relating to pharmaceutical companies was updated. Four years later in 1994 an FDA inspection at Novo Nordisk uncovered more than 100 incidents of non-compliance to the amended cGMP regulations. As a result, the company was at risk of being suspended from selling its main product - insulin - on the American market. To resolve the situation in the short term an agreement was made to discard a full six months of production, which required Novo Nordisk’s management to approach their largest competitor Eli Lilly and ask them to step in to fulfil its supply obligations, effectively giving away a 45% market share for free. The CEO, Lars Fruergaard Jørgensen, said at the time “this was just the last, the only thing, that was not to happen”.

Well, then how did this happen?

The updated cGMP requirements were known within Novo Nordisk, just not at the right level. A number of engineers, scientists and supervisors learned about the new rules when they participated in US conferences and a number of internal reports were published, concluding that Novo Nordisk production facilities would need improvements, alongside upgrades to validation and quality control processes.

But at this time Novo Nordisk was a very hierarchical organisation, and this message from the frontline had a long spiralling journey to make until it reached the relevant decision makers at the lofty heights of the executive management. Successive layers of middle management filtered the reports, gradually toning down the seriousness and immediacy of the message, meaning that by the time it reached the ears of senior management it appeared that nothing needed doing. As the organisational psychologist Robert Sutton once observed, in a steep hierarchy it is a happier and happier story that reaches the top ranks.

In the aftermath, Henrik Gürtler, then the HR Manager, said “I can not tell if it was caused by incompetence among the management or communicative inertia. The embarrassing fact is that the work, which was clearly necessary, did not happen. During the 1980s we experienced tremendous success. But the fatter an organisation becomes the harder it is to detect new signals. We experienced limited competition and everything was fine. As a result of this situation we were not paying particular attention to the [signals from FDA] in the early 1990s”.

Inspiration from the 15th Century

The board and the executive leadership team realised that there were two fundamental problems that had allowed Novo Nordisk to sleepwalk into this crisis. The first was a culture of arrogance within middle management, who assumed they knew what top management needed to know and who did not pay attention to the experts that worked for them. The second was that senior management had no direct ties to the lower levels of the organisation, resulting in an ever widening gap between what should happen and what actually happens.

When considering how to tackle these issues they were exposed to a set of ideas originally developed by Nicholas of Cusa, a cardinal, theologian and philosopher in 15th century Germany. He published his famous work De Docta Ignorantia (Of Learned Ignorance) in 1440, which examined ignorance and its relationship to knowledge. His core thesis was that ignorance highlights the limitations of human knowledge, and as a result provides the focus and the incentive for the actual pursuit of knowledge. “For a man – even one very well versed in learning – will attain unto nothing more perfect than to be found to be most learned in the ignorance which is distinctively his. The more he knows that he is unknowing, the more learned he will be”. Knowing that you don’t know everything is very important indeed.

The management at Novo Nordisk sensed that the concept of Learned Ignorance could help them get back on the right track. If they could seed a culture which understood that perfect knowledge is non-existing, while believing that knowledge gaps also can be detected and managed, and treats knowledge that worked in the past with requisite humility, then they could prevent Novo Nordisk from making the same mistake again. Henrik Gürtler was given a carte blanche to develop and implement a new system based on this concept.

Novo Nordisk Way of Management

Out of this was born the Novo Nordisk Way of Management (NNWoM), a set of values, business principles and commitments which are binding for all employees and managers worldwide. Based on that description, nothing really revolutionary - even in the 1990’s such management systems were common - but the devil is in the detail. The most interesting element of the NNWoM to me is the process of Facilitations.

Facilitations build direct ties between senior management and the shop floor, helping to detect and manage knowledge gaps by creating variations in how knowledge and opinions flow through the company. A facilitation is not an exam or an audit, but a service for local management to help them identify evolving flaws, knowledge gaps and inconsistencies. They are conversation-based assessments which are intended to gauge how well the NNWoM is embedded in the day-to-day practices at all company sites.

Facilitations are, appropriately, run by people called Facilitators, an internal group which holds a very special place in the Novo Nordisk organisation.

Facilitators, mount up

There are 14 Facilitators at Novo Nordisk, led by a Senior Vice President who reports directly to the CEO. To apply for a Facilitator position one must have attained at least Vice President level, so fairly senior, and have a minimum of 7 years assignment experience at a Novo Nordisk site outside one’s own country. They must have a good credibility within the organisation - because they will end up telling people how to run their business unit based on the company values - and be highly empathetic - as they will need to build trusted relationships with many, many people across the organisation.

As guardians of the Novo Nordisk Way of Management, they go around the world to understand how well the Way is embedded and to identify disconnections between what management believes is happening and what is actually occurring. They are on location around 120 days annually in teams of two that constantly change to combine different skills and experience. Within each pair there must be at least one with a long record of Facilitations, one who knows the Unit’s business, and one of them knows the Unit’s national culture.

They perform analysis of complex organisational problems, process flows, managerial- and cross-functional issues and provide temporary implementation support. The Facilitators conduct informal interviews with a minimum of 40% of the employees in a unit, with up to 100% at smaller sites.

Local management sometimes disagrees with the findings generated; for example, in one case the findings indicated that communication within the Unit was dysfunctional. The local manager suggested that the Facilitators had only talked with people who misunderstood what he said. The Facilitators turned the argument around and pointed out that if the employees were misunderstanding what was being said, it was the local manager’s problem. “We did not allow this manager to preserve his arrogance. We tossed it right back in his face“ said one of the Facilitators. Spoken like a true Scandinavian…

Are we adapting this to Process Safety Management?

The leadership teams of most multinational companies in the process industries use multiple means to assure themselves that the right things are happening at the coalface and that knowledge is being disseminated appropriately. Audits, regular training sessions, internal technical networks, peer assists, things like that. But how many have a dedicated department, staffed with some of their most experienced and talented employees, who go around the world talking to process operators, maintenance technicians and junior engineers just to make sure that everyone is aligned to the company’s loss prevention principles?

I was blessed to have spent the first eight years of my career with ExxonMobil, whose core management system (OIMS) was born out of the crisis that was the Exxon Valdez oil spill in 1989. Just like Novo Nordisk, they admit that their systems and practices stem from a crisis. Just like Novo Nordisk, they have a genuine commitment to never letting something like that happen again. But they stick with the tried and tested routine of conducting regular audits and promoting internal networks as a means of control and dissemination. They don’t have something which resembles what Novo Nordisk has done with Facilitations and Facilitators.

And they are far from alone. Most of the companies that I have visited in the past decade have audit processes - some of which were pretty ineffective compared to ExxonMobil’s to be frank - while only a few of them have functioning internal networks for technical and professional knowledge sharing. But, like ExxonMobil, none have anything like the Facilitation process.

I think there is a strong case for adapting this practice for any company which is managing major accident hazards at multiple locations. In fact I’d go so far as to say there is no real excuse not to.

Key takeaways

The way in which we organise knowledge puts limits on the knowledge that we can organise. When companies don’t create conditions that support the sharing of experiences, knowledge, and perspectives from a wide range of employees, they increase their fragility and vulnerability. Novo Nordisk learned the hard way that it is easy for a disconnect to arise between the decision makers in the executive leadership and key experts at the shop floor, and this can lead to real trouble.

Learned Ignorance, as outlined by Nicholas of Cusa, is the idea that our knowledge can never be complete, no matter how senior, experienced or organised we think we are. By developing this attitude an organisation acknowledges that it is ignorant, which is not the same as saying that it does not know anything, but that it needs active processes - such as Facilitation - to detect and manage knowledge gaps.

25 years of the Novo Nordisk Way of Management (NNWoM) shows that developing and maintaining a culture that supports knowledge exchange and humility has created a ‘collective mind’ which delivers sustainable results. Facilitators are change agents that continuously help Novo Nordisk to circulate knowledge and learn from both success and failure, and this concept could provide real value to Process Safety Management (PSM) in the process industries if it were adopted.

Preparing for emergencies? Then get real…

Too often we find ourselves simulating emergency response scenarios in ideal, tightly controlled environments. We give advance notice to participants of drills, which puts them in a wholly different mental place than if the drill was unannounced. We often spend more time analysing the risk of personnel injury due to conducting a simulation than we do in preparing the scenario to adequately test the capabilities of responders. And so we end up with a dress rehearsal rather than a live action spectacle, a canvas that is more paint-by-numbers than inspired by imagination.

How then can we have confidence that our well thought-out emergency response procedures will work as intended when we simulate these events without raising the heart rate, without generating a little bit of sweat on the brow? Truth is, we can’t.

Training that doesn’t elicit a stress response is not adequately preparing the people that you rely upon during an emergency for decision making under pressure. We must insert controlled stress into our drills and simulations otherwise we render our responders highly vulnerable to the effects of pressure and nerves.

Preparedness makes us powerful - or does it?

Emergency response drills - whether they be field simulations for fire or leak scenarios, or desktop simulations of control operator response to a loss of power - are typically simplified compared to how they would actually unfold in a real event. These simulations are often signalled in advance - especially if they are for major scenarios - and tend to escalate progressively and linearly, rather than erupting suddenly on a broad scale with unpredictable complexity. Drills also lack the realistic conditions of real emergencies, for obvious reasons - it would be a little strange to set fire to the middle of an oil refinery in order to see how well the Fire Team sets up their mobile equipment, or to cause a process unit trip in order to see how well the Control Panel Operator and Field Technicians react...

These factors mean that we don’t actually know how our personnel will perform under the stress of a real event. Some people may feel unable to cope with the continuously increasing demands of an emergency situation, others may thrive. It is vitally important to establish this before an emergency, and not learn it during the midst of one. As Mike Tyson famously said in 1987 in the build up to a fight with Tyrell Biggs, “everybody has plans until they get hit for the first time”.

So whilst we may think that we are prepared because we conduct multiple major emergency drills every year, we may actually be lulling ourselves into a false sense of security because we simply don’t know how our operators and fire teams will react.

What stress does to humans

During times of extreme stress, the body's sensory and cognitive equipment simply does not respond the same way that it does when we are calm and controlled. When confronted with an emergency situation, especially for the first time, our physiological functions can quickly move beyond our own control simply due to psychologically-induced stress.

When the heart rate rises above 145 bpm you can experience tunnel vision, lack of depth perception and deterioration of motor skills. You are also prone to auditory exclusion and a loss of global situational awareness at this point, making it easy to miss an important radio transmission or an instruction from a colleague.

Above 175 bpm, you enter a full “fight, flight or freeze” response. Blood will drain from the brain’s rational control centre (the forebrain), leaving the midbrain in full control. The midbrain doesn’t know rational thought or decision-making processes; it knows action. At this point, you will do only what you have trained to do, nothing more. People have been known to repeat a given action that they have recently used again and again, seemingly stuck in an endless loop.

Although our physiological response to situational stress cannot be prevented - it is instinctual - it can be mitigated. The more you are exposed to something stressful (within limits), the less it will affect you and the less likely your body’s response will hinder your performance. This is called stress inoculation and it underscores the importance of realistic training for all personnel who are expected to respond to emergencies at industrial facilities.

Keep calm & carry on

In the early 1970s, Donald Meichenbaum, a Canadian psychologist, developed stress inoculation as a training program to help deal with stress and reduce anxiety in his patients. The technique gained popularity in the military special forces, NASA and municipal fire fighting squadrons, where the tolerance for mistakes under pressure is virtually zero.

The essence of stress inoculation is that by exposing people to increasing levels of perceived stress they will find suitable coping skills from within themselves and eventually develop increased tolerance - or immunity - to a particular stimulus. Through the controlled exposure of stressors that simulate real-world emergencies, people can become better prepared to physiologically remain calm and collected if the real situation arises. When you are less stressed, you process more of what is happening around you and therefore make better decisions that lead to better outcomes.

Stress inoculation should be a feature of the training for any person who will have a role to play in an emergency situation at an industrial facility - from the incident commander and firefighters, to the control room operator and the field technicians.

Building immunity from stress

EDUCATION

The ultimate aim is to hold emergency simulations and drills which have more realistic levels of stress and pressure. But before the first fake alarm is sounded, your personnel need to be aware of what they are likely to face; there is no value to be had in surprising them.

Participants need to be informed that during emergency simulations under stress a deterioration in their faculties is normal; it is nearly universal, it is a natural result of our psychology and it is no way a sign of weakness or inadequacy. But by repeatedly exposing oneself to controlled stressors we can build up a sort of ‘immunity’ to the negative effects of stress, and gain access to the positive aspects, such as higher levels of energy and better decisiveness of action.

There is a body of contemporary research which strongly suggests that having a ‘stress-is-enhancing’ mindset allows people to better sustain themselves during stressful challenges, whereas holding onto a ‘stress-is-debilitating’ view will lead to disengagement and self-defeat. How you frame something in your head has a great influence on your neurobiological response to it… when you say to yourself “I know what to do here”, then that turns into a much more positive approach to the situation at hand.

Another preparatory aspect is to teach emergency response participants about the best breathing technique for calming oneself when the adrenaline starts pumping. When you’re under stress and feel your heart rate picking up uncontrollably, take four full seconds to draw a deep breath. Hold that breath for four seconds, and then exhale for the next four seconds. Pause for another four seconds before repeating the entire 16-second sequence at least three times. This technique slows the heart rate and brings the stress response under control, bringing back mental clarity and manual dexterity.

EXECUTION

The closer the emergency training scenario resembles the real thing, the greater the performance carryover into real life will be. So when drills are designed, consider including some of these elements in order to bring in additional stressors for participants;

Introduce purposeful distractions or annoyances - someone sitting on the radio button, nuisance calls to the control room panel, vehicles turning up at the cordon, etc.
Make the situation fluid, so that participants have to react - for example, whilst an emergency response crew has just set up to attack a rim seal fire on a floating roof tank, an order is given that the roof looks like it is about to sink and a full surface fire will inevitably commence, and have the crew change their attack plan immediately. Change the wind direction in the middle of the event that necessitates a fire team retreat and a second attack, etc.
Introduce elements of disorder - such as removing the radio from one of the Field Operators to simulate a loss of communications and see how that individual responds. Place a section of hose in the water supply line with a hole in it (obviously advise the fire truck operator ahead of time so minimal pressure is put on the line) and observe what happens when the crew realises they have a burst section to replace. Tell the Control Room Operator that the inert gas suppression system inside the control room has activated, and that they need to continue their work whilst wearing SCBA. When disorder is introduced in training, decisions will be made under less-than-ideal conditions, providing valuable lessons for all those involved.
Create a more realistic environment - this can mean things like artificially produced noise that makes communications more difficult and which makes structured thought more challenging; inserting panicked or angry participants who distract the Incident Commander; using props to make injured parties appear to have serious wounds, even up to the point of using actors to make more realistic injured parties, screaming in pain, clawing at first responders, not letting them leave their side, etc. Most drills happen during the day time, but major emergencies don't respect normal working hours so throw in a few drills per year during the night.

Even something as simple as having a lot of people observing (from a safe distance) can raise the pressure level significantly on the emergency drill participants. Based on the ‘social pressure hypothesis’, it has been shown in many different environments - from football players taking penalties to college students conducting difficult mental arithmetic tasks - that an audience of peers accidentally induces more stress on participants than even an adversarial, screaming audience.

Fire team members should be afforded annual live burn training as they - unlike their Civil Defence and Municipal Firefighting compatriots - have only infrequent experience of real fire situations. Training should be conducted in accordance with NFPA 1403 Standard on Live Fire Training Evolutions and made available annually if possible.

EVALUATION

And, finally, during the drill wash-up, it is important for everyone to discuss how they individually responded to the simulation. Did you feel yourself momentarily lose control? Were you hyperventilating? Were your hands shaky? If the education step has been done effectively then participants will know that there is no shame - only acclaim - in sharing the negative effects that stress had on them during the drill. Everyone should record their own responses and then monitor how these responses evolve across subsequent drills as they come more and more inoculated to the effects of stress.

Key takeaways

Practicing for major emergencies is an essential component of managing any facility with major accident hazards, but these simulations frequently lack the realism needed to truly familiarise participants with the levels of stress, fear and tension which arise in the midst of a serious incident. As Brasidas of Sparta put it nearly two thousand years ago, “Fear makes one forget, and skill that cannot fight is useless.”

Stress inoculation is a concept which emphasises the need to add obstacles, distractions and elements of realism into these emergency drills in order to get personnel acquainted with the chaos and pressure which inevitably rise up during an incident. When we design simulations which catch participants off guard it will help them learn to anticipate and work through disorder. We need to learn to not lose our heads when things go awry. Things will go wrong, and they will go wrong at the worst time possible. So be prepared.

Problems breed problems breed more problems...

Problems breed more problems. Perhaps that should be an addition to Murphy’s Law.

A wind power Client in South-East Asia suffered two nacelle fires in quick succession at one of their locations on the coast of Sri Lanka. In both cases the forensic evidence pointed to the fire beginning with a short circuit in the variable rotor resistor bank, a device which increases the ‘slip’ between the rotor and the generator to offset sudden gusts of wind. The OEM was perplexed; it had produced 2,738 turbines of this model over the previous 18 years and had only experienced two other resistor bank short-circuits within the fleet. How was lightning striking twice in Sri Lanka? (clue, it was nothing to do with lightning).

The cause began to reveal itself when it was announced that the 20 turbines at that particular location had been offline for almost five months in the year preceding the fires. This was due to a grid step-up transformer failure at the export substation, which was owned and operated by the national electrical board, followed by a protracted outage period while a replacement was procured and installed. Not being able to speed up this process, our Client could do nothing but sit on its hands while their wind turbines stood motionless in the breeze.

But what they hadn’t appreciated was that conditions within the nacelle would be anything but motionless. Being in a tropical coastal location, the atmosphere was humid and salt-laden, and because each machine was offline there wasn’t the same high, steady temperature within the nacelle as during normal operation. And so the temperature went up and down throughout the day, causing condensation of atmospheric moisture during the night and evaporation during the day, a cycle which repeated throughout the five month period that the grid step-up transformer was missing. Not ideal for circuit boards, wire terminations and power electronics. Given the fact that these two outlier failures happened shortly after this period it was deemed highly probable that this was the underlying cause in both instances.

Had this operational change (i.e. sitting idle for an extended period) been properly recognised, the potential damage mechanism (i.e. corrosion due to condensation and salt laydown) could have been identified and then mitigated, by installing a portable dehumidifier or preserving the most sensitive power electronic components underneath water-vapour-proof protection with desiccant.

But the original problem - the business disruption due to the inability to export power - had dominated management conversations, allowing this knock-on problem to creep up silently in the background until - BANG - they experienced two very costly nacelle fires.

Analogous situations tend to be more common than we would like. The knock-on problems can be relatively immediate, such as in the famous incident at the Texaco Pembroke Refinery in 1994, where a power loss due to lightning strike in the morning inadvertently led to a massive explosion in the afternoon, largely because the management were wholly focused on restoring the plant to normal operation on the same day, blind to the consequences that were rippling through the rest of the facility. Or they can be slow burners, like at a refinery in the Middle East which I visited which had suffered a Crude Distillation Unit main column fire due to a leak through the upper part of the tower shell caused by chloride-induced corrosion, which itself was a result of a crude oil desalter level controller problem which had occurred several months before.

When a problem occurs it is imperative to give some thought to how this problem can breed more problems, both immediately and further down the line, as well as getting on with the business of resolving the situation at hand.

Our brains tend to be good at getting to solutions quickly; it makes us feel comfortable to restore order to where there is chaos. But that can mean missing the wider picture. Site-wide shutdowns due to power loss are rarely incident free, so don’t take out the start-up procedure as your first action. Any operational change which needs to be taken to mitigate a longer-term issue should be subject to Management of Change (MoC), even if the implied change appears to be minor. And if equipment is to be shut down for an unusual length of time, thoughts should be turning to those novel damage mechanisms that can occur at typical atmospheric conditions which are not a problem when the equipment is at normal operating conditions.

Problems are thrown at us continuously in the Energy & Process Industries. Where best-in-class operators gain their advantage is in stopping these problems creating knock-on problems, because they are attuned to the reality that problems breed more problems. They recognise that one important aspect of resolving an issue is assessing where the original failure is negatively impacting other parts of the system and taking appropriate measures to nip those future problems in the bud. The alternative is to be chasing your own tail, and nothing good will come of that!

Page updated

Report abuse