IAMC

INDUSTRIAL ASSEST MANAGEMENT COUNCIL

Rely on Lessons Learned for a Robust Disaster Recovery Plan

By John Salustri

 

2102

Brett Manning

“The events of 1987 were over in a matter of minutes. But it took us years to recover. That sort of impact is something companies can easily underestimate.”

The speaker is Brett Manning, senior VP of Global Corporate Services for Silent-Aire in Edmonton, Alberta, Canada. The “events” he refers to were actually a single F4 tornado that ripped through Edmonton, packing top winds of 260 mph. Included in the estimated US$647 million in damages was the Silent-Aire global headquarters. “It wiped out the facility.” July 31 is still referred to as Black Friday by the Edmonton community.

https://www.tornadotalk.com/edmonton-alberta-f4-tornado-july-31-1987/

But Manning is a big believer in lessons learned, and as the provider of hyperscale cooling and modular data centers mounted a global growth initiative, leadership baked into those plans an increasingly robust disaster preparedness and recovery program. “The tornado shaped and molded how we guide our operations,” he states. “We know now what starting over is like.”

It was an expensive lesson, and he says that just like his shop, other IAMC members should tailor the lessons they’ve learned into a workable, bulletproof disaster recovery program. “To me, every company should have a robust continuity and resilience program,” he says. “And a big piece of that should look at potential locality risks in various markets, factoring those in and taking the necessary steps to minimize those risks.”

Which, of course, shapes the site selection process. In 2013, the company expanded into the US, and in addition to the normal checklist (square footage, rental rates, availability of direct flights to Edmonton, etc.), “we also looked at potential risk factors for manufacturing facilities,” he says. The firm settled on Arizona (Manning is based in Gilbert). Extreme heat aside, the Grand Canyon State is fairly weather-neutral. The same approach was in place for Silent-Aire’s 2017 Ireland expansion.


“Every company should have a robust continuity and resilience program, and a big piece of that should look at potential locality risks in various markets.”

-- Brett Manning, Silent-Aire


The problem for all corporates, unless the decision-makers keep resilience “top of mind,” (as Manning says), is short-term memory. For instance, Matt Whittaker, managing director of Property Services for Bis Henderson Consulting in Northampton, UK, says another “event,” this one of human origin, was Brexit, and it taught very few people any substantial lessons about preparation.

“We didn’t really learn about the value of resilient supply chains then,” he says, “so we didn’t prepare wisely. Then COVID came, and there were very few plans for maintaining inventory.” (For those with short-term memories, Brexit was the UK’s 2016 vote to quit the European Union.)

But having a risk-mitigation program in place is only half of the discussion. “Any plan is only as good as the frequency you refer to it,” Manning adds. Indeed. Disaster recovery plans need to be living documents, shared with all stakeholders and updated as glitches and improvements are uncovered. The Silent-Aire plan is reviewed by the executive committee annually and road-tested quarterly, with tabletop exercises that involve multiple departments since each has its own “issues and perspectives. That allows us to prioritize how best to mitigate those risks. It’s a continuous improvement process.”

COVID and a Canal

It’s no exaggeration to say that every company around the globe was caught off guard in February of 2020 when COVID-19 swept the world. And while not every firm needs a plan as robust as Silent-Aire’s, it did assist the firm in minimizing risk.

2102

Matt Whittaker

“We were able to quickly implement all the lessons we’ve learned and get ahead of some of our virus-related situations,” he says. “And now we’re better prepared for the next pandemic.”

How prepared global supply chains will be for that next time can be measured by the availability of product on your local store shelves. As Whittaker indicates, supply chains were clearly starved in the first weeks of the pandemic, just as they were strangled when the 1,312-foot Ever Given cargo ship ran aground and blocked traffic in the Suez Canal for six days this past March.

One ship. A second global crisis. It’s estimated that $9 billion worth of product slides through the canal daily--some 12 percent of the global supply chain. Recovery from that event is expected to take months to fix.

https://www.cnbc.com/2021/03/29/suez-canal-is-moving-but-the-supply-chain-impact-could-last-months.html

“A lean supply chain makes practical sense when it’s business as usual,” says Whittaker. “Then COVID or the canal comes along, and what’s the first thing impacted?”

Lean supply chains, often done in the name of efficient procurement, can actually be code for saving money, he says, and he “prophesies against that. I think we’ve gone too far down the route of commoditizing the supply chain to lowest possible costs. The result is supply chains that aren’t resilient.” Here too, a robust and forward-thinking emergency prep plan is key.


“A lean supply chain makes practical sense when it’s business as usual. Then COVID or the [Suez] canal comes along, and what’s the first thing impacted?”

-- Matt Whittaker, Bis Henderson Consulting


“The challenge is to put value back into the supply chain,” says Whittaker, who explains that, while the UK is somewhat immune from natural disasters, be they earthquakes or tornadoes, no one can escape the impact of what he calls “seismic events, like COVID or the canal blockage. Such events provide the opportunity to redefine that value for greater robustness, but it will come at a cost, and the consumer will have to bear some of that cost.”

Adventures in Disaster Mitigation

Forewarned is forearmed when it comes to natural disasters, and while pandemics and sideways container ships can’t be predicted, a living resilience plan can go far toward hastening recovery.

“Delivering affordable, reliable power is job one,” says Jeremy Sowders, Economic Development manager for Hoosier Energy, which is part of a rural electric cooperative, headquartered in Bloomington, IN. “However, we’re also in the disaster recovery business. It’s critical for Hoosier to remain online to help clients--residential and commercial alike—when they are not.”

2102

Jeremy Sowders

The provider’s job can be made a lot easier by companies that know what they need, another result of planning ahead. “When we work with companies building new facilities, they may have specific critical functions for their business and ask for additional buildout in their electrical infrastructure.” Armed with that information, the utility can identify those critical needs upfront and plan ahead to provide redundant power, “including dual service through multiple substations and switches.”

Manning hesitates to suggest what any other company might need to create a thorough disaster plan, since every firm is different “in scale and scope. Nevertheless, preparation applies to everyone, in every situation. Think about your business, the potential risks and steps you can take to mitigate that risk.” That said, he does suggest some essentials, based on Silent-Aire’s own list of needs, essentials, he says, that were taught by the Edmonton disaster, as well as by Silent-Aire’s internal team, its customers and suppliers.

“We looked at risk mitigation from three perspectives,” he explains. “First is our people.” Here he lists emergency communications and methods for ensuring all staff members are accounted for.

“Next is our physical plants.” This is a sort of triage to gauge the operational status of each facility and its ability to get back online. Here it should be noted that all Silent-Aire operations are stocked with the same equipment globally, so any facility can back-up another in an emergency.

“Finally, it’s about processes.” This is a longer-range consideration that (once again) falls under the heading of lessons learned. “Are there changes we can make that will mitigate any risk to the first two?”


“Providing contacts from the top of the chain of command down helps ensure we can connect when time is ticking.”

-- Jeremy Sowders, Hoosier Energy


Sowders agrees, especially as it pertains to communications. No matter what authorities you’re dealing with, police, utilities, firefighters, always make sure before disaster strikes, “that they have the most up-to-date information on who to contact and how best to reach us. Providing contacts from the top of the chain of command down helps ensure we can connect when time is ticking.” Regularly updating contact lists is important as well. There should never be on your list names of people who left the firm two years ago.

He adds that Hoosier, like most utility operations these days, has a “sophisticated system that can detect outages. But a quick phone call can confirm what we already know. Seconds can save hours of downtime for our customers.”

Ultimately, it falls to the individual company, it’s executive team and staffers down the line, to build a thorough, robust and resilient disaster planning and recovery program--and to keep it updated with new lessons learned. As Brett Manning points out, there’s no set formula, given the varying and diverse needs of the company.

But there is one universal truth that should drive every initiative in disaster planning, as Matt Whittaker states: “Hope is not a strategy.”