How to Make a Disaster Worse

Dan Fisher

With over 35 years in the financial industry, Dan M. Fisher has proven himself as a leader in the financial industry holding roles as the former director of the Federal Reserve Bank of Minneapolis and former Chairman of the ABA Payment Committee.

The recent snow and ice storm that Texas experienced brought to light that some core processing vendors have not learned from previous experience. Which has prompted many financial institutions to question not only their own disaster recovery plans, but their vendors’. 

“Those who cannot remember the past are condemned to repeat it.”– George Santayana, The Life of Reason, 1905.

Prophetic but true and Texas is a great example, so let’s be specific here.

If your primary business location, data-center and or network hub, has a disaster recovery plan that requires fuel to be transported to your location to keep the generator running you have the wrong plan!

This is not new. In many cases this is at the center of every major service interruption in recent times including Hurricane Sandy, Katrina, Harvey, and major adverse weather events such as last week. From Fires, Hurricanes, and Weather, losing power pretty much shuts you down.

Back to the rule. Even the state that is the largest producer in the U.S. for oil and gas can be shut down due to snow and ice. Go Figure…. So if you have a back-up generator that is reliant on the delivery of diesel fuel to stay on line here is what you need to do.

1. Have on site storage for back-up diesel fuel for three times the longest shut down. If the most recent period was 7 days, your back-up storage on site should be three weeks.
2. If you are unable to store the diesel fuel on site, then tie into a diesel pipeline.
3. If a diesel pipeline is not available, then convert your power generation to natural gas generator and tie into an in-ground supply.

Major Vendors (A Tough Discussion)

It is time you start having a conversation with your core application vendor about a number of things. Dig deep on the Disaster Recovery Business Continuity topic. Some of the questions you might ask are:

1. Why did their data center go offline?
2. Did they switch to the back-up site?
3. Why did it take so long to restore?
4. Why was the communication plan so un-informing?
5. Regardless of a state-wide issue, they should have been more prepared.
6. What type of back-up power generation system do they have?
7. What type of fuel storage do they have?
8. How many separate and independent internet access sources do they have?
9. Are they electrified by two separate and isolated power grids?
10. How is the telecommunications infrastructure designed?
11. Do they have satellite capability (for non-electrical storms)?
12. Do they have a hot-site online at a different location that can step in LIVE?

Here are some of the responses that some vendors have been providing.

1. No one can plan for an Act of God!
2. This was a widespread disaster and too big to control.
3. Everybody went down!
4. This was a unique and highly unusual occurrence.
5. We thought we were good and then this happened. We need to adjust our plan.

Note to self; have you noticed an increase is catastrophic weather over the last ten years? There is a trend here. Don’t just wait for things to happen, do something about it.

Finally, from our point of view, if your vendor responds to your questions with one of the preceding… I would start looking for a new vendor…

Short Sighted Strategies.

• Oh, the vendor will figure it out. That’s why we pay them, right?
• This will never happen again, it was a fluke?
• I hope I am retired before the next time.
• We can’t plan for everything!

You have a regulatory right to ask these questions and you should expect a transparent answer from your vendor. Be sure to document the questions and their responses. If they are reluctant to provide this information, then you have your answer.

The key is to plan for the unexpected and be prepared for it. The more financial institutions rely on and provide solutions to customers using the internet, the bigger this issue becomes.

Vendors are quick enforce their contract, justify their tactics and explain their pricing, you the Financial Institution client should enforce the vendor’s responsibility.

Finally, if the vendor stonewalls you, just refer them to the FFIEC and ask them to take a harder look at the BCP/DRP plans and test results. If we as an industry do not study this event, identify and mitigate the gaps quickly, we will see most certainly the Texas situation repeat itself if not in Texas, it will be elsewhere!

Dan Fisher

With over 35 years in the financial industry, Dan M. Fisher has proven himself as a leader in the financial industry holding roles as the former director of the Federal Reserve Bank of Minneapolis and former Chairman of the ABA Payment Committee.