In the middle of a disaster is the worst possible time to discover that the backups won't load, the cloud service provider can't be reached, the firewall has a leak, and the one guy with all the key passwords is at a no-phones-allowed yoga retreat.
According to A.N. Ananth, CEO of cybersecurity firm EventTracker, there are about 1.4 million fires a year in the US -- but the number of cyber incidents is 30 times higher.
Too often, cybersecurity preparedness is a checklist. Are there backups? Check. Is there a firewall? Check. Is there off-site cloud failover? Check. Is there a disaster preparedness plan? Check.
But until those systems are tested under real-world conditions -- or as close to them as possible -- you won't know that everything works the way it’s intended to.
"The number-one question asked by regulators after a data breach is whether the target company has an established breach response plan, and if so, whether the plan was ever practiced in advance of the breach," said Ananth.
And the technology is only half the battle. In a disaster, people panic. Lines of communication break down. People forget what they're supposed to do, make mistakes because they're in a rush, or just hide and hope that the problem goes away. Disaster preparedness plans go out of date quickly as people move around and as infrastructure is reconfigured.
"We humans are bad at assessing risks," said Danny Grander, co-founder at Snyk, a London-based cybersecurity company. "Often, we are of the mindset that if nothing bad has happened yet, it will continue to be that way in the future. For instance, if a breach has not occurred in two years, one might think that will always be the case."
As a result, many companies don't do disaster drills, do them infrequently, or do them only for a small subset of the disasters they are likely to face, because the drills are disruptive. The drills take time, and an effective drill needs to involve all the stakeholders -- not just data center staff but also key customer, partner, and supplier personnel.
And an effective drill may uncover problems that are difficult or costly to solve.
However, knowing where the problems are is the first step to prioritizing them and either fixing them or developing work-around strategies.
Preparing Customers for Disasters
If a data center has many customers -- whether internal or external -- it may be hard to get everyone to participate in a drill.
At a minimum, however, there should be a plan in place that customers are aware of and helped contribute to.
"When I've worked with data centers being brought up, I'll recommend that they put together a 'disaster recovery in a box,'" said Neil Weitzel, director of security research at Cygilant, a Boston-based cybersecurity vendor.
New customers should then either fill in all the details -- their response plans, their key contact people.
"I recommend either once or twice a year to run through all of that and ask if it's all still in place," he said.
But then, the disaster plan should be tested, he added.
"Actually do that drill," he said. "Actually take down the server during non-peak business hours and let them know it's a drill and play through that scenario as it would actually happen in a real situation."
Consider the Worst-Case Scenario When Choosing Vendors
An eye towards disaster recovery should also be a consideration when selecting key providers.
The initial contract negotiation is a good time to bring up this issue, as is the contract renewal process. In addition, if there's a high-profile disaster in the news, a vendor might be more motivated to step up and help their customers prepare for emergencies.
Vendors, particularly those that provide critical services during a disaster, need to be willing to either participate in drills or to provide the tools necessary for customers to do their own testing.
Akamai Technologies, for example, provides content delivery services and DDoS attack protection and regularly runs disaster response drills with its customers.
For Alejandro Ziegenhirt, Akamai's practice manager for global security operations, running the drills is part of his job.
"The attack is usually performed by a third-party provider chosen by the customer," he said. "A common test plan will be run through multiple vectors of DDoS attacks, increasing in complexity and scope."
But a useful drill doesn't necessarily have to be a full-scale attack, he added.
"In addition to our attack simulation test, and equally as important, we also recommend regular 'runbook' tabletop drills," he said.
These exercises are designed to make sure that everyone knows what they're supposed to do and that the response plan isn't missing any key elements, such as current contact information for the relevant personnel.
A table-top drill by itself, however, isn't enough, said Darrell Switzer, managing director for incident response and cyber resilience services at Kudelski Security.
Backups are the most common issue, he said. The customer says, "Yes, we have backups." But then in the actual disaster it then turns out that the backup process hasn't been running, or the backup isn't available for some reason, or ramsomware got into the backups and erased everything.
"I've been to more than one customer who had ransomware and lost everything," he said. "In fact, we just saw this happen two weeks ago. During an incident, this is the worst time to find out that the backups aren't working. And this is when most companies find out that their backups aren't working."
This is a particular problem with cloud backups, since the major cloud vendors typically won't participate in disaster preparedness drills unless the customer in the Fortune 50, he said.
"You just have to make sure that you know what they do and don't provide in an incident, and what support you can expect, and make sure that it's part of your drill," he said.
For example, most companies that use cloud service providers don't take advantage of all the forensic tools available, he said. A disaster recovery drill could reveal that important logging functions weren't activated, for example.
In addition to testing backups and resilience to DDoS attacks, data centers also should be prepared to respond to data breaches and malware attacks, both of which have become distressingly common, as well as to physical problems such as server, router or cooling system failures and power outages.
"By casting a wide net of scenarios, it becomes easier to spot potential blind spots in response plans," said Javvad Malik, security advocate at AlienVault, a San Mateo, California-based cybersecurity vendor.
Finally, the disaster response drills shouldn't be limited to just technical personnel.
If a cybersecurity disaster has the potential to affect a company's reputation and cost it customers, then senior business executives may need to participate.
"Business execs are often under the spotlight during times of an attack," said Malik. "Therefore, they should understand what the drill looks like, what steps are being taken, and how, so that they are prepared."
If there's a likelihood of bad publicity or compliance issues, then legal, compliance, and public relations personnel may need to be involved.
Responders should also be prepared for the possibility of a disaster that's beyond their capabilities to handle, said Wayne Lee, senior cybersecurity architect at West Monroe Partners, a Chicago-based business consulting firm.
"Security professionals should always have contact information ready for law enforcement, outside counsel, cybersecurity insurance, and specialized forensic firms," said Lee.
