Following last week's cyber-attacks on Meetup's infrastructure, Antone Gonsalves, a reporter from CSO Online, asked me, for an article he was writing, what steps I would recommend CISOs to take "if they came under a similar attack". I hesitated before giving a softball answer, "don't wait for the attack, prepare for it." I could sense the reporter's frustration immediately. He wanted to give his readers step-by-step instructions, to what I have observed to potentially be a very complex problem.
That night, I happened to be reading Atul Gawande's fascinating book "The Checklist Manifesto". In it, Gawande describes how medicine has evolved to a level of complexity that today we have hyper-specialization of doctors. The joke is that Gawande had to check before suggesting there might one day be separate surgeons for left and right ears. The premise of the book is that when a profession becomes so complex that not a single person can reasonably understand everything, methods and procedures need to be developed to safely execute that profession.
To illustrate his point he draws on the experience of the aviation industry, which as far back as WWII developed step-by-step instructions for pilots when operating aircraft. As is the case with most aviation safety innovations this happened after an accident. (Perhaps we should consider last week's DDoS a cyber-accident?)
Pilots call these step-by-step instructions "checklists", and they ensure important steps are not forgotten during the operation of these highly complex machines. Checklists are no substitute for pilots completely understanding the operation of their machines, but so ingrained is their use in commercial aviation that Jeff Skiles, first officer of US Airways Flight 1549, was following them as he attempted to restart the engines of his and Captain Sullenberger's crippled Airbus A320. Once those steps failed he moved on to the ditching checklist (yes, there's one on every airliner) and was nearly able to complete all the steps before the aircraft impacted the Hudson River.
My reporter was asking for an emergency checklist for DDoS attacks -- the cyber equivalent of the engine restart procedure. Today's eCommerce, Online Banking, B2B Portals and Media Experiences are quickly approaching the technical complexity of the aviation and medical industries. How else could companies exist who specialize in just DDoS mitigation? (Akamai recently acquired the leader in this industry - Prolexic - and is rapidly integrating the technologies into its Web Experience portfolio.) Back to the DDoS checklist. Why could I not rattle one off?
The truth is that checklists don't exist in a vacuum; checklists are not the only safety features on a modern airliner. The ditching checklist would have been pointless unless Airbus had contemplated the possibility of a ditching and built their aircraft to survive them. This fundamentally was my objection to the reporter's question: If you don't architect your cyber infrastructure with DDoS mitigation and prevention mechanisms, a checklist will not allow you to survive a ditching.
Akamai's architects have helped our customers build Web Experiences that have survived massive DDoS attacks. In one example, we had prepared an important Website for a high-profile site launch, and six months later the site experienced a surprising DDoS attack. The site stayed up because we had prepared for such an eventuality; we had implemented DDoS defense technologies and processes (yes, checklists) that would help us respond to the attack.
We had restarted the engines, so to speak, and recovered without getting our feet wet.
Akamai and Prolexic have pulled off such "saves" for our customers for years. We're not very public about such events, and in truth we see helping our customers as our responsibility, one that comes with delivering, accelerating and securing many of the world's most visited Websites.
Clearly we can always improve, so here's my "What to do when an attack hits" checklist (applicable to Akamai Kona Site Defender and WAF customers):
- Check Security Monitor: are the automated defenses effective?
- Check on-premise router and firewall monitors: for abnormal readings.
- Check Web, App, DB, etc. servers: for abnormal status.
- Consult Security Response Plan: for which team members to notify. (It should specify who does what and when.)
- Open a ticket with Akamai's Customer Care team: we may be able to correlate the attack with assaults on other targets, implement known mitigation measures, or help diagnose the attack.
We might consider taking one final lesson from aviation, as described in "The Checklist Manifesto"; creating the perfect DDoS checklist is a journey, not a one-time event. At Boeing, they have whole teams drafting, testing, and reviewing checklists. Should a specific DDoS reveal a gap in the defenses, the checklists may need updating.
Perhaps the biggest benefit of attempting to draft your first DDoS checklist is that it will help you identify what defenses you may be lacking. Match this with a business impact assessment of how much your business would suffer from, perhaps a day-long outage, and you've suddenly built a defendable business case. Does that sound like a checklist for creating a DDoS checklist?
Humans are often reluctant to expend effort preparing, particularly when the perceived likelihood of such preparation being needed is low. But we are also extremely bad at assessing risk. I would argue that the effort to better understand the exposure your organization has to DDoS and other Internet-based security threats is well worth undertaking - just as running through checklists is worth the time for pilots and doctors to improve safety.
In the end, the first step to defending against threats is to think about the threats, the impacts they will have if they happen, and whether the loss related to that impact warrants implementing mitigation measures. You could lose revenue in some cases, or suffer brand and reputation loss in others. Think about what is at stake, plan for acceptable risk and expect the attacks to come. Be prepared. Grab your checklist.