Akamai Diversity

The Akamai Blog

2003 Blackout: An Early Lesson in Planetary Scale?

On the drive to work this morning, I listened to a report about this being the 10th anniversary of the massive blackout that plunged an area from New York City to Toronto into darkness. I immediately thought of a post Akamai CSO Andy Ellis wrote recently called "Environmental Controls at Planetary Scale."

It might be overreaching to say the 2003 blackout was an early case study in the success and failures of controls at Planetary Scale. Andy was talking about the environmental controls in data centers around the world. The blackout wasn't something individual data centers had much control over, and the power failure was geographically limited to a section of the U.S. and Canada. The blackout's root cause was a software glitch in an alarm system inside one of FirstEnergy Corp.'s control rooms in Ohio. Workers apparently didn't realize they needed to redistribute power after overburdened transmission lines collapsed onto overgrown trees. A manageable local blackout thus snowballed into widespread electric grid failure.

Still, I can't help but think of the parallels. Andy's blog post examined the pros and cons of investing large sums of money in data center environmental controls. He wrote: 

Is the cost worth the hassle? If you run one data center, then the costs might worthwhile - after all, it's only a few capital systems, and a few basis point improvements in MTBCF will likely be worth that hassle (both in operational false positives as well as deployment cost). But what if you operate in thousands of data centers, most of them someone else's?  The cost multiplies significantly, but the marginal benefit significantly decreases - as any given data center improvement only affects such a small portion of your systems.  Each data center in a planetary scale environment is now as critical to availability as a power strip is to a single data center location.  Mustering an argument to monitor every power strip would be challenging; a better approach is to have a drawer full of power strips, and replace ones that fail.

I see lessons here in how we manage interconnected electrical systems where a failure in one place can spill over to many other places the world over. Security experts have said and written much in recent years about the threat to global power grids. Among other things, they've warned, a hacker could compromise SCADA controls in one power station and maximize the damage if the target is the weak link in a much bigger chain of power distribution centers.

The ways in which we manage the threat carry similar pros and cons to that of the environmental control management Andy wrote about.

On this particular anniversary, I throw it out there as food for thought.


1 Comment

Well, you almost connected the dots. The 2003 Blackout was NOT caused "after overburdened transmission lines collapsed onto overgrown trees." That is incorrect. It was a SCADA hack. I personally spoke with the Defense Intelligence Agency (DIA) task force leader in charge of finding out why that happened. The truth is not the official story, but you decide what to believe.

Before I explain the truth, ask yourself, if this was a simple power screwup, why did the DIA have a task force on it? The actual truth was that the software controlling Power Plants around the world was hacked. It is SCADA as you know. wiki/SCADA

Two weeks after the US power goes down, England has blackout

Soon after England, Italy went down.

All these blackouts were simply Hackers controlling the SCADA software. It would seem strange that a downed power-line happened each time to take down power from 10s of millions of people so close together in time don't you think?

The truth was that Special Forces troops had found SCADA hacking coding on Taliban Laptops in Afghanistan that spring. I told the DIA task-force leader to look into SCADA when he told me that the only thing they knew was it was not an explosion. There was NEVER any talk about a screwup by anyone in Ohio. That was the Government story to the public. But did anyone ever see a picture of that fallen power-line. You would think that would be HUGE news to witness how one fallen tree could blackout half the country. GRIN and people bought that story. Why wouldn't they?

Many years later others now see the problem…

There is even a website teaching who to HACK the SCADA software.

Syria has tried to hack the Israeli SCADA

Top level CyberSecurity personnel know the truth..greentechmedia.com/articles/read/Maltese-Cybersecurity-Group-Claims-SCADA-System-Hacks

And now so do you. Don't believe the government's Official story if it does not make sense.