Plenty of things have “broken” the Internet before. The release of the iPhone 6. The ice bucket challenge. Kim Kardashian’s posterior on a magazine cover. The potato salad Kickstarter. But on Tuesday, things were taken to a whole new level.
An error at cloud-computing company Fastly took down many of the web’s biggest sites, including Amazon, Reddit, Spotify, Twitch, the BBC, CNN, and the New York Times. And it all appears to be due to the actions of a single person.
While many speculated the outage was caused by a hacker, Fastly says the troubles resulted because of a service configuration change made by an unnamed customer. That triggered a deeply buried bug that was hidden in a software update Fastly deployed on May 6. And that bug resulted in the meltdown that caused 85% of the company’s network to return errors.
Unsurprisingly, Fastly is not identifying the individual who made the service configuration change.
The company says it has rolled out a fix to prevent the situation from occurring again and will be doing an internal audit to figure out why it didn’t find the bug in its own testing. A complete postmortem of the incident will also take place.
“Even though there were specific conditions that triggered this outage, we should have anticipated it,” said Nick Rockwell, senior vice president of engineering and infrastructure at Fastly, in a blog post. “We provide mission critical services, and we treat any action that can cause service issues with the utmost sensitivity and priority. We apologize to our customers and those who rely on them for the outage and sincerely thank the community for its support.”
Fastly says it detected the disruption within one minute of it occurring and had 95% of its network back up and running within 49 minutes. The bug fix was deployed less than eight hours later.