Amazon Web Services’ third interruption in a month reveals a weak point in the Internet’s backbone

Amazon’s massive cloud operation on Wednesday suffered its third outage in a month, briefly shutting down a large number of online services that are critical of everyday life and again highlighting the vulnerabilities of an increasingly interconnected network.

Amazon Web Services reported on its status page that a power outage at a data center in Northern Virginia triggered connectivity issues that started around 7:30 a.m., disrupting a wide range of online giants, from chat rooms to Slack to the Epic Games game store. The network connection had normalized around 1[ads1]0 o’clock, the company said.

It is the latest of several recent AWS outages that have taken down large parts of the digital economy. Two weeks ago, service issues related to malfunctioning network devices hit offline Amazon ringtones and Roomba vacuum cleaners. Another power outage occurred last week.

Cloud systems like AWS allow businesses to rent servers and computing power over the web, and they have revolutionized the internet with the promise of a reliable online backbone, available at any time.

But the outbreaks have highlighted how this consolidation of the Internet’s once distributed capabilities also means that a single error can lead to extensive ripple effects, weakening the hidden backbone behind large parts of the network.

“A single failure in a high-profile vendor will have huge implications for countless organizations of all sizes, in often very unexpected ways,” said Ed Skoudis, president of the SANS Technology Institute. “Service outages are huge and affect thousands of companies and millions of users. We lay more eggs in fewer and fewer baskets. More eggs are crushed that way. “

Amazon did not immediately respond to a request for comment. Amazon founder Jeff Bezos owns The Washington Post.

It’s difficult to keep a giant “cloud” of international data centers online, said Steven Bellovin, a professor of computer science at Columbia University. Each change must be tested before it is distributed and closely monitored afterwards, with an automatic way to go back in case of problems and a security network of redundant software and backup servers, for security reasons.

Amazon has not released technical details about the underlying faults, and occasional power outages are expected. But so many errors in a short time indicate that some of the backup systems may be insufficient for the task, Bellovin said.

“The short answer is that I’m disturbed,” he added. “I’ve been a fan of cloud services for a long time … and it’s possible that this is just an evil coincidence for Amazon … but if they can not accept growth, they’re in a bad place.”

[What caused Amazon’s cloud network outage earlier this month? Will there be more?]

AWS is the world’s largest provider of cloud services, with 40 percent of the worldwide infrastructure last year for infrastructure cloud services, according to market research firm Gartner. Microsoft was a distant number two, with about 20 percent.

But moving among the largest cloud data services – Amazon’s AWS, Microsoft’s Azure and Google Cloud – is a challenge, because each system works differently and depends on its own infrastructure.

Several companies, Skoudis said, are starting to talk about using multiple cloud systems at the same time, although the approach is costly and “a bit ridiculous, given how the cloud was advertised to give us reliability and affordability.”

The causes of the three power outages this month reveal how the cloud’s increasing complexity and demands have led to more potential for disaster. The five-hour outbreak on December 7, AWS engineers wrote in a postmortem, was caused by a bug in some automated software that led to “unexpected behavior” that then “overwhelmed” AWS network devices and hit computer systems on the East Coast.

The second power outage, which lasted less than an hour on December 15, mainly affected units from the west coast and was accused of “network downtime” due to some internal construction that “incorrectly moved more traffic than expected to parts of the AWS backbone affecting the connection” “, according to a statement from the company.

During Wednesday’s power outage, which Amazon said was due to power outages in the data center, users on Downdetector, a website for measuring Internet outages, said they had trouble accessing sites including the video streaming service Hulu and investment site Fidelity.

Last year, large parts of the network were turned offline after Amazon’s servers in Northern Virginia were overwhelmed. And Skoudis suspects that more problems will arise as the internet becomes more complex.

“In the IT field, we sometimes joke about how we spend 15 years centralizing computing, followed by 15 years of decentralizing, followed by another 15 years of centralizing again,” he said. “Well, we’ve spent the last 10 years centralizing again, this time on (the) cloud.”