On Monday, a former employee of Amazon was arrested and charged with stealing more than 100 million consumer applications for credit from Capital One. Since then, many have speculated that the breach may have been the result of a previously unknown "zero-day" error, or an "inside" attack in which defendants took advantage of access obtained from their former employer. But new information indicates that the methods she used have been well understood for many years.
The following is based on interviews with nearly a dozen security experts, including one who is interested in details of the ongoing investigation. Because this event deals with somewhat jarring and esoteric concepts, much of what is described below has been dramatically simplified. Anyone seeking a more technical explanation of the basic terms referenced here should explore some of the many links included in this story.
According to a source with direct knowledge of the breach investigation, the problem stemmed in part from a misconfigured open source Web Application Firewall (WAF) that Capital One used as part of its operations hosted in the cloud with Amazon Web Services (AWS).
known as "ModSecurity", this WAF is distributed along with open source Apache Web server to provide protection against multiple classes of vulnerabilities that attackers most often use to compromise the security of web-based applications.
The misconfiguration of WAF allowed invaders to trick the firewall into forwarding requests to a key back-end resource on the AWS platform. This resource, known as the "metadata" service, is responsible for distributing temporary information to a cloud server, including current credentials sent from a security service to access all of the cloud resources that the server has access to.
In AWS, exactly what this information can be used to hinge on the permissions assigned to the resource requesting them. In Capital Wed's case, for some reason, the misconfigured WAF was granted too many permissions, ie it was allowed to list all files in any data bucket and read the contents of each of these files.
The type of vulnerability exploited by the intruder in the Capital One hack is a known method called "Server Side Request Forgery" (SSRF) attack, where a server (in this case, CapOnes WAF) can be tricked into executing commands such as the shall have never been allowed to run, including those who allow it to talk to the metadata service.
Evan Johnson, leader of the product security group in Cloudflare has recently written an easily digestible column on the Capital One hack and the challenges of detecting and blocking SSRF attacks targeting cloud services. Johnson said it is worth noting that SSRF attacks are not among the dozens of attack methods that detection rules are sent by default in WAF utilized as part of the Capital One intrusion.
"SSRF has become the most serious vulnerability to organizations using public clouds," Johnson wrote. The effect of SSRF is exacerbated by the provision of public clouds, and the major players like AWS do nothing to fix it. The problem is common and well-known, but difficult to prevent and has no mitigation built into the AWS platform. "
Johnson said AWS could address this shortcoming by including additional identifying information in any query sent to the metadata service, which Google has already made with its cloud platform. He also acknowledged that doing so can destroy a lot of backward compatibility in AWS.
"There is a lot of specialized knowledge that comes with running a service within AWS, and to someone without specialized knowledge about AWS, [SSRF attacks are] not something that would show up on all critical configuration guides," Johnson said in an interview with KrebsOnSecurity .
"You need to learn how EC2 works, understand Amazon's Identity and Access Management System (IAM) and how to authenticate with other AWS services," he continued. "Many who use AWS want to interface with dozens of AWS services and write software that orchestrates and automates new services, but in the end, people really listen to AWS galore, and with it comes a lot of specialized knowledge that's hard to learn and hard to get right . "
In a statement provided to KrebsOnSecurity, Amazon said it was inaccurate to claim that the Capital One breach was caused by AWS IAM, the instance metadata service or AWS WAF in any way. [1
Amazon pointed to several (mostly a la carte) services, offering AWS customers to help mitigate many of the threats that were key factors in this breach, including:
– Access Advisor, which helps identify and include AWS roles that may have more permissions than they need;
–GuardDuty, designed to raise alarms when someone searches for potentially vulnerable systems or move unusually large amounts of data to or from unexpected locations;
–AWS WAF, which Amazon says can detect common exploitation techniques, including g SSRF attacks;
–Amazon Macie, designed to automatically detect, classify and protect sensitive data stored in AWS.
William Bengston, former senior security engineer at Netflix wrote a series of blogger posts last year about how Netflix built its own systems to detect and prevent legitimate compromises in AWS. Interestingly, Bengston was hired about two months ago to be director of cloud security for Capital One. My guess is that Capital One would now wish they had somehow managed to lure him away earlier.
Rich Mogull is the founder and chief technology officer of DisruptOPS, a company that helps companies secure cloud infrastructure. Mogull said that a major challenge for companies moving their business from scattered, expensive physical data centers to the cloud is that often the employees responsible for managing that transition are application and software developers who may not be as permeated as they should be in Safety.  "There is a basic skills and knowledge gap that everyone in the industry is struggling to handle right now," Mogull said. “For these big companies that do, they have to learn all this new stuff while maintaining their old stuff. I can make you safer in the cloud easier than on-site in a physical data center, but it will be a transitional period as you acquire the new knowledge. "
Since news of the Capital One breach broke Monday. KrebsOnSecurity has received numerous emails and phone calls from security leaders who are desperate for more information on how to avoid falling prey to the bugs that led to this colossal breach (those requests were actually part of the driving force behind this story.]
Some of these people included executives in major competing banks that have yet to step into the cloud quite as deeply as Capital One has done. But it's probably not a lot to stretch for them all lining up in front of the diving board.
It's been interesting over the last couple of years how different cloud providers have responded to major power outages on their platforms – very often shortly after have published detailed post-mortems on the underlying causes of the power outage and what they are doing to prevent such incidents in the future Similarly, it would be wonderful if this type of public accounting were extended to other large corporations in the wake of a huge breach.
I have no great hope that we will officially get such a detail from Capital One, who refused to comment on the record and referred me to their statement about the breach and to the Justice Department's complaint against the hacker. This is likely to be expected, as the company is already facing a class action lawsuit over the breach and will likely be targeted by several lawsuits going forward.
However, as long as the public and private response to data breaches remains orchestrated primarily by lawyers (which is certainly the case now with most major corporations), everyone else will continue to lack the benefit of being able to learn from and avoid the same mistakes.
Tags: Apache, Capital One breach, CloudFlare, DisruptOPS, Evan Johnson, metadata service, ModSecurity, Rich Mogull, server side forgery, SSRF, WAF, Web Application Firewall, William Bengston
You can jump to the end and leave a comment. Pinging is not allowed at this time.