What Was the Capital One Breach? The Firewall Fetched the Keys Itself
The request went to 169.254.169.254. Every machine on Amazon's cloud knows that address. It is where an instance goes to ask about itself: what am I, where am I, and, if a role has been attached to me, what are my credentials right now.
The firewall asked. The firewall was told. And the firewall passed the answer back out to whoever had asked it to ask.
Everyone remembers Capital One as a hack. It is better understood as a misunderstanding about who was talking.
What was the Capital One breach?
On 22 and 23 March 2019, an outside party obtained credit card application data belonging to approximately 100 million people in the United States and 6 million in Canada, covering applications made between 2005 and early 2019: names, addresses, dates of birth, self-reported income, credit scores, credit limits, balances, and fragments of transaction history from 23 days across 2016 to 2018.
Most of it was not the kind of data that changes a life. Some of it was:
- about 140,000 US Social Security numbers,
- about 80,000 linked bank account numbers, belonging to secured-card customers,
- about 1 million Canadian Social Insurance Numbers.
No card numbers and no login credentials. Capital One was careful to say so, and it was true, and it was also the least important thing about the incident.
How it worked
Capital One ran a web application firewall in front of its cloud infrastructure, built on ModSecurity. The firewall's job was to inspect incoming requests and reject the malicious ones. It was running on an EC2 instance, and that instance had an IAM role attached, which is the normal way a cloud workload gets permission to do things without anyone typing a password.
The firewall was misconfigured in a way that let a request from outside instruct it to make a request of its own, to any address, and relay the response. That is server-side request forgery, and the address that matters on AWS is the metadata service at 169.254.169.254, because among the things it will tell an instance about itself are the temporary credentials for the instance's role.
So: ask the firewall to ask the metadata service for its credentials. Receive them. Use them, from anywhere, as the firewall.
The three commands
The FBI's criminal complaint describes what was done with the credentials, and it is short enough to quote in outline. The first command asked the metadata service for the role's security credentials; the complaint refers to the role only as *****-WAF-Role. The second was a list: with the role's credentials, enumerate every storage bucket the role could see. There were roughly 700. The third was a sync: copy the contents of the interesting ones out. Around thirty were copied.
Three commands, one of which is ls. Every credential in the chain was legitimate. Every permission was granted on purpose. Nothing was cracked, forged or guessed. The perimeter held exactly as designed against requests from outside. It had not been designed against a request that originated from inside it, from the component whose whole function was to be trusted.
How it was found
Not by Capital One. On 17 July 2019, nearly four months later, someone emailed the company's responsible disclosure address to say that data which appeared to belong to Capital One was sitting in a public GitHub post. The post was under an account belonging to a person who had also been discussing the intrusion in a Slack channel and on Twitter, under a handle she had used for years, and who had listed the other companies she had done the same thing to.
Capital One confirmed the intrusion on 19 July and announced it on 29 July. The FBI arrested Paige Thompson, a former Amazon Web Services engineer living in Seattle, the same day. Her posts had referenced not only Capital One but around thirty other organisations whose cloud environments she had entered by the same method, none of which had noticed either.
The interval matters. From 23 March to 17 July nothing inside Capital One noticed that its firewall's credentials had been used from outside the company to list seven hundred buckets and copy a hundred million records. The detection was a stranger reading GitHub.
What it cost
- $80 million — civil penalty from the Office of the Comptroller of the Currency, 6 August 2020, for failing to establish effective risk assessment processes before migrating a major portion of its operations to the cloud, and failing to correct the deficiencies in a timely manner. The Federal Reserve issued a separate cease-and-desist the same day.
- $190 million — settlement of the consumer class action, given final approval in September 2022.
- Capital One's own estimate of the direct incident cost, in its earnings release, was $100 to $150 million in 2019 alone.
Thompson was convicted in June 2022 of wire fraud and computer intrusion, and acquitted of identity theft and access device fraud. In October she was sentenced to time served and five years of probation, with the judge noting she had not sold the data, which was true, and which had not been the point for anyone whose Social Security number she had. The government appealed the sentence as too lenient; the appeals court agreed, and the district court then reimposed it.
What changed at AWS
In November 2019, four months after the announcement, AWS shipped a second version of the metadata service. IMDSv2 requires a caller to first obtain a session token with a PUT request, and it sets the time-to-live on the response to one network hop, so that a request relayed through a proxy or a firewall arrives without a token and is refused. The blog post announcing it named the problem it was for in its title: open firewalls, reverse proxies and SSRF.
Nothing about the underlying design was wrong. A machine has to be able to ask about itself. The change was to make the machine prove it was the one asking.
The part the write-ups skip
The lesson usually drawn is about configuration, and it is a fair one. The firewall's role should not have been able to list every bucket in the account. Instance metadata should not have been reachable through a relay. Both are true.
But notice what the vulnerability was in. Not the application. Not the database. The firewall, the one piece of the stack that exists specifically to stop this kind of thing, with permissions granted precisely because it was the trusted piece. It was not bypassed. It was used. The request that emptied the buckets was made by the security control, in its own name, with its own keys, and every log that recorded it recorded a firewall doing something firewalls do.
The firewall did not fail to fetch the keys. It fetched them itself.
Further reading
- Capital One — 2019 Cyber Incident: Information on the Capital One Cyber Incident, capitalone.com/digital/facts2019.
- United States v. Paige A. Thompson, criminal complaint, W.D. Washington, 29 July 2019.
- Office of the Comptroller of the Currency — consent order and civil money penalty against Capital One, 6 August 2020.
- Neto, Madnick, de Paula, Borges — A Case Study of the Capital One Data Breach, MIT Sloan working paper, 2020.
- AWS — Add defense in depth against open firewalls, reverse proxies, and SSRF vulnerabilities with enhancements to the EC2 Instance Metadata Service, November 2019.
EXHIBIT 0029 — CAPITAL ONE
We put it on a shirt. An ornate brass bank teller's cage, drawn as an engraving, bars and arch and marble counter, with a ring of keys on the counter pushed halfway out under the grille toward you. On the grille, one plate: 169.254.169.254.
Callouts: THE FIREWALL MADE THE REQUEST. TEMPORARY CREDENTIALS, REAL ACCESS. FOUND ON GITHUB BY A STRANGER. REPORTED BY EMAIL, 17 JULY.
The firewall fetched the keys itself.
That's the whole idea.