Transcript:
RSA Breach: The Untold Story, Pt. II
Welcome back to Cyberson’s Malicious Life, I’m Ran Levi.
The following is the second in a two-part story about the 2011 RSA breach. In part 1, we heard from Sam Curry & Dave Castignola, about how the breach was discovered by an attentive employee, who noticed sensitive data exiting from RSA servers. Sensitive Information regarding SecureID - the company’s flagship product - was stolen. It was yet unclear whether the attackers had managed to get hold of the cryptographic key that they needed in order to decrypt the stolen data. If they haven’t - RSA was in the clear. But if they did have the key, then hundreds of RSA’s clients were at serious risk. It was, in a sense, Schrodinger’s Hack...
RSA’s management then had a tough choice to make. Should they disclose what happened and suffer the consequences - even though it was possible that no real damage was done - or should they maybe keep quiet and hope the whole mess just...goes away? Art Coviello, RSA’s CEO, made the call: “We are going to do the right thing,” he said. And they did.
CONVENTIONAL NARRATIVE
History has a way of repeating itself.
Last December, the New York Times reported a breach at FireEye. It was shocking, for obvious reasons: few entities in the world are as secure as a top cybersecurity company like them. It became an even bigger mess when we realized that the FireEye breach was just one, small part of the much bigger SolarWinds supply chain attack.
In the months since, experts have been trying to make sense of such a momentous event. A nation-state supply chain attack that even cybersecurity companies couldn’t stop, because they themselves were victims. It’s unprecedented! Never-before-seen.
Sam Curry is the CSO at Cybereason and Visiting Fellow at the National Security Institute:
[Sam] I’d say the narrative is security company gets hacked by a nation state and it leads havoc in the world and probably one of the first times when I think a very public security company, one that was tied into a lot of things, was seemed to potentially be vulnerable as well.
What Sam’s just said there about FireE-...
Agh, sorry listeners, slip of the tongue. I must be getting mixed up, because the breach Sam’s talking about happened ten years ago.
[Sam] I think purely external perception was, “Uh, how could this have happened?”
In 2011, Sam was the CTO at RSA Security. Like FireEye, RSA was a top cybersecurity company. Like SolarWinds, their software was basically everywhere, so they invested heavily in making sure it was secure.
[Dave] A week before the breach, I had a very, very well-known financial services customer in for a briefing and they pointed to our security operation center, literally pointed to it and said, “I want that.”
But even the best security can usually be breached somehow.
[Dave] A week later they said, “How could that happen to you?” And they were shaken to their core.
[Sam] I said it publicly at the time, we were probably a nine, eight, nine, maybe even a ten out of ten in terms of security capability but even we got hacked. And so, I think that also made people afraid that if it could happen to us, it could happen to anyone.
It may seem like the big data breaches of today are unbelievable--that of course we didn’t see them coming, because they’re so complex and novel. But when RSA was breached a decade ago, only the fine details were different--their attackers used many of the same tactics that still work today.
BREACH EXPLAINED
[Sam] what they did was they effectively went after our HR partners, right? So they hacked upstream in our supply chain.
Hacking RSA head-on would’ve been very difficult. Hacking a company close to them was not.
[Sam] they had much weaker security and they had R&D capable of giving them enough to make it through that first crack in the armor.
To breach the human resources company, the attackers used the oldest, best trick in the book.
[Sam] the initial access to the environment was from an email from a partner who was exchanging an Excel spreadsheet for recruiting back and forth.
The email looked like it was coming from a partner but, of course, it was not.
[Sam] And they had essentially embedded a, we call it a backdoor, but effectively, they silk-threaded and forked another process when that Excel was opened. There was no, “Do you want to run this file?”
No warnings, no nothing. The malware triggered as soon as the file was opened.
[Sam] He double clicked it and they were in the system.
The attackers had full access over the victim’s computer.
[Sam] It was a unique build of a Poison Ivy, which is remote access Trojan, that is still being used by the way to my shock and horror because I thought it was even fairly old technology at the time but it was unique build.
And then they proceeded. They did the daisy chain, right? It escalated privilege. It moved through the organization.
They hopped from the HR partner, into RSA itself. Even then, RSA had measures in place to prevent sensitive data from leaking. But the hackers got around it.
[Dave] It was also a corruption of the tool on Exfil. It wasn’t supposed to allow out encrypted data and it was.
So those are the basics. But the deeper you get into the RSA hack, the weirder it gets, and the more it changes. In the weeks and months and years that followed March 17th, 2011, news outlets around the U.S. and worldwide told the story of RSA’s breach, while failing to understand or outright misreporting so many of the important details.
ATTRIBUTION
[Sam] I’ll tell you two things the world probably doesn’t know unless they sat in one of my Anatomy of an Attack presentations because we couldn’t say it publicly.
One of the things everybody missed about RSA’s breach had to do with who actually did it.
Shortly after discovering the breach, RSA’s analysts spotted their attackers’ footprints and realized they were still in the system. Counterintuitively, the analysts chose not to boot the intruders from the network.
[Sam] So incidents, your job is to protect the company, privacy of the customers, the shareholders, a lot of things. As long as you fulfill that mission, you then try to understand as much as possible the opponent. And that’s why you don’t kick them out necessarily right away.
They studied the perpetrators’ every move, gathering intel.
[Sam] the first clue that we had, who it was, was the time of day that they operated. They were almost exactly 12 hours out at sync with us. [. . .] they took nice, predictable lunch breaks. They came to work and they went home and they weren’t very talented. They were about 8 people at the keyboard. And we were pretty sure from the start they were in China.
They figured it all out. Or so they thought.
[Sam] We knew where they were, what they were doing. We had them boxed in. We were going to kick them out on the 15th. That was the plan when we found the second attacker.
A second attacker. A Russian nesting doll of APTs.
[Sam] They both came from the same nation state. And the second one hacked the first one.
[. . .] The second attacker, they were very talented and they had highly idiosyncratic behavior and that each behavior is different from each other and they worked very long hours. They were a much more professional and well-trained squad and they attacked the first.
Prior to the release of this Malicious Life podcast, only a small number of industry insiders have ever been privy to this side of the story.
MEDIA BLAME
[Sam] Oh yeah.
[Dave] Everybody wants to come up with a simple point the finger or blame a certain specific thing. It is extremely difficult to go back from a forensics perspective and really determine all of these things. That was a giant take-away for me.
[Sam] Yeah, and Dave, you make an excellent point. Even 10 years later, it’s hard without going to a whiteboard to explain it. But an excellent conclusion, I hear about breaches and I go, “All right. Hype cycle is all about exposing a brand and shocking all.” My own wife, I said to her the other day, I said, “Oh, there was a big breach recently.” She goes, “Oh, there’s one every day.” Right? So the media is trying to catch these huge stories. And then I wait for – let me talk to somebody who is really there because unless you’re in the inside, nothing in the media gets it right. And a lot of the books written about it have gotten it wrong, including ours.
People have gotten the story of the RSA breach wrong for years, over and over again. Because they weren’t in the trenches. Because they didn’t see what Sam and Dave saw.
BACK TO 3/17
[Nate] So it’s Thursday [. . .] and the news is let out into the world. What happens?
Sam Curry: It went crazy.
We left off our last episode at the end of the workday, Thursday, March 17th, 2011. It’d been about 30 hours since RSA realized they’d been fleeced for their flagship product--50% of their entire business--the SecurID authentication system. The one thousands of companies around the world used to secure their most sensitive systems.
They published the news of their breach, and that’s when it all really started.
ASSEMBLING THE TROOPS
[Sam] It was like going to war. [. . .] We were trying to figure out who do we call, what do we have to say, what’s the wording, how do we keep – we knew we control the IT environment because we had shut it down. We shut it all down and we started the painful process of rebuilding it. All of that was happening at once.
[. . .] When Dave came in and took control of the communications arm, the crisis response, we were moving at full tilt.
[Dave] We had to build 8 departments, each with a job description, workflows, job, task. We automated everything using one of our tools, the Archer tool, that we …
[Sam] Which wasn’t designed for that, but yeah.
[Dave] It was not designed for it. No. And I’ll never forget the moment [. . .] we had a meeting and it was literally a stand-up meeting that lasted 3 minutes and we got done and somebody looked at me and they said, “What’s wrong?” And I said, “I’m going to need a bigger boat. I need more people. I need way more people than I had even thought about all day today. This is going to explode beyond anything.”
And they ran down the hall and they said, “We are going to get volunteers. How many do you need?” And I’ll never forget coming 10 minutes later looking down the hall and seeing 60 people. It looked like they were marching down the hallway in slow motion.
[Sam] What happens is that people don’t leave when the going gets tough. You think they will.
Dave was now commanding an army. RSA employees from around the country were streaming into Bedford, Massachusetts by the dozens.
[Dave] The day of the breach announcement was that entire building and if you look at it from the parking lot looking in, every light was on, every room had meetings, and everybody was at that point just hard at work, working on their particular part and solving this problem.
[Sam] within a week out of at the time we had 2,400 employees I think, a full 1,200 stopped their day job and were doing just this.
[Dave] I remember those first few weekends as we just piled through 7-day workweeks and have a look out at the parking lot on a Sunday morning and see the entire parking lot of our campus full. And I remember somebody joking saying, “I don’t think there’s any Chevrolets left at the rental car lots at Logan Airport.
INTENSITY
Few rental cars were left at Logan airport, and few lights were off even in the overnight hours at the RSA building.
[Sam] just to paint the picture, you know those lovely conference rooms people have in buildings with the glass beside the doors and the frosted glass, we had to paper it up. We had to leave our phones outside. We suddenly couldn’t use computers for the craziest of things. We had to scan the room for bugs. We had to scan the woods. And we found things. It was espionage happening.
Actual, real life spy stuff.
[Sam] We had to start tracing where people went because we suddenly were worried that there were insiders who had been blackmailed or compromised. It felt like living in a thriller novel.
[Dave] The intensity level was off the charts. The alignment and the focus, I don’t I’ve ever seen that in my career since. Everybody was laser-focused. [. . .] I just felt like every part of my entire body was alert and entirely focused on this mission.
APOLOGY TOUR
The mission wasn’t exactly what you’d imagine it to be. Repairing systems, finding and defeating attackers--that was the easy part. The biggest, scariest threat to the company wasn’t any Chinese APT. It was angry customers.
[Nate] Who do you have to talk to and how did these conversations typically go?
[Sam] They started usually very ugly. And Dave and I, we did hundreds, right?
[Dave] There were hundreds of customers that each of us spoke with. And in some many cases, that first conversation led to three, four, five additional conversations.
It’s hard to imagine a less appealing job than being the face of a corporate apology tour.
[Dave] Sam was put out in front of this with Art Coviello and a few execs.
Sam got stuck doing video calls, and jumping state-to-state doing explainers.
[Sam] I flew to Bentonville to talk to Walmart and I addressed 600 people there and they later gave me a unit coin for the conversation. I spent 5 hours at the whiteboard. I went to Mastercard who invited me to St. Louis to address local CISOs.
[Dave] And my job was to organize, schedule, follow up, follow through, document, track, monitor, report to the board, and I remember just how intense those conversations were, Sam.
[Sam] Yeah, and I got put on the hairiest ones. [. . .] The hardest meeting I had because you asked for the rough too, I had to go to DC and address the FFIEC and the assembled banks. So it was a large room with a couple of thousand people in it and I had to stand up at a table because it was a large, very large hall. And basically talk in a microphone about what happened and take questions. It was incredibly difficult because these people were going, “You’re our trusted partner, what’s going on?”
[. . .]
And the only thing you could is just run through the facts with people and answer the questions directly. Yes, this is what happened. This is – we will do these things for you. What do you want? How can we help?
[Dave] I had the good fortune of almost all of my conversations being somewhat professional, I only had less than a handful of people that screamed at me and yelled, but I opened up every conversation with, “You’re upset, I’m upset. Believe me. This is difficult.”
It went on like this--call after call, video conferences and flights and NDAs and anxiety for weeks on end.
STARTING TO TAKE A TOLL
[Sam] We went to simply working all the time. There was nothing else. People didn’t go home and make food for their kids. They found babysitters if that had to happen. People descended to help reinforce the breach literally and figuratively. It was exhausting. It was grueling. It was – I don’t think I’ve ever done anything in such a sustained high-intensity way before or since. And I’ve been through some pretty big things. This was for me 30 days in the office without let. And you fell asleep when you have to and you got up and kept going.
[Dave] And a lot of it is actually a blur. How can you work straight through 18, 20-hour days? What were you working on? The answer is everything. [. . .] I hope to never go through anything like it again.
Dave had a room at the nearby Marriott, but was almost never there--coming back late at night, leaving early in the morning, if he came back at all. It got so bad that people started noticing.
[Dave] it was my fourth week and just living there. I would come – the hotel by the way figured out something was going on.
The hotel staff felt so bad for Dave that they started doing his laundry, without him even asking.
[Dave] it was just kind of funny knowing that they were taking care of me.
But at one point, I was told to go home for a day. They said, “You need to go home and see your family.” And it was a Saturday. And I recall flying home and I was on the Delta flight and I had my laptop up and I was on Wi-Fi and I was just watching so much email come in. I was staring at my screen watching email just come. And my screen was just real-time refreshing through with all these new messages that were piling in.
And I got – landed in Detroit, went home, had a phone call with a well-known entertainment company and that person was not in a good spot and we had a couple of phone calls that night. I went back. I was looking at my computer and it was Saturday evening and I went upstairs and my wife looked at me and said, “You’re leaving.” I said, “Yeah, I got a ticket on the 11:30 PM flight tonight. I’m flying back.” It just – it was all-encompassing. It was everything. It was our entire life.
…
SECOND ATTACK
Earlier in this story, Sam Curry said there were two things nobody in the world has known about the RSA breach prior to this Malicious Life episode. The first thing was that there were two attackers.
[Sam] The second and I don’t know how much time we have to go into this, is that nearly 7, 8 months later, in October, we went back and revisited that first question of whether we had a – was the screen green or red because there’s no such screen.
What Sam’s referring to by “green” and “red” here is the thing we never actually figured out, even this late in the story: whether the Chinese hackers had stolen SecurID’s coveted encryption key.
[Sam] I think I said we have come to the conclusion that we had had a breach and we then moved into action.
After 7 or 8 months, they had enough information to make a call. A call as to whether they really did get truly and properly breached or not.
[Sam] We came to the conclusion that we probably actually didn’t.
All those months of suffering--the hundreds of calls, the international media spotlight, the late nights sleepwalking back to their rooms at the Marriott--it turned out SecurID was secure the whole time.
[Sam] And that’s quite a shocker because 5 weeks after the first attack, we were attacked again.
A second attack, in April.
[Sam] And they didn’t succeed in that second attack because we were still shut down, not because we were better at security at that point.
It seemed obvious, to the guys at RSA, what the attackers were coming back for. The first time, if you remember, they’d only managed to exfiltrate half of the SecurID data.
[Sam] And we then came to the conclusion, “Oh, they must be coming back for the other half of that file they exfiltrated.”
But this assumption turned out to be wrong.
[Sam] Well, we found evidence in October that no, they probably were coming back for the key. And that’s a vital distinction.
[. . .]
[Sam] they came back for the key, which means the RSA breach was probably a non-event.
The fact that the attackers came back for the key meant they didn’t have it in the first place, and the SecurID data they had managed to exfiltrate was useless from the start.
[Sam] we sat around in October and said, “So do we tell anyone this?” By now, the damage is done. People have replaced whatever security controls from us they need to. They’re all – risk is under control. And we said, “No, it’s not worth a third news media hype wave that could do more damage to customers and to partners. Who needs to set the record straight if people are healing?”
In 2011, the world assumed RSA had been hacked because they had failed in their security. It turns out they were attacked twice, and managed to protect their data both times.
So yeah, they really did have eight, nine, ten out of ten security. Sadly for Sam and Dave and their colleagues, it took ten years for anybody to recognize them for it. Until today; until this podcast.
TAKEAWAYS
[Nate] If there’s one thing that you could leave people with, I mean everything that we just talked about, one take-away from the story, what would you say?
[Sam] Honestly, the one I would say is if and when this happens to you, you can’t be a victim. You cannot be a victim. Nobody ever says, “Oh, poor so and so.” It stinks to be them when that is a big company or a brand. Having data is a privilege, not a right. And so in the moment, you’re going to have to make a choice somewhere with saying, “Am I going to be a hero or a villain?” And it seems like an obvious thing. My advice is if you don’t have that Art Attack to smack you around and make you make the decision, choose to be a hero. Really.
Not only is it the right thing to do and a good look but be part of the community of folks that are trying to prevent victimization of others. Don’t hide it because of the court of public opinion will make a decision and how you behave has lasting consequences for yourself, for your shareholders, and for others.
[Dave] And I would just say that if that advice, while heated advice from Sam is followed, you will survive. You will survive even a very big public breach. You can survive.
When they first got the news of a breach, neither Sam nor Dave thought their company would live past it. But rather than cover up the bad stuff, or try saving their own personal reputations, they did their best to be responsible and help customers. They told the world what happened almost immediately, they put their faces on camera and in front of audiences, and tried to do the right thing every time, even when it hurt.
LIFE AFTER RSA
And yet, their predictions of doom didn’t come to pass.
[Sam] The company did return to normal. There was life after the breach.
When you’re in the eye of the storm, it seems like things will never get better. But, eventually, the wind and the rain stops and the clouds part.
[Dave] I recall coming home once I was released from my engagement and I was going into my day job and I was saying goodbyes, just how emotional that was and the tears that were shed. And then I remember coming home and my daughter looking at me when I dropped my luggage in the kitchen and just kind of went, “Ugh!” because we all literally, literally physically aged during the episode I mean like gray hair.
Sam stayed at RSA for another three years, Dave another six.
[Sam] And then amazingly, my career resumed. I don’t know what it would have been otherwise. I thought, hey, my prospects have vastly diminished but my career resumed a more normal course. Dave, how about you?
[Dave] I tell you what. I wore it on my sleeve. What we do matters. And since that period and since leaving RSA and having an opportunity to work with some other wonderfully great companies, every time I joined a new company recently here at Bugcrowd five months ago and as I gotten to know my colleagues here in the sales organization, I really, really always work hard and try to remind all of us about why we’re here and what we do matters
[. . .]
So to me, it just – it really solidified frankly my passion. And I’m not afraid to talk about it. I don’t care if it comes across as corny. This is more than just about getting purchase orders and selling and growing businesses. It’s really about trying to help these customers who are still on a tough spot today.
