Malicious Life Podcast: Shawn Carpenter - A Cyber Vigilante

Malicious Life Podcast: Shawn Carpenter - A Cyber Vigilante Transcript

Cyberspace is full of vigilantes: Anonymous became famous for it, as did Edward Snowden. Hackers ransack major corporations on the internet, or deface government websites in the name of political ideology, and whistleblowers uncover the crimes of the powerful.

The vigilantes don’t ever seem to be on the side of those corporations, or those governments. It makes sense: after all, you can’t have a vigilante working cybersecurity for your organization, or vigilantes inside of, I don’t know, the CIA. It doesn’t work like that. There are rules and regulations and laws to abide by in the real world.

Except those regulations and those laws can be limiting. Hackers move fast; they don’t care about your rules. If you want to stop them, sometimes, it’s too slow or prohibitive to go through the proper channels–to contact the proper authorities, fill out the necessary paperwork, and so on and so on.

When the prescribed, legal thing to do just isn’t enough, you end up with someone like Shawn Carpenter. 

Shawn Carpenter never conceived of himself as a cyber vigilante. But, in the summer of 2003, he found himself at a crossroads. Down one path was what he knew he was supposed to do–as a citizen under the law, and a government employee. It was what he was ordered to do, in no uncertain terms, by important people who knew what they were talking about. Down the other path was what he felt, deep down, was the right thing to do. Which happened to be the exact opposite of what he was ordered; what he expressly wasn’t allowed to do.

Maybe you’ve faced this kind of situation before–it’s almost universal to the human experience. It’s head versus heart. Do you take the job that pays well, or the one you’re passionate about? Marry the girl your parents choose, or the one you love? I face this conundrum every day. Do I eat the green beans that’s good for me, or the french fries I desire deep down in my heart?

The difference, in Shawn Carpenter’s story, were the stakes at hand. If he went with his heart–did what he felt was right, the vigilante way–he would face, potentially, years in prison. By going with his brain, however, he was putting the cybersecurity of the entire United States of America at stake.

LOCKHEED BREACH

When Shawn Carpenter boarded a flight from New Mexico that summer of ‘03, he didn’t know his life was about to change. He couldn’t have guessed that he was about to become one of the most important people in the country. But he did know that something very serious was going on.

As an employee of Sandia National Laboratory, he did the kind of work that, usually, dealt with serious matters. Sandia researches and develops nuclear and other high technologies on behalf of the U.S. government, so you can imagine what it’s like to work there in cyber security.

On this occasion Carpenter was headed to Orlando, to the offices of Lockheed Martin. Lockheed happened to be the parent company to Sandia Corp., the parent organization of Sandia National Labs. And so, when hundreds of Lockheed’s computers started shutting down, all at once, without warning, Carpenter and his colleagues were summoned onto the first flight out of Albuquerque.

Lockheed, if you don’t already know, is arguably America’s premier defense contractor. They build the best planes, drones, satellites and defense systems in the world, on behalf of the U.S. military. The information stored on their computers is worth billions of dollars. More importantly, it contributes to the politics, the deaths and the wars that define our modern history. You can probably guess, then, what kind of attacker it’d take to successfully breach such an organization, and what their motives might be in doing so.

CLEANUP

It didn’t take too long, after arriving in Orlando, for Sandia’s security team to figure out what caused Lockheed’s shutdowns. Buried in their IT systems were sophisticated rootkits that allowed unknown parties to covertly access broad swaths of sensitive data. Compressed and encrypted files were sitting there, waiting to be exfiltrated–presumably, many other documents had already escaped. The malware itself suggested that these files were going to be sent to a server in…Here, I’ll give you a second to guess.

China.

Okay, that was too easy. There are really only two suspects in stories like this, right? I doubt too many of you guessed Canada or Switzerland.

After discovering what appeared to be some kind of Chinese cyberattack, Carpenter and his colleagues dutifully cleared the malware from Lockheed’s network, and prevented any more documents from exiting the system. Then, with everything up and normal again, they flew back home.

And that was the end of the story, for everyone but Shawn. He still had concerns. Wouldn’t you? It’s kind of like discovering a UFO crash-landed in the middle of the desert, cleaning up the debris, and then going home. How could you not be curious? Maybe it was as he was leaving–as he was boarding that flight back home, gazing out that little airplane window–that he felt a little twinge of dissatisfaction. Of not being done with the job.

HACK BACK

So he came up with an idea. He went to his superiors, and asked them for permission to hack Lockheed’s hackers. A “hack back.”

If you’re a longtime listener of Malicious Life, you might already grasp the problem here. We did a show about hack backs a few years ago–episode 19. It’s a great subject because, usually, hacking back is a bad idea. But, usually, not hacking back is also not good. 

Cyberspace tends to favor attackers. There are just so many ways to achieve a breach, and so many ways to hide your identity in doing it, that ordinary methods of cyberdefense and attribution often fail in the face of even reasonably capable adversaries. Sometimes you have to take that extra step–following the predator back to its nest–if you want to figure out who they are, or how to stop them.

But hacking back can turn a battle into a war. You don’t know who you’re going to meet at the other end of the tunnel until you’re there. More importantly, in most countries, hacking is simply illegal. Even if you work for a government entity, like the NSA, there are processes and regulatory hoops to jump through. If you’re a citizen, it’s just not an option.

Carpenter did work for a qualified government agency. In an interview with ComputerWorld, he noted how, quote, “I regularly used similar “back-hacking” techniques in the past to recover stolen Sandia password files and retrieve evidence to assist in system and network compromise investigations.” End quote. This time, though, his superiors rejected the idea.

CROSSROADS

And you can understand why. 

• A) Their job was done, Lockheed was secure again.

• B) This was clearly a highly sophisticated adversary: striking back might inspire them to attack again.

• C) They couldn’t steal back any of the data that was lost anyway. It was a sunk cost.

And there were a dozen other reasons not to do something so brazen. But that’s where a guy like Shawn Carpenter differentiates himself from the rest of us. You get the sense, from interviews and court hearings, that he wasn’t a disobedient person. As a former Navy man, he was much more stereotypically a soldier than a hacker: square-jawed, good looking, patriotic, but careful–mindful of his words and actions. At just 35 years old, though, he still had that eagerness and idealism we tend to grow out of in our old age. One New Yorker reporter described him as “a lean, excitable man who speaks in tangent-filled bursts.” 

You’ve talked with people like that, right? They’re so excited over whatever they’re going on about. Like, at one point, that reporter asked him why he wouldn’t give up this lead, even after his managers expressly told him to forget about it. He replied:

“I was pissed that they were stealing all this shit and nobody could fucking do anything.”

So what was he supposed to do? Just…let it go? Or pursue the attackers, in opposition to the direct orders given to him by the powerful leaders of Sandia National Labs? At the most significant crossroads of his life, Carpenter made his decision.

DECISION

And I think you know where this is going. But it’s worth emphasizing, before we continue, just how radical his course of action was going to be.

By pursuing Lockheed Martin’s hackers without permission, Carpenter would be operating completely on his own. Were he discovered, he would almost certainly be fired and then shunned from other military-defense jobs. More importantly, though, as a private citizen, he would be subject to the Computer Fraud and Abuse Act, which bans unauthorized computer hacking of all kinds. The maximum penalty for violating the law is 20 years in prison.

Carpenter understood this when, back home in Albuquerque, all by himself, he began to infiltrate China’s most dangerous APT. He later recalled what the experience was like. Quote:

“My objective started out with a purpose similar to the other investigations I engaged in while at Sandia. The difference in this instance was that the rabbit hole went much deeper than I imagined.”

HONEYPOT

It’s Spring, 2004, and, according to TIME magazine, another routine night for Shawn Carpenter.

“Carpenter, 36, retreated to his ranch house in the hills overlooking Albuquerque, [New Mexico], for a quick dinner and an early bedtime. He set his alarm for 2 a.m. Waking in the dark, he took a thermos of coffee and a pack of Nicorette gum to the cluster of computer terminals in his home office. As he had almost every night for the previous four months, he worked at his secret volunteer job until dawn [. . .] tirelessly pursuing a group of suspected Chinese cyberspies all over the world.”

How does one breach a sophisticated, possibly nation-state-level APT all alone, from a home office in New Mexico? 

Carpenter created a honeypot. Posing as a contractor, carelessly storing sensitive government documents on his home computer, he began by collecting a trove of actual government documents. They were declassified, but not obviously so, so to the naked eye they might look like serious stuff. He created fake search histories, and a whole network of other data to round out the ruse. When he exposed his made-up network to the web, he watched as hackers came in, looked around, then exfiltrated the documents. Then he followed them – exactly how, was never revealed to the media – to wherever they went. 

It was like this for months–sleeping only a few hours, waking up at 2 A.M. in order to be online during China’s working day. Then, after hours of tracing hackers around the globe, going to do a full day’s work at a national nuclear research facility.

ATTACKERS SPOTTED

It was during one of those early mornings in May that he spotted an intruder with the same profile as those he was looking for. Intruders who were clearly a level above the rest. 

“Carpenter had never seen hackers work so quickly, with such a sense of purpose. They would commandeer a hidden section of a hard drive, zip up as many files as possible and immediately transmit the data to way stations [. . .] They always made a silent escape, wiping their electronic fingerprints clean and leaving behind an almost undetectable beacon allowing them to re-enter the machine at will. An entire attack took 10 to 30 minutes.”

Carpenter was in awe–like a nature photographer, spotting a majestic predator in the wild, watching as it ceaselessly chases, catches, and then sinks its teeth into helpless prey. With “most hackers,” he told TIME magazine, “if they actually get into a government network, get excited and make mistakes. Not these guys. They never hit a wrong key.”

After flying across networks filled with far-flung computers, the group landed at a server in South Korea. Carpenter brute forced its password.

Inside was a hacker’s paradise: beacons, malware, and gigabytes-worth of data that definitely didn’t belong there. Millions of pages of documents included details of particular U.S. troop movements and body armor specs, schematics for the Air Force-commissioned F-22 Raptor stealth fighter, and the propulsion systems, fuel tanks and other parts being created for NASA’s Mars Reconnaissance Orbiter. It should be noted that the F-22 and the Orbiter were both ongoing projects of Lockheed Martin.

This South Korean server was just one of many designed to throw off the scent–there were others in Taiwan, too, and Hong Kong. They all led back to a centralized hub in the southeast Chinese province of Guangdong. Not only that: all the attacks stemmed from just three routers, connecting a single local network to the wider network.

In other words: Shawn Carpenter, on his own, had discovered the profile, the stolen data, and the exact location of one of the world’s most dangerous hacker groups. A group that would later come to be called “Titan Rain.”

Now, with insight into their IT systems, he began his hack back. He wrote a bug and installed it to the primary router in the network. Every time the attackers did anything on the web–whether it be hacking a U.S. defense contractor, or Googling “is Gal Gadot single?”–the script was to send an alert to a nondescript Yahoo email he controlled. Within two weeks, the Yahoo account had received over 20,000 alerts.

As he combed through the data, a bigger picture began to form. For each of the three routers on the network were somewhere between six and ten workstations, operated around the clock, presumably by employees on shifts. With that much manpower, they were able to launch attacks and siphon away files at an unprecedented rate. The files they obtained were stashed on the decoy servers in South Korea, Taiwan and Hong Kong, then pulled into Guangdong.

In addition to stolen NASA and Air Force schematics, the files included data from other U.S. government organizations, the World Bank, and a whole lot more. TIME Magazine noted how, one morning at 2 A.M.:

“Carpenter copied a huge collection of files that had been stolen from Redstone Arsenal, home to the Army Aviation and Missile Command. The attackers had grabbed specs for the aviation-mission-planning system for Army helicopters, as well as Falconview 3.2, the flight-planning software used by the Army and Air Force.”

There are two important things to say about all this data. Firstly: while it was all serious, none of it was, necessarily, critical to safety or national security. Generally speaking, important government agencies keep their most classified data on servers unconnected to the wider internet and, therefore, even powerful nation-state attackers cannot access it with ordinary network attacks.

But the content of the breaches was, arguably, not the primary issue at hand. You see, we’re talking about this in 2021, but the breaches Shawn Carpenter uncovered occurred in 2003 and 2004. Back then, it wasn’t so much a given that a sophisticated attacker could just, you know, hack any major U.S. government organization it wanted. And China, no less–China was hardly done building the internet in their country by this time, far behind the West in every respect. It would be like if, today, we discovered the most sophisticated cyber attack in the world being perpetrated by, I don’t know, Turkey, or Poland.

It was unprecedented, with no clear solution. First Lockheed, then others. In late 2003, it was the Department of Defense. An internal government memo described how, quote:

“These compromises … allow an unknown adversary not only control over the DOD hosts, but also the capability to use the DOD hosts in malicious activity. The potential also exists for the perpetrator to potentially shut down each host.”

This was a time of great change in cybersecurity–when the nature and scale of attacks, and the identity of attackers, was fast changing. The government didn’t quite know what it was, or what to do. 

Maybe, perhaps, Shawn Carpenter was the only one who truly did.

FBI PARTNERSHIP

Have you ever witnessed a crime while playing hookie? Or caught a boyfriend texting other girls while snooping through his phone? Shawn Carpenter had just uncovered a breach of national security, with some of America’s most significant military-defense projects in the hands of foreign adversaries. And yet, if he told anyone, he’d be revealing the crimes he’d committed in finding that out.

A couple of weeks after they were contacted by him, the FBI told Shawn Carpenter to stop cyber sleuthing. It was against the law, they said, at least until they received authorization through the proper channels. But he didn’t stop, because they didn’t actually want him to, because that authorization didn’t really matter.

David Raymond–nickname “Doc”–a tall and thin guy with short hair and a goatee–was assigned to Carpenter’s secret case. He was shocked by what Carpenter had uncovered, and didn’t want his new informant to stop any time soon. According to Carpenter, at least, Raymond wasn’t particularly troubled with how Carpenter had obtained the data he had prior to coming to the FBI. Raymond also gave him full permission to hack back the target.

And so, without the knowledge of anyone outside the FBI, the Army, maybe a few other government task forces, and his wife, Shawn Carpenter began secretly hosting strategy sessions with FBI agents in his Albuquerque home. His research fed into eight separate U.S. investigations, and three secret Army operations, and reached the highest levels of FBI counterintelligence. He was now one of the most important informants in the country. As Raymond told him, quote: “You have caused quite a stir, in a good way.”

FBI PARTNERSHIP SOURS

It went on like this for five months: the secret meetings, the hacking back. But all along, Shawn Carpenter–the patriot, the Navy vet–never quite trusted his FBI handlers. Raymond assured him, quote, “you’re very important to us,” and “we’re not going to prosecute.” End quote. But he knew that they could turn on him at any point, accusing him of illegal hacking and putting him behind bars. And so, in secret, Carpenter bugged his home with hidden microphones.

He began to wire tap the FBI. Just in case.

And he wasn’t entirely unjustified in doing so. While working with him to track down the Chinese hackers, the FBI began an investigation into Shawn Carpenter. 

In fact, they seemed to be investigating their informant better than they were their target. According to Carpenter, the FBI never asked for the credentials and other tools Carpenter had gathered, which might have helped them track down the attackers. He later recalled his frustration with the whole process, telling a reporter:

“It’s just ridiculous. I was tracking real bad guys. But they are so afraid of taking risks that they wasted all this time investigating me instead of going after [them].”

SANDIA FINDS OUT

The FBI didn’t prosecute Carpenter, but they did snitch on him. In late 2004, Sandia National Laboratories’ head of counterintelligence, Bruce Held, was informed that Carpenter had been covertly working with U.S. intelligence, without informing Sandia, expressly against the orders they’d given him a year prior.

Held–a balding gentleman with dark brown eyes, thin glasses and small, yellowed teeth–didn’t take kindly to the news. In an interview with Computer World, Carpenter described a meeting he had with his boss. It was January 7th, 2005. Quote:

“[A] semicircle of management was positioned in chairs around me and Bruce Held. Mr. Held arrived about five minutes late to the meeting and positioned his chair inches directly in front of mine. Mr. Held is a retired CIA officer [. . .] At one point, Mr. Held yelled, “You’re lucky you have such understanding management & if you worked for me, I would decapitate you! There would at least be blood all over the office!” During the entire meeting, the other managers just sat there and watched. At the conclusion of the meeting, Mr. Held said, “Your wife works here, doesn’t she? I might need to talk to her.”

Indeed, my wife did work there — in Sandia’s International Programs section, working on nuclear counter-proliferation, port and border security issues. In the context of that meeting, it was a chilling comment. Shortly after the meeting, which management described at trial as “a fact-finding session with Mr. Carpenter,” my director showed up at my office, escorted me to the gate and stripped me of my badge. That was the last time I was ever at Sandia.”

Shortly thereafter, Carpenter was fired and stripped of his top secret Q-level clearance, on the grounds of “being insubordinate” in “violation of the law,” and for “utilization of Sandia information outside of Sandia.”

Two months later, without notice, the FBI dropped all communication with him.

Carpenter’s investigation into Titan Rain was complete. He’d successfully determined where they were, what they’d stolen, and what they were capable of. But it still felt incomplete. It was that little twinge of dissatisfaction–of not being done with the job. “I’m not sleeping well,” he told one reporter. “I know the Titan Rain group is out there working, now more than ever.”

VIGILANTISM

The cyber vigilante is a romantic but dangerous thing.

We have laws in place for a reason. If everybody could simply hack whomever they felt necessary, or deserving, who knows what would happen? One company, one neighbor or spouse might hack another on grounds that turn out to be less than legitimate. Past that, should hacking really be in the hands of ordinary citizens? Societies have legal systems for the express purpose of preventing intercommunity violence–so that, for example, when somebody hurts someone you know, you don’t hurt them back, the government handles it on your behalf. Without that system in place, we’d revert to the ways of old. Eye for an eye. Every man for himself.

But then there are stories like Shawn Carpenter’s. Sandia Labs’ best interests were to avoid pursuing an important lead–because it would bring scrutiny on them, because it might invite more attacks, because it wasn’t their business. The FBI, which had the motive to pursue an investigation, couldn’t on their own move as fast as they needed to. So rather than shutting down Shawn Carpenter, or working with him reluctantly, they welcomed him wholeheartedly, making him a centerpiece of several counterintelligence operations that might have otherwise been lacking if not for their dedicated, law-breaking cyber vigilante.

So should we maintain the old laws, or make allowances for such cyber vigilantism? That was the decision facing a jury of 13 of Shawn Carpenter’s fellow countrymen, in a New Mexico district court, in February, 2007.

Following his firing, Carpenter had sued Sandia National Labs for defamation and wrongful termination. The events of 2003 to 2005 were put before the court. Carpenter and his lawyers requested millions of dollars in damages.

The jury decided to award him more than double. 4.7 million.

The jury forewoman called Sandia’s handling of Carpenter and the hackers “reckless” and “cavalier.” One jury member put it best, though, saying how, quote: “If [Sandia] have an interest in protecting us, they certainly didn’t show it with the way they handled Shawn.”

So, according to a representative panel of American citizens, Carpenter was in the right in disobeying his superiors and breaking the law. Still, that doesn’t mean future Shawn Carpenters will always be on the right side of history–that vigilantism is sometimes, even when it’s useful, the right answer.

ISIGHT

In his older age, Carpenter became, as he put it, “a lot more measured.” Like how, shortly after his firing from Sandia Labs, he became Chief Technical Analyst at a cybersecurity firm called iSight Partners. iSight handled cybersecurity investigations for major organizations like banks, and they did it by the book. For example, Carpenter recalled for a reporter how they managed to nab a hacker group–not quite a Chinese state actor, but pernicious nonetheless. From The New Yorker, quote:

“In his Sandia days, he said, he might have hacked members of the collective in order to gather intelligence on it; that would have been the easiest route. But at isight he went to great lengths to comply with the law. His colleagues spent sixteen months cultivating sources inside the collective, persuading them, slyly, to relocate their operations from a secure server to one that isight had legitimate access to, through the consent of its owner.”

Through their long and drawn out operation, iSight obtained comprehensive access to the innermost rung of the hackers’ network. And Carpenter–the “lean, excitable man who speaks in tangent-filled bursts”–couldn’t help but contain himself. 

“Boom! Just like that, we could see every plan—everything that they were doing.”

Just like old times, Carpenter caught his malicious hacker group. This time, he did it totally by the book.  Of course, this time–even with the resources of a full cybersecurity company behind him, against lesser attackers than Titan Rain–it took sixteen whole months.

He didn’t need to be a vigilante, but it would’ve been convenient.