Transcript
Fifteen years ago, a number of high-profile Tibetan monks received the same innocent-seeming, nondescript email (yes, monks sometimes have email). Those who clicked the attached DOC or PDF files downloaded a trojan horse which, often, led to a second malware: GhostRAT.
GhostRAT underpinned the GhostNet operation we discussed on this podcast a couple of years back, in which China spied on economic, political, and media targets in over 100 countries. It worked by taking advantage of vulnerabilities in Microsoft Office and Adobe Acrobat, often flying under the radar but sometimes causing a flash onscreen or outright crashing an application. The result, though, was powerful, enabling China to download further malware, steal documents, or simply spy on their targets via their keyboards, microphones, and webcams.
This, generally, is how most of us think of hacking: an infection vector — usually simple social engineering — followed by different stages of malware which allow an attacker to establish persistence, move laterally, and cause further actions in a host computer or network.
In recent years, though, hacking has started to not look like this in some meaningful ways. Cyberattackers — particularly the most advanced, nation-state-level APTs — have made a characteristic shift in their tactics. Their new, favorite strategy is changing the ballgame not just for them, but for their victims, and the people trying to defend those victims.
To demonstrate, we’re going to focus on one recent case study.
First Signs of Volt Typhoon
“[Lambert] the origin story for us goes back to August of 2021, where there was disclosure on an attack on a port in Houston.”
John Lambert is a Security Fellow and Corporate Vice President at Microsoft who founded the Microsoft Threat Intelligence team. Two years ago, his threat research operation was called in to analyze the forensic data leftover from a cyberattack against a port.
“[Lambert] I think what stood out is interesting in the beginning of this is when a cyber attack happens to a port.”
Cyberattackers target large corporations all the time, usually for money. Small and medium companies too, since they’re easier to pick on. They sometimes target governments, too, either for money or for political reasons. But a port? Why exactly would hackers target a port?
More Incidents: FBI, Guam
The analysts started to tie together some initial pieces of intel.
“[Lambert] A few months later, in November of 2021, the FBI released a report about an actor that was using a zero day in a VPN appliance known as FatPipe.”
FatPipe MPVPN, a VPN security solution. The vulnerability, assigned as CVE-2021-27860, was given a “Critical” 9.8 out of 10 rating by CERT — the kind of rating saved for only the most serious bugs out there.
“[Lambert] The actor they described was using that zero day to drop web shells, which are a form of a backdoor onto networks.”
There was no known connection between the port attack and the FBI report. For all anybody knew, they were entirely unrelated phenomena.
But as the months went on, like a serial killer’s murder spree, more attacks kept popping up in different places, but with a few, important shared features. For example, the attackers encrypted their communications using a specific encryption key…
“[Lambert] That was something like MAGA2024.”
A joking reference to Trump’s presidential reelection bid.
“[Lambert] And it was just important that it was unique and that allowed us to start fingerprinting you know, where did we see this pop up? And understand where in the world this might be and and get a sense of timeline.”
One year after the port — while Americans were distracted by a giant Chinese spy balloon hovering over the U.S. mainland — there was an overlapping cyberattack, this time against a telecommunications company in the small U.S. island territory of Guam. Shortly thereafter in the fall, another against another telco in Guam.
(For the record: there are only three telecommunications providers in Guam. So cyberattacks against two of them stands out.)
“[Lambert] Guam is of geopolitical significance and a telco is certainly a very important type of collection target. And because telcos are involved in many aspects of ICT infrastructure, they are commonly in the crosshairs, and a compromise of a telco is pretty broad reaching. So right there. That’s sort of the kind of thing that makes an analyst go – well, this is important.”
Volt Typhoon Attacks Explode
“[Lambert] And a few months later, as we progress into winter of 2023. We start to see a increasing preponderance of us victims, and in a variety of sectors.”
The same group’s fingerprint was discovered in attacks against transportation, communications, construction, maritime, and education sector organizations. And IT companies, manufacturing plants, government agencies, power plants, and water treatment facilities. Basically, every industry that’s critical to the everyday functioning of a nation. Which posed a dilemma.
“[Lambert] There are many kinds of organizations that are targeted, because of just the data that they have, you know, or the access to data that they have, and so, you know – diplomatic organizations, military organizations, the defense industrial base, all those kinds of organizations are pretty traditional geopolitical based targeting have broad relevance. But there are some organizations that really have no intelligence value. They don’t really have any data. And that’s what we started to see in the mix of victims here. When we saw water companies and different states in the United States. Water companies don’t have really any real intelligence that’s worth collecting.”
It led to one uncomfortable, but inescapable conclusion.
“[Lambert] Disruption is the potential impact.”
Government services shutting down, power or internet going out, water going untreated.
“[Lambert] And so it’s even more important, or it raises the importance of being able to find and evict these actors from those networks.”
Scanning for Exposed ֱDevices
Microsoft gave the adversary a name: “Volt Typhoon.”
As Volt Typhoon claimed more and more breaches, the researchers could get a better sense of how, exactly, they did what they did. What made them so successful? How had they slipped past authorities and analysts for so long, while attacking high-value targets around the world?
Well, like just about any APT, they had a set of tactics and tools they use in each of their attacks which we can analyze.
“[Lambert] So a typical attack pattern would look like they would eventually get sighted on a particular organization that they wanted to go after. They would understand its external network presence: what IP addresses mapped to that organization, what is exposed on the Internet, what kind of software is running on those IPs?”
This isn’t quite as ordinary as it sounds.
Obviously, hackers will aim to eventually compromise desktops and servers in a cyberattack. Those are the most important machines in an IT network, where you can download malware, steal data, and so on.
But Volt Typhoon does something more elegant. Instead of using social engineering to try to convince their targets to download malware to their computers, they scan the internet for devices in their target networks which are exposed to the internet. It’s not as difficult as it sounds — plenty of tools allow you to do this quickly and easily, like the Shodan search engine.
With Shodan, you can find devices around the world which are exposed to the open web, either by accident or, often, by design. For example, network routers — developed by companies like Cisco, NETGEAR, and ASUS — that connect corporate networks to the wider internet.
“[Lambert] Devices on the edge that had vulnerabilities in them, often vulnerabilities that were patched [by the vendor] but not patched by the victim network.”
By scanning for internet-exposed devices running vulnerable software, Volt Typhoon completely bypasses social engineering. This not only means less work for them, it also completely eliminates the risk of a smart employee flagging a suspicious phishing email or text.
Advantages of Network Devices
Maybe you’re wondering: if hackers can take advantage of them, why would any organization leave their devices exposed on the open web? The thing is, they’re often that way by design. A network router, wireless access point, firewall, VPN, or remote server management tool — these are made to connect to the wider world.
It’s a bit of a Catch-22 — organizations need these devices to operate, but using them carries serious risks…
“[Lambert] Not only are those devices exposed to the internet by design, they often operate with high privilege with elevated credentials: VPN appliances often have credentials of many users that are using them and they serve as a bridge into an internal network that allowed them to quickly escalate privileges.”
Besides their openness and privileges, network or “edge” devices enable Volt Typhoon to design a base for their operations that’s extra resistant to analysis.
“[Lambert] They often want to be able to have infrastructure that is temporary, ephemeral and hard to attribute back to them. So they certainly wouldn’t take out a credit card of their own and rent a virtual machine somewhere. They would try to get either infrastructure that belongs to somebody else that they can compromise, which was indeed the case here, and then just accumulate that over time and then be able to stitch together a routing path through those devices on networks around the world. [. . .] because there’s so many of them, and many of them that are vulnerable, they’re able to accumulate a pretty big collection of infrastructure that they can use.”
These weren’t the only challenges facing Microsoft’s and the government’s analysts.
“[Lambert] They’re pretty dark devices from being able to understand what’s happening on them and so forth.”
There are sophisticated visibility and monitoring systems that give us a window into what’s happening on our computers, automatically tracking and picking up on abnormal behavior.
But network devices don’t really have users — we don’t often conceive of them as locations where bad guys can do malicious things — so we don’t have the same kind of software for looking inside of these devices, monitoring anything abnormal going on inside, and doing anything about it if something like that were to happen.
“[Lambert] So that helps them preserve their stealth. [. . .] and from there, they would route their attack over a covert network and eventually seek to get credentials on that device. So for example, if they have an exploit for a firewall or VPN appliance that that victim is using and that victim hasn’t patched it, then they’ll have a shell onto that device. And then from there, they’ll dump credentials, and once they get elevated credentials, they’ll begin the process of pivoting to the internal network of the victim.”
In this way, these vulnerable, overlooked devices on the outskirts of a company’s network act as a launchpoint. With regular command line functions, Volt Typhoon can gather credentials from devices, package and exfiltrate them, and then use them to establish persistence and spread further into and around the network, compromising more machines and gaining more credentials at each step of the way, with the ultimate goal being to reach the very highest level of access possible.
LotL Techniques
But notice how throughout this whole process, besides a simple shell, they’re not actually deploying any cyber attack tools.
“[Lambert] They’re not using custom malware. They are using the built in functionality on computers like in Windows that they’re using WMI, or other commands that are just native system utilities.”
There are so many ways in which Volt Typhoon did the same kinds of things you’d expect malware to do, but without the malware. Like masking their actions not through any sort of fancy encryption, but by using command line functions which are otherwise totally legitimate, alongside all of the other, perfectly innocent traffic happening on the network.
There’s a term for this:
“[Lambert] Many of the techniques they’re using as they’re going along are this type of technical Living Off The Land.”
“Living off the land” involves using the native functionality, tools, processes of a system to carry out malicious activity. It first came into public consciousness in 2018, but only recently has it become perhaps the single biggest trend among malicious actors, because it just works so well.
Think about it: analysts and security software regularly thwart cyberattacks by identifying foreign software and unrecognized traffic. But here, there isn’t any software raising a red flag. And just imagine, in monitoring a large organization with tons of traffic, how terribly difficult it would be to somehow identify which entirely legitimate functions may be being used illegitimately by hackers.
And if they do need any software in the course of their attacks, Volt Typhoon can use commercial off-the-shelf or open-source tools, like Fast Reverse Proxy.
“[Lambert] This is a piece of software that is used for benign purposes and networks, because proxies are just networking related software, and let them take a server that’s inside the victim network and then expose it to the Internet where they could directly access it.”
More often than not, there’s an even easier solution than using open-source or commercial software. Hackers already have more than enough at their disposal by using what comes packaged in any standard Windows computer.
For example, they could try sneaking a suspicious remote access trojan like GhostRAT or Trojan.Hydra past cybersecurity defenses and onto a host computer, then try to use it without raising any alarms. Or, alternatively, there’s PsExec, Windows’ built-in feature for allowing users to execute processes on remote systems.
“[Lambert] That they can use for the purposes of dumping credentials, getting persistence on machines and lateral movement, and so on. And then once they’ve done this process, they have access to this network.”
Lambert and his team observed Volt Typhoon leveraging Windows features all over the place, like during the first Guam telco attack.
“[Lambert] We realized they’re looking at logon events inside of a victim network. And a reason an actor would do that – In Windows we call those “Event ID 4624”. And network people will be familiar with that: that tells you it’s like a record of when somebody logs onto a system and what actors. The reason they’re interested in those is they’re interested in what admin accounts are in a network logon events would certainly tell you when IT admins logging onto a system and importantly where that admins workstation is, where are they logging on from? Because if the actor can get to that machine, then they can actually get the credentials of that administrative user.”
Today, cyber threat actors across the world are using LotL to do what they previously did with malware. Other Chinese APTs, like “BlackTech” and “Flax Typhoon,” do the same thing Volt Typhoon does, but so do the Iranian groups “Remix Kitten” and “Charming Kitten.” Besides Volt Typhoon’s targets, other U.S. government and defense targets have fallen victim to attacks using LotL tactics.
And it’s not exclusive to the highest-level nation-state groups — ransomware actors love LotL more than anyone. Many organizations use remote monitoring and management tools — RMMs — so that administrators can connect to their systems over the web. Groups like LockBit have figured out that they can more easily spread their ransomware simply by using these existing RMMs.
It’s almost certain that LotL will only grow in the years to come, as bad actors devise more clever ways to use our devices to do their work for them.
Scale of Volt Typhoon’s Threat
Despite its many intrusions into the organizations responsible for maintaining daily American life, Volt Typhoon has yet to enact any tangible, visible damage. But this, according to the Biden administration, is no reason to wait and see.
In June, The New York Times reported that Volt Typhoon had already inspired a series of meetings in the Situation Room between the president, the National Security Council, the Department of Homeland Security, and the Pentagon. Reporters spoke with more than a dozen officials about the classified details. Quote:
“They say the investigations so far show the Chinese effort appears more widespread — in the United States and at American facilities abroad — than they had initially realized. But officials acknowledge that they do not know the full extent of the code’s presence in networks around the world, partly because it is so well hidden.”
One congressperson called it “a ticking time bomb,” not least because we still don’t fully know what China is planning to do with such information, and such access. Quote:
“There is a debate inside the administration over whether the goal of the operation is primarily aimed at disrupting the military, or at civilian life more broadly in the event of a conflict. But officials say that the initial searches for the code have focused first on areas with a high concentration of American military bases.”
That military facilities would be the primary targets would help contextualize those early attacks against telecommunications providers in Guam. Guam is an essential front from which the U.S. exerts its influence over southeast Asian affairs, including in the South China Sea — where China has steadily increased its military presence in recent years — and in the protection of Taiwan. If Chinese state hackers breached the island’s telecom providers, well, you can do the math there.
Advice for Listeners
“[Lambert] if someone was an organization that would be in the crosshairs here, there’s a number of things they could do.”
In recent months, John and his team have been trying to spread word about how organizations can defend themselves against Volt Typhoon and its LotL tactics, before the ticking time bomb goes off.
The first, most obvious step is to try to block them out before they can get to the devices where LotL becomes possible.
“[Lambert] While many organizations understand their computers need to be patched and we’re all familiar with that on our phones and our desktops. Anything with a blinking light, in a network? All of these devices, those also run software and need updates.”
Organizations running updated, patched, well-defended networking devices sever Volt Typhoon’s primary attack method at its head, forcing the group into alternative strategies.
But how can you stop LotL if it’s already happening? That’s the real trick — detecting it.
If you already have the attacker’s fingerprint — for example, commands they like to use, or hashes for their custom binaries — you can look for evidence of them in your network. Without that, the best you can do is behavioral analysis: analyzing data gathered by monitoring and detection tools to try to find anomalies in network traffic.
That’s why Lambert emphasizes the simple things any company can do to make a hacker’s job more difficult.
“[Lambert] When it comes to credentials, multifactor is a huge hammer. That’s very effective for customers, if they just choose simple passwords that could be guessed, you know, that’s the kind of thing these actors love. The only people that really love passwords are hackers. And so multifactor is a very strong recommendation here. [. . .] And then there’s also just modern defenses: EDR systems, Event Log collection, those kinds of things that make the process of discovering and investigating attacks, not only easier for the victim organization, but also for any security responder that would be there to help them.”
Scope of the Issue
There is still a remarkable amount of ground to be made up, though, even in the organizations that can least afford to fall victim to a Chinese cyber attack.
“[Lambert] not every organization and critical infrastructure is a sophisticated IT organization. Some of these water companies are quite small, and for example, might even be family owned.”
In critical water treatment plants, hospitals, even the government — where, you’d imagine, computer networks are highly protected and follow strict regulatory guidelines — there are a surprising number of opportunities for adversaries.
Back in the spring, the threat hunting company Censys scanned for exposed devices running at more than 50 government organizations and sub-organizations. In all, they discovered 13,000 distinct hosts, with hundreds running exposed software with potential vulnerabilities or misconfigurations.
Which brings us to May 24th of this year, when, after two years of investigating, Microsoft finally published its report, and CISA — the Cybersecurity and Infrastructure Security Agency — with the FBI and NSA, and authorities from Australia, Canada, and the U.K., published a own joint security advisory warning about the new threat actor Volt Typhoon.
Just three weeks after the world learned of the sophisticated Chinese group leveraging network appliances and LotL techniques to compromise high-value American targets, CISA released the “Binding Operational Directive” 23-02, ordering that government agencies eliminate all internet-exposed management interfaces running on edge devices within 14 days.
If the notoriously slow-acting government can make such meaningful progress in just two weeks, so can the rest of us. And we need to. If foreign adversaries are able to use our own devices against us before we’re able to protect them, there’s no saying exactly what will happen. You might just long for that time, way back, when China created immensely powerful cyber weaponry.
Ah, the good old days.
