Examining the importance of post-incident review for security teams
Post-incident review is a detailed retrospective that allows an enterprise to carefully understand each part of an incident, from start to finish. It is one step in the incident response process that requires a cross-functional effort from all individuals and technologies connected to the incident to truly understand the root cause and full scope of the attack.
Use a post-incident review to assess all the processes and people that were impacted by the attack so that it never happens again.
See Weaknesses
By completing a post-incident review, security teams are able to uncover network vulnerabilities and weaknesses that have been compromised by a threat actor. Further, they should identify all IOCs and TTPs associated with the attack, as well as internal issues that may have been manipulated by the threat actor to gain entry.
What does good look like?
A good post-incident review results in a list of practical actions that address each of the issues that allowed the threat actor to succeed. These actions should minimize the impact of an attack and teach the security team, the security tools, and the wider enterprise how to prevent, detect, and respond to a similar attack in the future.
Address the problem from the very beginning, not just the end.
Iteratively assess the complete picture of damage to prevent future incidents.
Mature your security program by investigating attacks from start to finish.
WHY IS POST-INCIDENT REVIEW IMPORTANT?
Many Hollywood movies conclude with a “replay” in the last third of the film: going back in time, adding details not shown during the movie, and pointing out the ones a viewer may have missed. These details fill the holes in the plot and resolve the who, why, and how - the stealthy information viewers try to identify for the first two thirds of the movie. These make up the hook, the point, and the solution for the complexity of the plot.
For Security
In a similar way, a cybersecurity incident is a whole story, with its own structure, characters, methods, complexity, and unnoticed details. However, many defenders tend to be satisfied with the rapid remediation of an attack before they move on immediately to the next threat.
With commoditized attacks, this approach is enough.
But with more sophisticated attacks, especially low and slow or targeted attacks, the initial incident is just a battle in the bigger war.
Low and Slow Attacks
With low and slow attacks, the attacker’s progress across the kill chain may be scattered across a long timeline involving different network segments or third-party channels to the network. Attackers may use different tools and techniques in each part of the attack, access the data they want, and then remove any trace of activity. These attacks pass under the radar of many security products where data retention time is short and event correlation over time is weak.
SEE THE FULL SCOPE OF THE ATTACK
In these instances, defenders are unable to get the full picture of an attack and are only able to remediate the final, most recent stages. This leaves exposed areas for attackers to target.
The Nation State
For example, nation-state threat actors may begin their attack with a commodity cyber attack framework. They scan the perimeter for vulnerabilities, gain access to a specific endpoint or server, steal local admin credentials, and wait. They wait to use the credentials weeks later to regain access to the network. Once in, they use different tools and techniques to accomplish their main attack objectives. In this case, the attack is only detected in its last stages. If the retention time of the SIEM or EDR is smaller than the length of the attack, the defender is going to miss the initial compromise in their review. They will only respond to the final stages and be totally oblivious to the bigger picture. This leaves the network exposed to those same exploits and entry points the attacker used at stage 1.
Low and slow attacks easily bypass traditional security defenses by incremental actions that on their own are too small to detect, but put together can devastate an organization.
The majority of organizations lack the in-depth visibility to identify the entire lifecycle of a sophisticated attack like this. Answering the question of, "when did this attack begin?" is increasingly difficult.
A single, low-priority alert in an ocean of incidents is able to operate below the radar, and may even look like legitimate activity.
These attacks are only becoming more common, especially among nation state threat actors that want detailed information on an enterprise or government.
Attackers spend an average of 197 days of dwell time in your network before being detected.
In 2018, the Cybereason Nocturnus team identified an advanced, persistent attack targeting global telecommunications providers carried out by a threat actor using tools and techniques commonly associated with the Chinese-affiliated threat actor APT10. This multi-wave attack focused on obtaining data of specific, high-value targets and resulted in a complete takeover of the network.
Read About the Attack →
HOW TO PERFORM A POST-INCIDENT REVIEW
Follow Post-incident Review Best Practices
STEP 1: State the Goal
Goals for a post-incident review should cover four tiers and revolve around learning and improving.
Learn how to detect and respond to similar attacks, then improve the existing defense.
Learn how to prevent similar attacks in the future, then improve preventative actions.
Evaluate the cost-effectiveness of the current defense given its performance in this incident.
Assess the damage the incident caused and confirm no further damage will occur.
STEP 2: Assess Roles
Identify each individual affected by the incident, whether they were involved or should have been involved. Were they able to successfully defend? Were they impacted by the incident?
STEP 3: Ask Questions
Create a list of questions derived from the goals that concern at least one stakeholder. Group the questions by the stakeholders identified when assessing roles.
Assign each group of questions to a single dominant stakeholder. This stakeholder must have access to the appropriate systems and resources to be able to accurately form a detailed answer for each question. Allocate appropriate time and resources so they are able to do a thorough job. This may involve internal or external resources depending on the skillset of the stakeholder.
Bear one thing in mind: Sometimes it is important to leave the “who” and focus on the “what”, “why”, and “how”.
STEP 4: The Technical Stakeholder
The technical stakeholder must simulate the chain of events from before the incident began until the very end. This should be meticulously crafted in a very high resolution, down to the last detail. It is critical to scope out missing parts and leave no stone unturned. Some areas to consider include:
Vulnerabilities
Weaknesses
System Configurations
Network Operations
By the end, the technical stakeholder must be able to explain how the threat actor was able to infiltrate, as well as where the attacker succeeded and failed. This must include all relevant data for the scope of the attack, no matter how far back in time the attack started.
STEP 5: Build a Timeline
Once each stakeholder has answered their questions, every member of the team should come together to connect the dots. Build a complete timeline using the perspectives of every stakeholder.
STEP 6: Establish Counteractions
Assign counteractions, remediations, improvements to the defense, architecture changes, and any other actions that will help prevent this type of attack in the future. Be practical, clear, and detailed.
If there are issues that cannot be addressed immediately, try to find a workaround. For example, if a certain server cannot be patched, make sure the detection system covers it and that its connections are limited and secured.
Assign each task to the appropriate stakeholder: the one that can accomplish the task in full. Set follow-up meetings for accountability and to address any lingering issues.
STEP 7: Help the Community
If this type of attack has the potential to impact other enterprises, find a way to inform them of the threat. Give them the much-needed details to prevent these incidents and be part of a greater community of defenders.
Look back in time to before the incident started. Are there any indications the attacker is already in your environment?
Walk through the incident step by step. Identify what happened before detection, during investigation, during response, and after remediation.
Talk about if the SOC was well-prepared for the incident. Were they able to resolve it quickly? Was someone on call to handle it?
Consider the information your team was able to gather about the incident. Were they well-prepared to react? How much time passed before remediation?
Consider whether the processes followed was as efficient and effective as it could be. What can be optimized? What can be removed?
Consider if each member of the team contributed as effectively as possible. Does the team understand what they need to do? Do they need training?
Retrospectively analyze every piece of the attack. How far back can you find the attacker in your system?
Did your team have the visibility it needed to resolve the incident completely? Think of good wish list items that you want your team to have to succeed.
Consider if management was well-prepared for an incident. Were lines of communication open? Did management understand the gravity of the situation?
Network forensics is useful, but is limited to 2-4 weeks worth of raw data, with a few additional weeks for metadata. Further, capturing and storing packet capture is expensive.
The main driver for longer data retention is compliance. Vendors offer a way to efficiently compress and archive data, but no easy way to access and correlate it.
Cloud-delivered platforms are a step in the right direction, but the cost of data shifts to the vendors, who have little incentive to store and pay for all that data.
When a customer suspects a breach and deploys an incident response team, their first task is to obtain all data available about the incident. Incident responders are very good at sifting through data to uncover the needle in the haystack. However, this is a unique ability that takes a long time to hone, and one the majority in the industry does not have.
Data Retention
Compounding this, most organizations do not have all data available as part of their regular investigation because of compliance reasons or, alternatively, the inability to store it. Further, the initial point of compromise may have happened weeks or months before detection, which could easily go well beyond an organization’s data retention limits.
Chain of Attack
With retrospective hunting, enterprises are able to understand how widespread a campaign is by accessing data as far back as their team needs to go. They can identify all users and entities involved in the incident, and graph that behavior as a detailed chain of events.
Answer “What If”
Retrospective hunting lets security teams answer the critical question of, “Are we impacted by this world-wide breach?”. The second a large attack on another organization is reported, an enterprise can use retrospective hunting to search for the new zero day and confirm they are not vulnerable.