How Google Does It: Collecting and analyzing cloud forensics

Ever wondered how Google does security? As part of our “How Google Does It” series, we share insights, observations, and top tips about how Google approaches some of today's most pressing security topics, challenges, and concerns — straight from Google experts. In this edition, Google staff security engineer Aaron Peterson shares an inside look at Google’s approach to cloud forensics, and offers top tips for hunting down evidence in the cloud.

Some security incidents are too widespread or complex for initial responders to tackle alone — even at Google. When a situation calls for a more advanced response, we call on a number of internal groups, including the Incident Management and Digital Forensics team.

Their mission: Manage large-scale incidents and determine the impact on our customers and users. That’s a formidable task considering these investigations unfold across planet-scale infrastructure, including the world’s largest Linux fleet and Kubernetes clusters running hundreds of thousands of nodes.

However, tracking down digital evidence in the cloud to uncover the truth behind an event requires a modern forensic playbook. While the fundamentals of traditional forensics remain, investigation techniques have to evolve to meet the requirements of today’s environments, where incidents can involve numerous actors, devices, networks, and locations.

Here are some of the key aspects of our approach to cloud forensics.

Acquiring evidence in the cloud

Attacks are rarely confined to a single system, and artifacts — the digital traces used to reconstruct incidents — are frequently spread across different locations and shared infrastructure. Evidence can be difficult to locate and identify, and sometimes it can vanish completely before an investigation begins.

From a security perspective, we’re lucky at Google to have extensive telemetry that makes many traditional forensic methods unnecessary. Cloud Logging , for example, contains a wealth of detailed, chronological information of actions and events, including those occurring in Kubernetes environments. This rich log data provides critical insights into system activity, and also eliminates the need to access individual virtual machines in many cases.

At the same time, cloud forensics still heavily depends on the ability to conduct deep system analysis. Given the scale and scope of our operations, it can be challenging to do an in-depth analysis of every system and component. This means that to investigate security incidents effectively, we also need to acquire targeted digital evidence, often in real time, live through the network.

While traditional forensics once relied on physical access and time-consuming imaging, modern at-scale response has shifted toward live investigation and cloud-native capabilities. We employ specialized forensic tools to help us acquire targeted evidence without disrupting operations. GRR Rapid Response (GRR) enables us to automatically hunt for and collect targeted evidence from specific systems and machines while they’re running.

We also use Cloud Forensics Utils to capture disk images and other artifacts from cloud platforms. Rather than acting as a hurdle, the cloud enables near-instantaneous remote snapshots. In the rare instances where traditional imaging of a local device is required, we also have capabilities to capture physical disk images and securely upload them to the cloud for analysis.

Together, this powerful combination allows us to navigate many of the inherent challenges of investigating in the cloud while enabling our analysts to retrieve all the relevant evidence needed to solve an incident.

Automating the forensic workflow

One of our favorite practices from Site Reliability Engineering (SRE) is eliminating toil from our work, and cloud forensics is no exception. Our goal is to make the initial stages of investigations as easy as possible for forensic analysts from the start.

We automate as much of the forensic process as possible using an orchestration tool to initiate and manage our workflows and tools. Once we acquire relevant artifacts, our orchestration triggers a distributed processing engine to efficiently manage our forensic workloads, allowing us to process large amounts of evidence at scale.

Part of that processing runs a timelining tool that extracts all time based artifacts and organizes them into a clear, chronological timeline. These timelines are then integrated into a collaborative timeline analysis platform , enabling analysts to quickly search, examine, and collaborate on the data.

Automating our forensic workflow gives our analysts more time and energy to spend on what matters: Understanding and investigating incidents. Analysts receive a direct link to the timeline and analysis environment with the disk evidence attached for any necessary manual forensics. To simplify forensic tooling setup, we also provide an easy way to deploy our stack of open-source forensic tools to make it easier to set up a complete analysis environment.

Building a solid incident response plan

At Google, we believe that preparation is key. Cloud forensics is most successful if you already have a deep understanding of your tools and environments before an incident occurs. That means always having a clear incident response plan .

We have documented incident response plans in place, establishing defined roles and responsibilities during an incident, communication protocols, decision-making frameworks, and escalation procedures. We also maintain Service Level Agreements and guidelines for response times, which help us determine when it’s time to escalate, or bring in additional support.

For example, Google’s security teams can escalate high-priority incidents to the Incident Management Team. This system allows the other teams to focus on technical investigation, with Incident Management stepping in during critical scenarios to handle coordination and communication until the situation is under control.

Crucially, we also regularly refine our plans to strengthen the way we manage and respond to security incidents. We incorporate blameless postmortems after every incident to identify what went well, what didn’t, and where we simply got lucky.

This philosophy is another core SRE tenet, enabling us to document incidents and ensure all the contributing causes are well understood, without allocating blame to specific individuals or teams. We then take all of these key learnings and integrate them back into our processes, continuously improving our ability to manage, investigate, and respond to future incidents.

This article includes insight from the Cloud Security Podcast episode, “ Ghostbusters for the Cloud: Who You Gonna Call for Cloud Forensics .”