Prerequisites

• Athena set up so you can query CloudTrail.

The Lesson and Lab

Today we'll do a bit of a skills challenge. Unlike some of our other skills challenges, this one's not designed to test new or existing skills. It's designed to actually help you learn and work your way through a particular problem.

The Setup

In our last lab we learned how to set up Athena to query CloudTrail logs, and tried a few basic queries. For this challenge I’ll point you towards a blog post I use as my reference for initial incident response, and then we will simulate an attack on your account.

Your challenge, should you choose to accept it, is to use those queries to piece together more of what happened. Being early in our incident response learning, I don’t expect you to run completely through the process, but you should work with a few of the queries to get a sense of how things piece together. In the next lab I’ll run through it myself, showing you how I approach the problem.

Our goal is to reinforce how to write Athena queries for security — this isn’t a test, but a chance to learn and explore more on your own! 

Here’s how it will work:

You’ll run a CloudFormation Template in TestAccount1. This will build some things, including an IAM User with static credentials, and share that information with me over EventBridge. On my side it will trigger a simulated attack: a series of API calls (run in a Lambda function) similar to an attacker finding credentials and then using them to exfiltrate data.

Everything is tightly constrained and I’m happy to share the Lambda source code if you have concerns. The TL;DR is that my simulator can only work with resources with a very specific naming pattern which embeds the account ID, and the account ID is cross-referenced with the metadata from EventBridge, so you can’t spoof the system and trick it into touching an account other than the one which originated the message. Nor can you trick it into touching resources which don’t match the names it expects.

You will:

1. Run that CloudFormation.

2. Let it run for a few minutes.

3. That simulates an attack.

4. If you’ve done our other labs, you’ll get an alert in email!

5. Then it shuts everything down.

6. You investigate with the queries.

7. You delete the stack (you can delete it whenever you want after it initially runs — what we need is already in the logs at that point).

It’s a one and done, and all the hard work happens with the logs.

Video Walkthrough

Let's Do This!

First open the blog post at https://securosis.com/blog/aws-cloud-incident-analysis-query-cheatsheet/

Sign into the console, go to TestAccount1 > AdministratorAccess > region us-west-2 (Oregon) > CloudFormation > Create stack:

At this point, you know the drill:

• Use the S3 url https://cloudslaw.s3-us-west-2.amazonaws.com/lab61.template

• Name it simulation.

• Leave everything else at defaults and Submit.

You’ve done this like 87.5 times now, so no more screenshots!

What's Happening Behind the Scenes

In the background this:

• Creates a VPC (because you didn't have any in your account) with a private-only subnet.

• Launches an instance in that subnet.

• Immediately stops that instance so you don’t get billed for it.

• Then creates an IAM user with an access key and secret key.

An access key and a secret? Remember when I said never to allow this? Well, now we're going to learn why, because we will actually simulate what an attack would look like.

That access key and secret key are sent to me over EventBridge. Those feed into a Lambda function. Now, just so you know, I don’t storing them anywhere. They are not logged. They are not stored in S3. They're not stored anywhere.

That Lambda function uses those credentials to create a snapshot of our specially-named instance, and to make the snapshot public.

Finding Your Starting Point

Two things should happen within a few minutes:

1. If you set up Access Analyzer correctly in our previous labs, you'll actually get an email alert coming out of Security Hub from Access Analyzer that something is now public. If you get that email, copy the snapshot ID.

2. Otherwise, you can go into EC2 > Snapshots and look for one named deleteme which is public. Copy the snapshot ID.

I love how our controls work together, so a lab from months ago triggers an alert near real-time. With that snapshot ID it’s time to start your queries.

Time to Investigate

Will we query the logs here in this account? Nope, we close the tab > sign in portal > SecurityAudit > SecurityFullAdmin > Athena. Why? Because that’s where Athena is set up from previous labs.

You likely have tabs open from our last lab. Close them out. While I want you to work with the queries over at the Securosis blog post on incident investigations on your own, I’ll show you the first one to get started.

Your First Query

Here’s the first query, taken right from that post, with the table name pre-filled, since we are all using the same one:

SELECT
useridentity.arn,
eventname,
sourceipaddress,
eventtime,
resources
FROM cloudtrail_logs.organization_trail
WHERE requestparameters like '%<your snapshot id>%' OR responseelements like '%<your snapshot id>%'
ORDER BY eventtime

Copy this, paste it into your Athena window, replace <your snapshot id> with your snapshot ID (don’t leave those brackets!) and wait about 45 seconds for results:

You will see all results where the request or response included the snapshot ID, which is a pretty good representation of everything involving that snapshot.

Your Mission

There are a bunch of queries on this list. What I’d like you to do is play with the queries on your own to gather more information. Your objective: piece together the story of what my little attack script did.

You’re starting with that snapshot ID because that’s where you got your alert. From that alert, you want to piece together, well, what actually happened? We have all of our events — that's what we did with Athena. But there are other areas to look at, such as:

• Who or what made the API calls?

• What is that snapshot of?

• What permissions did that user have? (You may need to go back into TestAccount1 to look).

Your objective here isn’t to do everything you would in an incident response. I want you to focus on the basic Athena queries. Try to spend about 15 or 20 minutes working on this, but feel free to do less or more. I’m not your dad. (I can say that because my kids never read anything I write anyway).

If you follow the RECIPE PICKS process, you should be able to figure out the timeline of events, the potential origin (hint, it was leaked credentials), the blast radius (what else could that identity do?), and what was exposed.

Hints and Cleaning Up

I told you exactly what happened, since this is about testing out those queries — not performing a full incident investigation. We’ll do that next lab, as I walk through the solution step by step.

The biggest hint is to have a private window or another browser open that stays logged into TestAccount1. That will enable you to look at the identity and resources (instance, storage volume, and snapshot) so you have context for your investigation. These are much harder to figure out when you are just looking at the logs.

Once you’re done go back into TestAccount1 to clean things up. Delete the snapshot and the CloudFormation stack. Yep, it’s that simple.

That’s it, and see you soon with a full walkthrough of how I approach this!

-Rich