Prerequisites:

• Your admin IAM user credentials

The Lesson

My dogs love bacon, but not nearly as much as we security pros love our logs.

The thing is, and I’m sure this will shock you, but logs in cloud aren’t exactly the same as the logs we are used to dealing with. I like to summarize the differences as:

• Different log sources, requiring different collection paths

• In cloud, it’s critical to understand the speeds of your feeds

• “Events” are a critical data source, and must be treated differently than saved log files

As with my intro to IAM, this is just the first little sip of our work in security telemetry. That’s right, I don’t even refer to them as logs — they’re telemetry, and logs are only one type.

Today we will focus on the single most important telemetry source for any cloud platform: management plane activity. In AWS this is called CloudTrail, in Azure it’s the Activity Log (but be careful, there are a few), and in GCP they are Cloud Audit Logs. These feeds ideally provide telemetry for every API call in our deployments. Unfortunately they don’t always quite get there. They capture most activity, and in the case of AWS it’s usually robust, but not all activity.

A key point here: cloud management plane logs are cloud API call logs. They capture the API calls used by internal systems and customers.

There can be gaps for several reasons. I’ve narrowed it down to three primary types:

• AWS/Azure/GCP/etc. aren’t single monolithic platforms — they are collections of hundreds of services, run by many dozens of teams. Many of these services have their own API endpoints (the things we connect to in order to take actions, even if the web interface hides that) and they all have to instrument their endpoints, figure out what activity to collect, then normalize and send it in to the log/audit/etc. service. There are differences in quality and structure of data.

• Every provider has undocumented APIs, usually for backing their web interface frontends. Not all of these will necessarily be logged the same, if at all.

• Microsoft doesn’t think you need to see Read activity in your logs. I have no idea why. So… have fun trying to detect recon.

I’m old enough to remember the years back when we didn’t have any management plane telemetry. There was no way at all to know what was going on in our accounts. Also, we lacked indoor plumbing. The thing is, once we got this telemetry, it provided a massive security advantage over what we have in datacenters. Think about it: one central source for nearly all administrative activity. It’s insanely powerful and useful.

I’ll skip over more history, but before we dive into CloudTrail there are a few more important points to understand:

• Management plane activity is only one source of cloud security telemetry, and there are many more sources to discuss later.

• In nearly every service this telemetry can be managed as logs, events, or both. Events are something we will — you guessed it — spend a lot more time on later.

• As I already mentioned, these only capture API activity from the management plane. Individual services also have their own telemetry sources, some of which are important to security. This isn’t really all that different than logs in a data center; while you might have your central authentication logs for SSO… that doesn’t track when an administrator logs into a particular server using an SSH key. Resource and service telemetry is not the same as management plane telemetry.

Today’s key points are:

• The primary security telemetry source to collect in cloud is management plane activity.

• Management plan activity typically includes most API calls made to the cloud provider’s … management plane.

• This activity may be available as events, logs, or both. (Events are a real-time stream of activity… more later).

• Management plane activity, while a single telemetry feed, isn’t perfect and has some gaps. These gaps are usually edge cases, especially in AWS, which should have minimal impact.

• Every provider is different.

• This telemetry source is a security dream — nearly all administrative activity in the ‘datacenter’.

There are a few really important things to know about CloudTrail, which we will cover in the lab.

The Lab

Today we will configure CloudTrail, which records nearly all management plane API activity in your account. There are some really cool things about CloudTrail, as well as gotchas to be aware of. I’ll explain some of this as we go through the process of turning it on, but CloudTrail is so deep we will cover the bulk in future labs.

Here are the most important things to know:

• CloudTrail is always running in your account. It keeps 90 days of activity which is accessible via API or the Event History in the console. This is not enough for security, so our lab will show how to save telemetry to files. CloudTrail does not save to files unless you enable a ‘trail’.

• CloudTrail runs in every account and region in AWS. But it isn’t a central feed; technically you need to turn it on for every account and region, and they are all different trails. AWS regions are highly isolated so this makes sense.

• To make life easier, AWS allows you to set up an all-regions trail. This is great because you can turn it on in one place in the console for all your regions (and even accounts, but we’ll get there). This doesn’t create one trail — it technically creates a new trail in every region, but saves the telemetry to one central location. Think of it as 20 collectors pushing centrally, and the “all regions” trail turns on the collectors. This is the best way to ensure you get all the logs, and it is the default.

• There are Management Events and Data Events. Management Events is all we will use for now, and includes core API activity. Data Events tracks access to certain resources, like S3 buckets, but…

• … you get one Management Events trail for free. Data Events always incur additional cost.

• When you turn on a trail you are saving logs to an S3 bucket which is private and encrypted by default. You can also use your own encryption key, but we will skip that.

• While the trail is free, the S3 bucket isn’t. This won’t be an issue for most of you, but if you get to really large scale that can add up, and managing storage costs becomes important. Don’t worry — it’s still cheaper than your SIEM.

• CloudTrail also creates Events, that real-time event stream. We will do a lot of cool things with Events down the road.

• As we progress we will show you how to ensure that CloudTrail is always enabled for every account and region.

I could go on and on and on, but these are the important points for now. Let’s jump in, enable CloudTrail, and configure a few important settings.

Video Walkthrough

Step-by-Step

First go to https://console.aws.amazon.com and sign in. Then double check that you are in the us-west-2 (Oregon) region by clicking in the upper right hand corner:

Then click in the search box, type cloudtrail, and click CloudTrail. (In a couple labs I’ll just start telling you which service to go to, skipping over how to get there).

You might need to click the ‘hamburger’ menu icon to see options along the left:

CloudTrail is always running in your account, even before you enable it, but initially it only records internally, and does not save log entries anywhere you can read them. Also, CloudTrail is regional! You can tell by clicking on Event History and… seeing nothing. Why? Because we are in us-west-2 and all IAM for AWS runs out of us-east-1 (Virginia).

Switch regions and you’ll see all your login activity and the IAM lab from last week (assuming you are following along in order):

NOW SWITCH BACK TO US-WEST-2!!!!!

From here click Trails and then Create trail:

First set the name. The default works:

This will start saving CloudTrail logs to S3, and the default bucket is totally fine. But if you scroll down a little, you’ll see one setting we want to change.

By default all data in S3 is encrypted. When we get to S3 labs you’ll learn more than enough about S3 encryption to make you unpopular at parties. The option on this page allows you to set and use your own private key. This will increase costs and, IMHO, doesn’t really add much security. It also adds friction to handling log files. So let’s turn it off (and if you disagree, please drop me a line or comment on the blog or YouTube).

Leave the rest at defaults. I like to keep Log file validation: Enabled, even though I’ve never needed it. It’s important if you ever have to do legal forensics. Then click Next.

On the next page just accept the defaults, but although we will cover them more later, it is important to understand the different options here.

• Management events are all the management plane API calls. This is admin stuff.

• Data events are only supported on some services, and track people (technically identities) making API calls to read or write things.

• Insights events are a thing you pay more for, to have AWS perform magic to identify unusual activity. I won’t say whether this is an “eldritch terror” kind of magic or “automated reasoning”, but let’s not take the risk today (we will use them later).

If you change those options you can configure some other settings which, you guessed it, we will cover later. Click Next.

On the next screen you can review settings. They should default to a multi-region trail, but please double-check that (and everything else).

Then click Create trail.

If you see this, you won!

We will return to CloudTrail soon enough. It will be a core tool through these labs. For those of you with experience, we will update this later to be an Organizations trail, so don’t send hate mail yet.

Thanks, and have a great day!

-Rich