Prerequisites

• Your S3 bucket from Part 1

• The correct bucket policy on the bucket from Part 2

The Lesson

Just a quick note: with labs like these it is not uncommon to have small errors, especially in policies where we are cutting and pasting a lot. If you get blocked and can’t figure it out you can always email me at [email protected]. It may take me a bit to get back to you, but I’ll do my best.

With all the setup complete, this week it’s time to learn a little about Organization CloudTrails and finally flip the on switch. I might even be able to constrain my verbosity and keep this post fairly short, since we have already done most of the hard work.

One of my favorite parts of AWS Organizations is the Organization CloudTrail. To understand why, let’s break down how I think CloudTrail works (any of you AWS friends, please correct me if I get his wrong).

If you recall, regions in AWS are designed as isolated geographic centers to manage resiliency, latency (you can build closer to your customers), and compliance (there are a lot of global regulations that restrict where you can store someone’s data). AWS takes this very seriously, and regions are very well isolated, even though they share a common management console. But as you’ve seen, what we are really doing in the console is switching to different regions, and behind the scenes they are running entirely independent API endpoints. The only exception is the few global services such as IAM.

Now let’s look at CloudTrail. CloudTrail’s job is to log all the API calls. Since they happen in different regions, collection must occur in the region where the activity takes place. When you look at the Event History in the console that’s all you see: activity in the current region. AWS is always collecting this activity and keeping 90 days for their buffer in that region (I’ll call it a buffer if I want, even if that isn’t exactly how the internals work).

When we create a Trail, what we are really doing is telling AWS to store that data someplace other than Amazon’s internal buffer, and we get to set some additional options like what kind of data we want to keep and where to keep it. In our case we kept it simple and started with just a trail for the management account and “management events”, which are all the API calls to the management plane (as opposed to data events, which are read/write API calls to resources like S3 buckets).

When we enabled our Trail for all regions, what that did was set a consistent configuration in all regions, even though each region actually runs its own trail. Handily, AWS will push this out to any new regions enabled. Each Trail is independent, but running a consistent configuration and saving to the same destination (S3 in our case).

An Organization trail is the same thing, but BIGGER AND BETTER! It:

• Creates a trail in every account and region in the Org, all using the same configuration.

• Centralizes all the logs across accounts and regions into one S3 bucket.

• Handles all the paths and naming for that bucket so you can figure out which account and region a log came from (it uses the account ID and region in the log path).

• And the BEST PART! Accounts can’t turn off or change their CloudTrail. Why? Management of the trail lives in the management account, and the other accounts don’t have permissions to make any changes. This means that even if an attacker gets the root user account they can’t turn off the CloudTrail.

  • But don’t let them get the root account. Because with that they could disconnect from the organization if you don’t have SCPs in place to block that. Oh, but we do have them for these labs! Aren’t we smart?

• You can store the logs outside the management account. This is very important because it reduces anyone needing to access the management account to read those files. We want to eliminate as much access to the management account as possible because it’s so powerful.

• Any new accounts will automagically be configured the same.

We are dumping our logs into the SecurityAudit account (for now — I’m really into foreshadowing and stuff, and some of you might notice a little naming problem here). Here’s a visual, because I hear images help SEO:

Now a big warning: some of you probably work in environments where the size and scale of the organization trail is … a lot. In future labs we will cover how to handle massive amounts of logs, which would break some of the things I’ll teach you during these fundamentals.

Lesson Key Points

• An Organization CloudTrail creates a trail for every account and region in your organization.

• You can designate an S3 bucket in any other account to store the logs.

• New accounts and regions automatically log to this destination.

• Within an account you can’t disable or delete the organization trail, which really pisses off attackers.

The Lab

UPDATE: Some people run into an error. This is a problem with the AWS console and I’m reporting the bug. The instructions for the fix are in the post.

This one is pretty simple: we’ll grab our S3 bucket info and then, with just a few clicks, enable the Organization Trail and redirect it to the bucket in our SecurityAudit account. After that you’ll get the chance to explore CloudTrail a bit more. We won’t go deep into that yet, since we will cover the log format and content in more depth relatively soon.

Video Walkthrough

Step-by-Step

First, go to https://console.aws.amazon.com and sign in. Then Switch Role from the upper right menu to your SecurityAudit account. (Go back to Part 1 if you need a refresher). If this is the same browser you used last week it should pop right up.

Then in SecurityAudit, go to S3 and copy the name of your bucket:

Now Switch back to your management account:

You’ll be in S3 and see your old CloudTrail bucket in the management account, but ignore it and go to CloudTrail. Then click Trails and default.

You’ll see the details for your trail. If you want, Switch role back to SecurityAudit and notice that there isn’t a trail in that account yet. Then switch back. Or you can just believe me. And if you just believe me, I promise to keep that Bitcoin safe for you, just send it over to my wallet. Really. I promise.

Huh. Kind of figured at least one of you would fall for that. Oh well, back to the grind. From here, click Edit.

Now make 2 changes:

• Check Enable for all accounts in my organization.

• Paste in your new S3 bucket name.

• Then Save changes.

WARNING: IF YOU GET AN ERROR!!!!

I had to update this post because AWS has a bug in the console. Follow these steps if you hit it.

The Problem

If you see this, you’ve found the error:

We used to have to enable these services manually but AWS changed that and now the services are enabled via the console of that particular service. It looks like the CloudTrail console is failing if you go to change a trail to an org trail vs. creating an org trail from the start. I have gone through all the documentation and we should NOT have seen this, and not everyone does.

The Fix

Go to Organizations and then Services:

Then scroll down to CloudTrail and click on it. Then ignore the warnings and follow the process to enable.

Now go back and repeat the steps in the main lab to enable the trail. I’m adding these instructions to the lab on the web and in future emails, and will remove them when AWS fixes the bug.

Congratulations, it was just that easy.

If you have time you can poke around a bit to better internalize how this all works. For example if you switch back to your SecurityAudit account, you can see that the trail was created in that account, but with a warning that you can’t modify it.

You can also hop into S3 and explore the object hierarchy CloudTrail creates to organize the files:

Or poke around the Event history in the different accounts and see how they differ (just Switch role, and the console keeps you in whatever service you are using).

And that’s a wrap for this week.

-Rich