Prerequisites

• An AWS Organization with a SecurityAudit account and delegated administration for running StackSets.

The Lesson

You know how I said we’d build all this security stuff in pretty much the same order as I would do it for real?

Yeah, I kinda blew it.

This is a lab we should have done quite a while back, but there are a lot of little logistical things that kept putting it off. But today is our day! (How’s that for motivational speaking?)

Today we finally get around to starting our setup to deploy a Cloud Security Posture Management (CSPM) tool. In real life this is something I try to do on day 1 when I walk into a new org, but these are the sort of tools that make a lot more sense after you have a strong cloud security knowledge foundation.

So congratulations, young padawans. You are ready for the next big step on your journey to cloud security Jedi status.

(We may have had a little coffee issue today and perhaps my brain is… being weird).

Cloud Security Posture Management

I’ve mentioned CSPM a few times in previous posts. You saw hints of it when we enabled Security Hub (but not the actual CSPM part), and when I talked about it as an essential element of cloud incident response.

The TL;DR is that CSPM tools connect to your cloud accounts via API, run a crap ton of read API calls (e.g. Describe/List/Get) and try to identify misconfigurations which could lead to security and compliance issues. Modern tools can do a lot more, but they still all rely on this core capability.

There was a time when I was one of a very small group of people performing cloud security assessments. Many of us had to write our own tools since there were no features in the cloud platforms (heck, Azure didn’t even exist) and trying everything manually via the CLI or console wasn’t scalable. Netflix was the first to talk about building their own internal tool (Security Monkey) and, long story short, eventually commercial tools flooded the market (including mine — we’ll get there).

What’s a misconfiguration you ask? It can be as simple as a public S3 bucket or port 22 exposed in a security group, or as complex as a misconfigured role trust policy that could lead to account takeover.

These aren’t traditional vulnerabilities per se, because we aren’t talking about an unpatched flaw an attacker can exploit. They are more like "unforced errors” that can potentially be exploited by an adversary in an attack. Why am I wording this so weirdly? It isn’t my coffee issue (broken grinder, fixed now), but rather that every misconfiguration is something that technically has a valid purpose or the capability wouldn’t be in the platform in the first place.

We’ll talk a lot more about CSPM over the next few labs — including CNAPP, CSPM’s younger, bigger cousin.

Stop Confused Deputies with External IDs

To explore CSPM we will use some external tools (Prowler Open Source and FireMon Cloud Defense). Prowler will run inside one of our accounts, but FireMon is a SaaS platform out there… on the Internet someplace.

This creates an interesting situation. We will allow an outside account (via a role) access inside our Organization… and that account will have a lot of power.

But that outside account? It’s a multi-tenant application connecting to a lot of different customers. And the normal pattern for these is for the SaaS platform to use the same IAM role to connect to all their customers. This creates an interesting dilemma: what if one of those customers figures out how to trick the platform into doing something in our account instead of their account? They… confuse… the application (the “deputy”) into accessing the wrong account.

This is known as the confused deputy problem, and it isn’t unique to AWS. Heck, it’s well known that compromising a security tool is one of the best ways to get broad access to a target.

Okay, let’s break down how this works:

• Security Tool A in its own AWS account has access to Bob’s customer account via the SecToolVendor role, which has permission to assume the SecToolCustomer role in Bob’s account.

• Alice uses the same security tool, which uses the SecToolVendor role to assume access to her accounts.

• Alice (why is it always Alice?) tells Security Tool A that they own the account with ID of Bob’s account.

• Since Bob already trusts the SecToolVendor role, if the tool is tricked by Alice and assumes into Bob’s account, the tool is now doing whatever Alice tells it to, to Bob’s account.

AWS came up with a good way to reduce this risk. The External ID is a shared secret we build into our role trust policy. The External ID has to be provided during the AssumeRole API call, and must match. It’s a value in the SecToolCustomer role trust policy. Thus…

• Alice tries to add Bob’s account, since she knows they both use Security Tool A.

• When Alice tries to AssumeRole into Bob’s account, Security Tool A uses her External ID in the AssumeRole call (the vendors keep these in a database or secure configuration someplace on their side).

• That External ID doesn’t match Bob’s external ID in the SecToolCustomer role, so it fails.

But if Alice can trick Security Tool A into using Bob’s External ID, this falls apart. Or if Security Tool A shares its external ID across customers, well, no one would be that stupid. (Narrator: “they were that stupid”). (Thanks to Kesten Broughton for that excellent research and fwd:cloudsec presentation).

Here’s our old AssumeRole diagram, updated with the External ID:

Okay, enough background for today. This is the first lab set where we will deploy third-party tools into our, organization so it’s a good time to play with External IDs. Then, CSPM is so central to cloud security that we will explore multiple options over a series of labs:

• Preparing our organization (today’s lab)

• Open source (using Prowler)

• Commercial (using FireMon Cloud Defense’s free trial)

• Security Hub

You’ll see different deployment options, user interfaces, and capabilities. We’ll also see how to use CSPM for incident response, not just as a “vulnerability scanner”. As always, the labs are configured to be as inexpensive as possible.

Key Lesson Points

• Cloud Security Posture Management is a foundational cloud security tool.

• CSPM identifies cloud configurations that could lead to security issues.

• A common pattern to access our accounts is for a multi-tenant, external third-party tool to assume a role inside our accounts.

• This could enable another user of the tool to trick their way into our accounts. This is called the confused deputy problem.

• An External ID — a separate, shared secret — reduces the risk of a confused deputy.

The Lab

Today is the boring setup lab. The main learning objective is how to configure a cross-account role with an External ID. Although we don’t technically need the External ID, since everything is running inside our organization, this is an important skill to learn for connecting external tools.

Our tool of the day is Prowler — an Open Source (and also commercial) CSPM. We will go into more details on CSPM and Prowler in the next lab!

But! We also need to configure a hodgepodge of other settings to make our CSPM labs run smoothly:

• Change our session timeout so we can stay logged in longer.

• Deploy a CloudFormation template which creates:

  • A role for Prowler to use

  • A VPC for Prowler to run in

  • An instance with Prowler running

• Use a StackSet to deploy the role our tool will assume when it assesses accounts.

Video Walkthrough

Step-by-Step

As I was building this lab I got super frustrated with session timeouts. The lab steps themselves are pretty quick, but running Prowler can take some time, and definitely may blast past our default of an hour. Let’s expand that to 4 hours. This is some pretty simple ClickOps so I’ll just blast you through it with a couple screenshots.

Start in your Sign-in portal > CloudSLAW > AdministratorAccess > IAM Identity Center (like, is there a different Identity Center?) > Permission Sets > Administrator Access > Edit > Session duration > set 4 hours > Save changes > Close the tab.

It won’t kill you if you skip that, but… you’ll regret it.

Let’s go deploy the Prowler template. Head over to SecurityAudit > AdministratorAccess > CloudFormation > Create stack (with new resources): 

Use the template https://cloudslaw.s3-us-west-2.amazonaws.com/lab65-1.template

Name it “prowler” and set your External ID. This needs to be 8-64 characters, and you need to write it down or remember it. The External ID isn’t stored in IAM or with the role — instead it is passed in as User Data with a bash script that saves it to a file in the ec2-user home directory. This image includes some scripts which launch Prowler and pass it the External ID from the saved configuration file.

Click through the next couple screens and Submit. Then wait for it to finish, since we need the IAM Role that Prowler will use to be created before we try to deploy our cross-account role which trusts it. (You can’t create the trust relationship until the role is created — AWS will error out on a role ARN in a trust policy if the role doesn’t exist.)

Now Close the tab > Sign-in portal > SecurityOperations > AdministratorAccess > CloudFormation > StackSets > Service-managed > Create StackSet. It’s really darn important to swap into SecurityOperations, since that’s the account with delegated administration permission to deploy stacks. SecurityAudit does not have any permissions to make changes in other accounts.

This time use the template https://cloudslaw.s3-us-west-2.amazonaws.com/lab65-2.template

Then name: prowler-role, set the SAME EXTERNAL ID!!!! and grab the SecurityAudit account ID from your Sign-in Portal tab. It will look like this…

This ExternalId IS embedded into the role trust policy! It looks like this:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::533267272350:role/ProwlerLabInstanceRole"
            },
            "Action": "sts:AssumeRole",
            "Condition": {
                "StringEquals": {
                    "sts:ExternalId": "ProwlerLabExternalId456"
                }
            }
        }
    ]
}

You’ve done this more than a few times before, so the only other settings that need to change are:

• Deploy to organization

• Only the us-east-1 (Virginia) region, since this is an IAM role which will end up there anyway (if you try to deploy to multiple regions it will fail, since the role will already exist).

• Under Deployment options I recommend setting the max concurrent accounts to 10, and the failure tolerance at 9.

Then Submit.

No need to wait for this one to deploy. Close the tab > Sign-in portal > SecurityAudit > AdministratorAccess > Systems Manager > Session Manager > Preferences > Edit:

We will extend our idle session timeout for Session Manager. It won’t matter this week, but when we run Prowler scans next week, they won’t finish within the default of 20 minutes, and if the session closes you lose your scan. Max it out to 60 minutes:

Now let’s connect to our Prowler instance. Just so you know what that template did, it:

• Created a role for Prowler with a bunch of Read permissions. These are the default policies for read only and view only (don’t ask), and some extended permissions Prowler recommends to read settings that aren’t included in those two policies.

• Created a VPC with only a public subnet and a security group with no inbound access. Why did I use a public subnet? To save money — this instance will be running for multiple labs so if we used a NAT gateway it would cost a lot, and if we wanted to use service endpoints we would need them for every service Prowler uses to scan. We’ll rely on our locked-down security group.

• An S3 bucket to save Prowler results. Prowler has a pretty great Web UI, but then we would need to deal with opening up our security group/ports. You can do that on your own if you want, but I decided dumping reports to S3 was easiest when I have 5,000+ of you running your own labs in your own accounts.

• An EC2 instance of an image I created with Prowler pre-installed and some bash scripts to run it in different ways, using our External ID if needed.

Okay, go to EC2 > check the Prowler instance > Connect > Session Manager > Connect:

Now run the following command lines to test it:

• sudo su - ec2-user # switch to be ec2-user, in that home directory

• bash quick-scan.sh # run a quick Prowler scan and save the results to S3

That should execute within a couple minutes, and look like this when done:

This scan ran just on the local account and didn’t use our External ID — we’ll get to that next week. If you want a little side quest, you can check out the User Data by going to Instance settings > Edit user data.

Before we look at our results, go back to your instance and Instance state > Stop instance to save a little moolah.

Okay, let’s look at our results! Go to S3 > prowler-reports-<your account ID> and follow the directory tree until you reach the reports. Click the HTML report:

Then click Open and it will pop out the report in a new tab:

And there you go! These are just quick results — in the next lab we will run a deeper scan on a target with… some problems.

Lab Key Points

• External ID is a shared secret to prevent the confused deputy problem.

  • It’s a condition in the trust policy of a role being assumed.

  • The assuming party (e.g., your SaaS vendor) only provides it during AssumeRole, and it should be different for each customer.

• Prowler is a CSPM tool for identifying misconfigurations.

-Rich