Prerequisites

• None, but you’ll probably be lost if you haven’t done most of our other labs :)

The Lesson

Way back in our second lab we created our one and only IAM user. After that we immediately moved to using IAM roles, and not too long after to using IAM Identity Center. Heck, the subtitle of the post says:

We are still in the network and workload learning block, but we need to spin back into IAM because these two areas are so closely coupled. As our last string of labs has shown, the vast majority of our instances (and other workloads) will have IAM roles which grant permissions to do things in our AWS accounts. Pretty soon we’ll see how AWS handles that internally and how Instance Profiles and IAM roles work, but to build up to that we need to understand how AWS handles API call authorization.

Hopefully you recall that authorization (shorthanded as AuthN) is the process of asserting an identity. It’s basically the process of logging in. That’s easy enough to see when we log into the console, but what if you use the command line for API calls? What if you want your computer at home, or a server in your corporate data center, to make API calls? You can’t assign it a role because it doesn’t live inside AWS (actually you could these days using AWS IAM Roles Anywhere, but that’s relatively new and it would ruin my narrative if you knew it existed).

Before roles, and loooooong before IAM Roles Anywhere, AWS created Access Keys. 

Death to the AKIA

Access Keys go back to the origins of AWS. In those dark days, we didn’t even have IAM — just the root account. To make API calls you would create an Access Key/Secret Key pair. Think of them as a username and password for API calls, which you could embed in code or save in configuration files. So why not just use a username and password? For many good reasons, but those of you reading probably understand that storing usernames and passwords in plain text configuration files or software code is… bad. Like Stay Puft Marshmallow Man bad. Not that a static access/secret key in plain text is a whole lot better.

More importantly, the way the AWS APIs work is that every request is “signed” with the access/secret. (I am deliberately simplifying to the point of inaccuracy to make these concepts easier to understand… if you want to dig in more, read the presentation A Day in the Life of a Billion Requests). It’s called request signing, and it enabled AWS to support API calls over unencrypted network connections because the calls themselves were encrypted.

An access key for an IAM user looks like AKIAIOSFODNN7EXAMPLE, and a secret key looks like wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY.

Notice those first four characters? AKIA? That’s the starting sequence for an access key associated with an IAM user. If it were a role, it would start with ASIA, as we’ll see soon enough.

Those AKIAs are one of the biggest sources of cloud security breaches. Why? Because the access/secret key combination is just another kind of static credential, embedded in code and configuration files — which makes them super easy to expose or steal! Developers leave access keys in code posted to GitHub (like me!). Admins keep them in configuration files on their laptops, and the latest malware knows exactly where to steal them from. They get embedded into containers, virtual machines, servers in datacenters, and… it’s a mess. Because if an attacker gets these credentials, they can use them from anywhere — with whatever permissions they are assigned.

IAM Users = username and password = access/secret key = static credentials = game over

Despite the many tools available to avoid access keys, they are the easy path for inexperienced devs/admins, and thus still one of the biggest sources of breaches. As an official officer of CloudSLAW, your mission, should you choose to accept it, is to hunt down and destroy every AKIA you can find.

Key Lesson Points

• When you create an IAM user you create static credentials.

• These are either a username and password for the web console, or an access key and secret key for API calls.

• Access keys are often embedded in code and configuration files to allow applications and systems to make API calls.

• These static credentials are often accidentally exposed, are targeted by attackers, and are one of the two largest sources of AWS security incidents.

The Lab

This lab has two parts. First create your own AKIA to see the process, and then delete it. For the second part of the lab you will launch a CloudFormation template which creates a new VPC with an instance. You will log into the instance and use the command line to abuse an access key I embedded in the image!

Am I insane?

I mean, clearly, but not for this. That access key will be for an IAM user which can only do one thing, and cannot cause me any harm.

We also need to do a smidge of cleanup for those following these labs in order.

Video Walkthrough

Step-by-Step

Once again, go to your Sign-In Portal > TestAccount1 > Administrator Access > CloudFormation. For our first step we’ll delete the stack deployed in the Let’s Get Hacked lab. Its run rate is relatively low if you want to leave it, but assuming you saw your GuardDuty alerts in email (via Security Hub) now is a good time to clean it out.

Select Exposed > Delete:

Now let’s launch our stack for this week. Click Create stack > With new resources and use the following settings (no screenshots — you got this):

• S3 template URL: https://cloudslaw.s3.us-west-2.amazonaws.com/lab37.template

• Name: AKIA

• Click through, click the usual blue checkbox saying IAM resources will be created, and Submit.

Okay — no need to wait for it to finish deploying. Cruising along here, we need to make a change to Session Manager and disable logging. Why? Because I’ve been having you delete things for cost savings as we go, and your S3 bucket for Session Manager logs isn’t around anymore and would break the lab (ask me how I know).

Go to Systems Manager > Session Manager > Preferences > Edit and disable Send session logs to S3:

Now we can get to work!

Be Dumb and Create an Access Key

To start, I will teach you how to shoot yourself in the foot! Uh, what? Yep. You will create an IAM user with an access key to understand the process and what it looks like. Then you’ll delete the user (and with it, the key). Don’t worry, you’ll get to (sorta) hack me soon enough.

Go to IAM > Users > Create user. Name the user delete and click Next:

Here’s the trick to keep you safe. On the Set permissions page, select Attach policies directly and then… don’t attach anything. Just scroll down and Create user. This user has no password and thus no console access. It has no policies and thus no permissions. It is a ghost, powerless to harm the living.

Notice how that flow doesn’t even give you the option to create an access key? AWS will do its best to stop you from shooting yourself in the foot (admittedly only in the console — there are no warnings via API, which is where most developers operate). We need to go in and create it. Click your deleteme user > Security credentials > Create access key: 

Okay, notice all those nice radio buttons to pick the kind of access key you need? It’s a lie — there’s only one kind of access key!!! These are all warnings to steer you away from using access keys for an IAM user. It’s almost like… AWS knows how most customers are compromised?

Click a few to see how they steer you to better alternatives, then channel your internal inexperienced developer and click Other > Next. Skip the tag and Create access key:

This is your only chance to see the secret, which is basically a password. Notice that AKIA? Sear it into your brain, because down the road if you see that in logs, the bile should rise in your throat. Access keys are static, they never change, and they last forever.

Alright, I feel gross just showing you all this — let’s delete that user.

Time to switch gears and have you poke around my account while learning how access keys are typically handled.

Use an Access Key to Explore My Account

Now the fun part. We already set everything up with our CloudFormation template, so let’s log into the instance. Go to EC2 > check the AKIA instance > Connect > Session Manager > Connect. Once you’re logged in it will look like this:

We need to use a few command lines. Since you are not all familiar with Unix, I’ll explain what’s going on as we go. I strongly recommend you copy and paste these.

First we need to Change Directories to the SSM user’s home directory. “~” is shorthand for the home directory of the logged-in user:

cd ~

Then type ls -a which means “list everything in this directory, including hidden files”. My screenshot shows that a simple ls doesn’t show what we need:

ls -a

Notice that .aws directory? That’s where a user keeps configuration for AWS tools.

Let’s go into that directory, see what files are there, and see what’s inside. First I’ll show the commands for copy/paste and the screenshots, and then I’ll explain what we are looking at. The cat command displays a file’s contents:

cd .aws
ls
cat config
cat credentials

Here’s what you see:

• The config file specifies parameters including what region to default to. It uses something called a profile. Profiles in these configuration files enable you to keep a bunch of different users (or roles) on your system, so you can switch between them like we’ve been doing in the AWS console. Profiles are just names between brackets like [SLAW], and then the different settings on individual lines below the name.

• The credentials file contains credentials! They can also be associated with a profile — in our case [SLAW]. You’ll see how this works with the next command line.

• That access and secret key? They live in one of my accounts. I have an IAM user sitting there for this exercise.

These two files in this .aws directory are the way static credentials and settings are stored for the command line and other code-level tools. In this case there is a single profile [SLAW], with a static access/secret. You can also use these with roles and rotating session credentials, but that isn’t our focus today (and much lower risk).

At this point, if you were an attacker, you would extract the credentials and start looking around to see what permissions they have. I’ll make it easy — just try this command:

aws ec2 describe-image-attribute --image-id ami-09fee31bea43efcec --attribute description --profile SLAW

This says “use the aws command, use the describe-image-attribute API call on image ID ami-XXX, and get the attribute description, using credentials in the profile SLAW. That’s the trick — the --profile option tells the command which credentials to use.

I am… not going to spoil the surprise. But let’s just say you are looking at a private resource in my account, which is definitely not shared anyplace else.

All done? Now try the same command without that profile. This will fall back to the IAM role assigned to the instance.

aws ec2 describe-image-attribute --image-id ami-09fee31bea43efcec --attribute description

Notice anything different?

Cleanup: delete the AKIA CloudFormation stack!

Instead of key lab points, I think a narrative summary is better this week. This is one of the most important labs you’ve completed.

What you just saw is a common, real way access keys become exposed. Someone puts credentials in code or a file, then creates an image. Or, thinking back to last week’s lab, they have credentials in an instance which is compromised because it was made public.

This. Happens. Every. Day.

There are many other ways AKIAs get exposed. We included examples of real-world incidents in the Universal Cloud Threat Model I wrote with Chris Farris.

IAM users are dangerous. They can last forever, the credentials are static, and access/secret keys are way too easy to expose.

Does that mean I never use them? No, I still have some, but that’s mostly of how long I’ve been using AWS. Some of the newer options — such as using IAM Identity Center for command line/code and AWS IAM Roles Anywhere — are relatively new, and not all my accounts are set up to support them. But I promise you we don’t have any in my production accounts at work, and I’m well through the process of reducing my need for them.

Hopefully fewer of you will have holes in your feet thanks to this post.

-Rich