Prerequisites

• All you need are admin credentials in an organization with the OrganizationAccountAccessRole available.

The Lesson

It didn’t fit well into the lesson, but during the lab we’ll be using the command line to explore more like a hacker trying to knock down annoying walls.

I’ve been teaching cloud security for something like 15 years now, and there is nothing I fear more than walking into a classroom, staring at a sea of personal and work laptops, and knowing I need to figure out how to get every one of them to run an SSH terminal so we can use class scripts and commands.

DO YOU POSSIBLY UNDERSTAND THE NIGHTMARE OF FIGURING OUT HOW TO MAKE AN SSH CONNECTION ON A WINDOWS CORPORATE LAPTOP?!?! DO YOU?!?!?!

I… have…scars. And, if you recall our prior use of Session Manager (from, like, a bunch of previous labs) we now have a great way to connect to instances using the web console without ever needing a local terminal program.

While we’ve made those connections to learn skills like managing credentials, we haven’t really used the AWS command line much. Nearly all our work has been using the GUI that is the AWS console — which we fondly call ClickOps because… we click all the things. We have also used a ton of Infrastructure as Code (IaC, via CloudFormation) which is closer to how you want to manage applications and infrastructure stacks.

As your cloud skills mature, you’ll also find yourself using the AWS command line more. Personally I don’t use it a ton — I weirdly tend to swap around between the console, IaC, and Python scripts for work. When I automate I use Python, but there are a lot of you out there who automate primarily with bash or even PowerShell.

Using the command line, and understanding its security implications, is a valuable skill I probably should have included earlier. But hey, better late than never! And it isn’t like most of you are paying for this!

One option is to use Session Manager and connect to an instance, and run commands from there. Although we’ve used this a few times, it still requires a running instance to connect to. Sometimes we just want to use some AWS command lines to manage our account, and don’t necessarily want or need a persistent image. This is where AWS CloudShell comes into play

On CloudShell, the CLI, and Credentials

CloudShell is a terminal which runs right in the AWS console. I don’t know what it is under the hood (okay, poking around, it’s a container), but when you launch it AWS creates a compute environment, connects 1GB of persistent storage, and pre-authenticates using your existing credentials. Just click the little icon, and your shell opens right up:

That environment includes the latest version of the AWS CLI, options for bash or PowerShell, and 1GB of storage, conveniently encrypted with KMS and backed up for you.

What about credentials? Those are loaded up similar to the way an instance gets an IAM role; we will dig in during the lab. Those credentials also use normal AWS credentials precedence. What’s that? Yet another thing I should probably have covered earlier. Whenever you run a tool, like the AWS CLI or an SDK, it pulls credentials in a particular priority order (unless some a-hole programmer decided to handle it some other way, in which case you do not need their piece of crap tool):

1. Direct credentials submitted on the command line or in code as parameters.

2. Environment variables, which we will use today.

3. The AWS credentials file located at ~/.aws/credentials (mature operating systems) or %USERPROFILE\.aws\credentials (Windows. Ugh).

4. The AWS config file, in those same locations.

5. Container or instance profile credentials (IAM roles) or CloudShell credentials (which are basically in a metadata service).

The CLI/SDK just checks each spot until it finds credentials, and uses the first set it finds. If you use a defined profile, it will use the first matching profile it finds, again searching in that order.

CloudShell is great because it never saves your credentials, they aren’t stored where all the nasty malware looks for them, and even if you store (other) credentials yourself they are encrypted. You also can’t SSH or connect to CloudShell from outside the console! It’s a browser-based shell. Unlike your computer, an adversary can’t pop a shell with malware and use your credentials while you’re sleeping.

And for those of you in enterprise environments, you can restrict CloudShell to a VPC. This means it is constrained by all your network security controls (and an attacker can’t just download their attack tools), but it also provides access to your service endpoints (including PrivateLink) in case you need to do network connectivity testing without having to drop an instance into the VPC.

Some pre-lab CLI notes

You’ll see the CLI is pretty straightforward to use (here’s the reference). You type aws <service name> --<request parameters>. You can override the persona/identity (by specifying different credentials or choosing a profile) and the region by setting those as parameters. We’ll do both today so you get to see it.

Here’s an example for running an instance:

aws ec2 run-instances \
  --image-id ami-0123456789example \
  --count 1 \
  --instance-type t2.micro \
  --key-name MyKeyPair \
  --security-group-ids sg-0123456789example \
  --subnet-id subnet-0123456789example \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=MyBasicInstance}]'

Here are my general rules of thumb for how I personally use the options:

• To try new things, or quickly enable/disable/build, I usually start in the console. The visual nature helps me wire the process into my head.

  • Sometimes I use the CLI instead. It’s great for quickly checking the full results of commands, which the console sometimes hides or tucks away in a corner.

  • The CLI is awesome for troubleshooting.

• When I need to create something repeatable, or manage the production version of what I’m building, I use IaC (CloudFormation or Terraform).

• For automation I use Python. Plenty of people use the CLI instead; it’s a personal preference.

• If I want to play with some other command line tool and don’t want to install it locally, I might use CloudShell.

CloudShell is great. Easy, secure, and a wonderful gateway drug to get out of the console and onto the command line.

Key Lesson Points

• CloudShell is a terminal which runs in your browser.

• Behind the scenes it’s a container with persistent storage.

• It’s important to understand credential preference order when using the command line and SDKs.

The Lab

I knew I wanted to do a CloudShell and CLI lab but it took me a while to figure out something interesting with a security message. Don’t worry, I cracked it!

Today we’ll launch a CloudShell in our management account. We’ll pull our credentials from the customized metadata service, then AssumeRole into one of our other accounts, and play with the CLI a bit. We’ll also download a file and peek into CloudTrail to see what CloudShell events look like, in case you ever want to build some threat detectors around it.

Video Walkthrough

Step-by-Step

Start in your Sign-in portal > Copy the Account ID for TestAccount1 > Then CloudSLAW > AdministratorAccess.

Then make sure you are in the Oregon region > launch CloudShell, which is located up on the AWS Console menu bar:

Double-check we are logged in using our role:

aws sts get-caller-identity

Okay, let’s try to grab credentials from the environment variables:

echo $AWS_ACCESS_KEY_ID

Huh. Well that didn’t work. Let’s see all the environment variables:

env

Oh, interesting! There’s a lot of cool information in there (like, this is how I know CloudShell is a container… see if you can spot it).

What’s that one that says “AWS_CONTAINER_CREDENTIALS_FULL_URI=http://127.0.0.1:1338/latest/meta-data/container/security-credentials”? That looks a lot like the metadata service, but instead of being at 169.254.129.254, it’s at localhost. (And is it on port 1338 because it’s one better than 1337?). In case you were wondering, this is not documented anywhere. It’s almost as if AWS doesn’t think you need to use these credentials (you don’t).

Well, I see an endpoint, and maybe it works like the metadata service, and maybe I can pull credentials? Hat tip to Christophe Tafani-Dereeper for doing all the hard work a few years ago (and, uh, I was a bit surprised he gave me some credit). Let’s follow the process we use with v2 of the instance metadata service (remember, this is a two-step process):

TOKEN=$(curl -XPUT localhost:1338/latest/api/token -H "X-aws-ec2-metadata-token-ttl-seconds: 60")

curl localhost:1338/latest/meta-data/container/security-credentials -H "X-aws-ec2-metadata-token: $TOKEN"

Whoo hoo, we cracked the Gibson! Well, the… something. 

Okay, let’s shift gears and start using the CLI. To start, let’s assume the OrganizationAccountAccessRole into TestAccount1. This works because we are logged into CloudShell as an administrator in our management account. I’m putting this as one big code block, so you can copy and paste the entire thing — you just need to paste in your own account ID and review the warning AWS may show (it’s a browser thing):

# Replace ACCOUNT_ID with the target member account ID
ACCOUNT_ID=<YOUR ACCOUNT ID>

# Assume the role and store the credentials in a variable
credentials=$(aws sts assume-role \
  --role-arn arn:aws:iam::$ACCOUNT_ID:role/OrganizationAccountAccessRole \
  --role-session-name cloudshell-org-access)

# Extract and export the temporary credentials
export AWS_ACCESS_KEY_ID=$(echo $credentials | jq -r '.Credentials.AccessKeyId')
export AWS_SECRET_ACCESS_KEY=$(echo $credentials | jq -r '.Credentials.SecretAccessKey')
export AWS_SESSION_TOKEN=$(echo $credentials | jq -r '.Credentials.SessionToken')

# Verify you're now operating in the member account
aws sts get-caller-identity

We’re in! (Note, hacker union rules require that I say “We’re in!” Don’t blame me — I just follow the rules). You can see that we are currently using the OrganizationAccountAccessRole, and it plops CloudShell right into the identity, which means it’s logged.

Notice that CloudShell emphasizes your current region. That doesn’t stop you from using the CLI to run a command in another region. Let’s play a bit:

aws cloudformation describe-stacks

aws cloudformation describe-stacks —region us-east-1

Type q to exit those results.

Now for a warning, if you swap regions in the console, that starts a new CloudShell in that region, and you lose your assumed role. This is why knowing how to change the region as a command-line parameter is still important.

Another cool capability is that your browser is still using the credentials you logged in with, but your shell is using the assumed credentials. This can be very helpful when troubleshooting or testing.

You can also open CloudShell into its own tab. Click the little icon in the upper right corner of the shell:

Okay, now let’s play with files. Get Account Authorization Details pulls down a detailed JSON report of all your IAM permissions in an account. We’ll save the results to a file accessible to our shell:

aws iam get-account-authorization-details > auth-details.json

Now go to Actions > Download File. You might hit the error/warning in my screenshot — if so just click and follow the instructions to open the shell up in a dedicated tab (this depends on your browser).

Type in the file name: auth-details.json:

To finish up hop over the CloudTrail and take a look at your activities. This will print out recent activity every minute. Pretty cool, eh?

# Poll for recent events every minute
while true; do
  aws cloudtrail lookup-events \
    --lookup-attributes AttributeKey=ReadOnly,AttributeValue=false \
    --start-time $(date -d '2 minutes ago' -u +"%Y-%m-%dT%H:%M:%SZ") \
    --max-results 10
  sleep 60
done

Lab Key Points

• CloudShell has its own metadata service for handling credentials, but you should never need to access it directly (it’s undocumented).

• You can assume roles into other accounts (or the same account) if you have permissions and don’t need to bounce around the console.

• The CLI is incredibly useful and powerful, and can even run scripts.

• It even includes Python, along with file upload and download capabilities!

-Rich