Prerequisites

• Ideally have completed the following two labs:

  • https://slaw.securosis.com/p/run-first-instance-finally

  • https://slaw.securosis.com/p/keep-private-subnets-private-vpc-endpoints

The Lesson

First, a little story of my personal pain and suffering.

Back in 2010 when we built and started teaching the Cloud Security Alliance CCSK class, we ran into a small obstacle. It turned out that a shocking number of students didn’t know how to make an SSH connection. It was a real WTF moment, until I remembered that not everyone comes from a Unixy/console background. If you were administering Windows servers, how and why would you know the ins and outs of setting the right permissions on an SSH key?

As for the 25% of students in my Advanced Cloud Security and Applied DevSecOps class that one year at Black Hat? Uh… what did you think you were signing up for? Did you really have to complain about using SSH in an advanced Black Hat training on your course evaluations?

Deep breath. I digress.

A few years ago AWS added Session Manager, which we’ve been using in these last few labs. This was a major boon for training, since we didn’t need to have students mess around with keys and hope their corporate laptop didn’t lock out SSH. But Session Manager is so much more, and today we’ll dig in a bit deeper to play with a couple additional features.

SSH Over the Internet Is Bad

When using public cloud — like, you know, AWS — we need to manage everything over the Internet. We can’t walk over to a server and log in — we need to connect remotely. No biggie, we’ve been doing that in and out of our datacenters for decades.

And, for decades, remote access via SSH (or Remote Desktop Protocol [RDP] for Windows) has been a major source of security breaches.

This is very true in cloud today. When you put something with port 22 (SSH) open on the Internet, it is nearly instantly attacked. You’re mostly fine if you are patched and up to date. Mostly.

To prevent this, we tend to use VPNs or jump boxes, which are hardened servers designed to give a devs/admins/legit user access to a private subnet. These are just hardened portals on the Internet, which serve as bridges to our private networks.

SSH has one other annoyance when operating at scale: keys. The best practice is to use a public/private keypair, instead of a username and password. This works great, but managing all those keys when dealing with a lot of systems is a pain.

Why Session Manager Is Awesome

As you’ve seen, with Session Manager we can hop directly into our Instances, using AWS as our bridge to private subnets. Heck, you can even use it to connect to resources in your datacenter, so long as they are running the Systems Manager agent and can talk to the AWS APIs. This eliminates the need for jump boxes or VPNs.

That alone is awesome, but Session Manager has a few other tricks I didn’t cover yet which make it even better:

• You can control access to resources based on IAM policies! We’ve been working with our AdministratorAccess role so we, of course, can use Session Manager on any instance (with the SSM role so it can talk to the service — don’t forget that). But in a real operation you can restrict who can use Session Manager. Want to get really fancy? You could write an IAM policy which only allows someone Session Manager access to a specified instance, or all instances with a particular tag. This is much better than using SSH keys. And it means MFA is easily enforced for logins.

• You can log everything someone does in a session! Since AWS is the bridge to the instance, it can track all those command lines. You can store them in CloudWatch or S3. Why does this matter? You get full logging of all access and commands, so an attacker can’t just delete the logs. (Note that I believe the Session Manager agent, not the console, sends the logs to S3. We will talk about that more in the next lab).

• It works from the command line, not just the console! This requires having the AWS command line tools, but that’s just a small change for your developers, works great with Identity Center, and eliminates the need for stored SSH credentials. You can even tunnel SSH itself directly over Session Manager. That’s more operational than security-specific, but you can learn about it here.

Session Manager has other features we won’t use those in today’s lab:

• Session Timeout: Tired of getting booted out after 20 minutes? Set your own timeout. Maybe 4 hours is more your style for those marathon debugging sessions.

• Run As: Want to connect as a specific user, instead of the default ssm-user? No problem! Configure the Run As setting to specify a different user for your sessions.

• Shell Profiles: Customize your shell environment across all your instances. Want aliases? Custom prompts? Set it once in your shell profile, and it's available everywhere.

AWS Session Manager keeps getting better. With IAM we get granular yet scalable access control. With S3 logging we got accountability. With CLI access they improved usability. And with these additional preferences we can fine-tune our experience.

Lesson Key Points

• Exposing SSH to the Internet is highly risky, but still common for system management.

• Session Manager eliminates the need to SSH over the Internet, and provides access to private subnets.

• Session Manager supports logging to S3, and has an option to run it from any command line — not just from within the web console.

The Lab

I originally planned this lab to demonstrate saving logs to S3 and showing off the command line, but in testing that combination took more than our normal 15ish minutes. Instead we’ll learn how to set the Run As user, and how to connect via the command line. In our next lab, we’ll learn how to properly configure an S3 bucket and the permissions for Session Manager logging.

Video Walkthrough

Step-by-Step

First we’ll log in and check out the Session Manager settings for the first time. Start in your Sign In Portal > TestAccount1 > AdministratorAccess. Then go to Systems Manager, which is where Session Manager lives:

Go to Session Manager and look around a little. Then go to Preferences > Edit:

Click Enable Run As support for Linux instances. There’s more going on here than meets the eye. We are setting things up so when we log in with Session Manager, we will be logged in as ec2-user, the default user on Amazon Linux instances. This matters because otherwise we are logged in as a generic ssm-user.

However, what if we don’t always want to log into that user account, every time, for every instance? Fortunately Session Manager enables you to tag a user or role with a key of SSMSessionRunAs, and a value of whichever operating system user you want! That user or role doesn’t need admin privileges, but it does need to exist in the instance.

This helps us scope down our operating system privileges, and still use Session Manager. While we’ve used all administrative rights in these labs, you don’t need to, and Run As is how you can designate which AWS users/roles can use which operating system users/roles.

Unless you’re on Windows. Kind of SOL there. But you were when you decided to use Windows in the first place.

Then scroll to the bottom and click Save.

With that setting… set… let’s launch our CloudFormation template to set up the following:

• The same VPC with 2 public and 2 private subnets we’ve been using.

• Our 3 required VPC Endpoints for Session Manager to work.

• 2 instances, both in the private subnet.

  • SLAW and SLAW2 are basically identical, based on an AMI (image) I built and shared. This image is the stock Amazon Linux, but I added the extra plugin to use the command line to connect via Session Manager.

    • Why didn’t I have you just install that as part of the lab? Because we don’t have a NAT Gateway, so our instance couldn’t yet access the Internet to install updates!

  • I added more IAM permissions to the IAM role, so we can use one instance to log into the other

Use this template: https://cloudslaw.s3.us-west-2.amazonaws.com/lab34.template.

And name it SLAW:

Just click through the screens and hit Submit. You should be pretty good at this by now. If not review the prior lab and swap out the template URL.

Now I want you to wait 5-7 minutes! Not only does everything need to launch, but it all needs to sync up. Otherwise you won’t be able to connect with Session Manager, and you’ll sit there hitting Refresh constantly. This is a good time to review the template in CloudFormation to see if you can figure out what it’s doing.

After 5-7 minutes, go to EC2 > SLAW > Connect:

Now type in whoami and it should say ec2-user. You are now logged in as that user — not the default ssm-user.

Nice. We could have created any user account we wanted in the operating system image, and tagged our AdministratorAccess role to log in with whatever user account we wanted, instead.

Now let’s see how to log in using the command line rather than the console.

Copy this command and paste it into your Session Manager window but don’t hit Enter/Return yet!

aws ssm start-session --target

Then go back to your EC2 tab and get the instance ID for SLAW2.

Paste that onto the end of the command like so it looks similar to this, and hit Return/Enter:

aws ssm start-session --target i-123456789

Even though you are in the same window in your browser, you have now jumped from one instance to the other:

Architecturally it looks like this. We have two instances in the same subnet, each connecting through the VPC Endpoints, and SLAW is logging into SLAW2:

NOW DELETE YOUR CLOUDFORMATION STACK!!!

I am not responsible for your AWS bill if you forget!!!

That’s it for this week!

-Rich