Prerequisites

• Your root account credentials (this is one of the few times we will need them)

• Your non-root admin user account credentials

• The SNS topic we created in https://slaw.securosis.com/p/timmys-first-cloudformation

READ THIS FIRST!

It was bound to happen sooner or later… and it happened sooner. I made a mistake!!! When I first published this lab I set it to run in Oregon but use data from Virgina. I hadn’t seen this as an option when I previously created this alert and thought it was a supported option.

I was wrong.

I updated our earlier CloudFormation lab that creates an SNS topic in the us-west-2 region to now ALSO create it in us-east-1. We now run this lab completely in us-east-1.

If you only ran the CloudFormation Lab in the us-west-2 (Oregon) region, please go back and run it again in the us-east-1 (Virginia) region! This will only take you 2-5 minutes. Here are the steps, without screenshots, to run through it quickly:

• Click on the region in the upper right and pick Virginia

• Got to CloudFormation

• Click on Create Stack

• Use the template in the code block below, and follow along the instructions from Timmy’s First CloudFormation if you need guidance again.

That’s it, really sorry for the mistake but… it won’t be the last.

AWSTemplateFormatVersion: '2010-09-09'
Description: Template to create a SNS topic named SecurityAlerts
Resources:
  SecurityAlertsTopic:
    Type: AWS::SNS::Topic
    Properties:
      TopicName: SecurityAlerts

The Lesson

Cloud security pros use this ONE TRICK to catch attackers!

Okay, did I meet my SEO requirements for the month?

Today we will create one of the most useful security alerts you can enable for your account (or Organization). This one has saved my bacon multiple times, and I know a few more times it would have saved me, had I not been an idiot and forgotten to create it. It can catch a wide range of attacks and mistakes, especially ones that don’t trigger other threat detectors.

Pretty much everything we do in cloud costs money. Even though all our new accounts are on the AWS free tier, it’s really easy to make a mistake and rack up hefty charges. The free tier is awesome, but has limits. And if you build or secure non-training environments, you won’t be on the free tier for long.

One of the best ways to detect multiple security and operational issues is to put a billing alert in place. Let me share the 2 times I didn’t put them in place and then paid the price (literally):

1. The first time I posted demo code on GitHub, I had some static credentials (Access and Secret Access Key) tucked away in a file outside the main code that still got uploaded. This was nearly 10 years ago so… I was first? Anyway, I got off easy with $500 of unplanned Litecoin mining from some friends in China, which AWS refunded me, but a billing alert would have caught it at $25 dollars.

2. When we were in startup mode and building the platform that is now https://defense.firemon.cloud one version of our architecture switched to short-lived containers. We had a multi-thousand-dollar increase in our AWS Config bill that month due to paying every time a new container spun up and connected to the network (every network interface and security group triggered a Config assessment/record). Nope, no billing alerts on that account yet, so we didn't realize it until our CEO saw the bill.

That’s one security and one operational example. For our training it will be especially important, because I want this training to be as close to free as possible, but if you mess up and leave something running or turn on something we don’t expect, you could end up paying more than you realize.

Billing alerts also detect a range of security incidents: essentially anything that increases resource usage beyond expected levels. The most common scenario today is exactly what happened to me a decade ago: cryptomining. It turns out the only way to reliably make money mining crypto is if someone else pays the bill.

Aside from cryptomining, our alert can help in other cases, including large data exfiltrations and certain kinds of denial of service attacks. It is a late indicator compared to many real-time threat detectors, but much better than no indicator. Plus, a threat detector won’t catch that distracted developer who spins up an extra large instance and forgets to turn it off over the weekend. Billing alerts make you look like a superhero to accounting!

There is one major downside. To access the billing console you need to use the root account. By default only the root account can access billing data. Don’t worry — we’ll fix that as part of this lab so it won’t be a problem again (for this AWS account).

Today’s key points are:

• Billing alerts trigger when you exceed a spending limit in your AWS account.

• They can help detect attacks which increase cloud and resource usage.

• The most common such attack we currently see is cryptomining.

• Billing alerts also help detect many operational issues. They are extremely important in lab environments like the one for these labs.

The Lab

This lab has 3 stages:

• Log in with your root account and change permissions so your admin user can read billing information.

• Create the billing alert and send it to the SecurityAlerts SNS topic we previously created.

• Subscribe to the SNS topic with your email.

It might seem like a lot, but the steps are quick and the entire process should take about 15 minutes.

Video Walkthrough

Step-by-Step

First, go to https://console.aws.amazon.com and sign in with your root account email and MFA. You might need to click the link to use root credentials:

We have to log in as root to change a setting which will allow other IAM users to see billing information. Without changing this setting, even your admin IAM user can’t see the data. Click in the upper right corner and then Account to reach the setting.

Scroll down until you see IAM user and role access to Billing information and click Edit.

Check Activate IAM Access and Update.

Then sign out of your root account from the upper-right menu.

For the rest of this lab, log back in as your non-root admin user (rmogull in my case). You can use the direct sign-in URL if you kept it someplace, or enter your account ID on the default login page. Sometimes I forget my account IDs. One trick is that my MFA app keeps the account ID in the name of the entry and I can pull it from there.

Click in the upper-right corner again, and go to Billing and Cost Management.

This takes you to the billing dashboard. For some reason I have a $.03 charge I need to keep an eye on. You may not be able to see your (probably $0.00) balance yet, if you haven't previously visited this page.

Scroll down on the left side to Billing Preferences, click, and then click to edit your Alert preferences.

Here you can tell AWS to send your Free Tier alerts to a different email than your root account email, by checking the box and adding an email address. This is optional but recommended if your root account has an email you don’t monitor as much.

The second box is the important one, which enables the ability to create and send billing alerts. Check the box.

Make sure you checked Receive CloudWatch billing alerts, or the rest of this lab won’t work! It should look like this after you click Update:

Now go to CloudWatch to build the alert. CloudWatch is the core performance monitoring and log service for AWS. It’s how you can monitor everything from CPU performance to serverless function logs to — you guessed it — billing metrics.

Also double check the upper right corner to make sure you are in the Virginia region.

Now click Alarms and then All alarms.

And Create alarm.

Billing won’t show up as an option until you select N. Virgina (us-east-1) (it should be the default) in the main window area as your region. You may need to click Select metric to see the popup list. Then click Billing.

We will use Total Estimated Charge, our total bill for the month. You can also create metrics for individual services, but I leave fancy economic stuff like that to Corey Quinn at the Duckbill Group (and you should read their stuff — I’ve learned a ton, about much more than billing). Next click EstimatedCharges and Add to search.

I set mine for $25 (scroll down if you don’t see it). If you are in another country you can pick your own currency. Then click Next.

On the next page we set up our Action. In this case we want to send the billing alert to the SecurityAlerts Simple Notification Service topic we set up in a previous lab (see the prerequisites). If you don’t see it double-check your region (Oregon!) and make sure you ran that lab. If your password manager obscures the SecurityAlerts popup, typing part of SecurityAlerts may be enough to get rid of the password manager matches.

If you don’t see SecurityAlerts as an option double check that you created the SNS topic in Virginia. An earlier version of that lab had you do it in Oregon only by mistake.

You may recall that we never set up a subscription to that SNS topic, so AWS warns us here. Click View in SNS Console and a new browser tab will open right where we need it. A good thing to remember is that if you see that box and an arrow symbol in the AWS console, it means AWS will open up a new tab so you don’t lose your place with what you are currently doing.

For now click to open the tab for later but come back to the CloudWatch tab. We will add the subscription after we finish creating the alert.

Now scroll down and click Next.

Name it BillingAlert and click Next.

Scroll down the next page (which reviews alarm settings) and click Create Alarm:

The alert is configured to send to the SNS topic, so switch to the open SNS tab in your browser and create an email subscription:

SNS can send to a lot of things, including email. Pick Email as your Protocol:

Enter your email address, then Create subscription.

Now check your email for a message from AWS and click the link to finish subscription. The first screenshot is of my email, and the next is what pops up after you click that link to confirm the subscription. This is required to prevent spam:

Now you’ll get an email if you ever have $25 or more in charges. Our alarm only checks every 6 hours — it’s not instant. Feel free to set it lower, or to set multiple tiers of alerts.

Key Lab Points

• You need to log in as root to enable IAM users and roles to see billing and cost information.

• Before creating a billing alert you need to also enable them in the Billing and Cost Management settings.

• Then you can create a billing alert in CloudWatch and send it to an SNS topic.

• Our alert will run every 6 hours and send email if you have $25 or more in charges.

• You can set multiple amounts and timing thresholds if you want.

• We created the alert in us-west-2 (Oregon), even though billing information is in us-east-1.

-Rich