Prerequisites

• Have completed the Security Hub lab

The Lesson

In recent labs we set up monitoring using various security tools. We started with CloudTrail for management plane activity, then added GuardDuty for threat monitoring, and finally Security Hub to aggregate all the findings and events from various AWS security tools. Today we’ll talk about events, as distinct from logs, and how to manage events for security.

First we need to understand events, which every cloud provider supports. I like to think of them as a stream of activity. These are all the behind-the-scenes notifications the various cloud services use to communicate things like “job started”, “job finished”, and “error: the world is ending and I’m taking you down with me”.

Events aren’t saved anywhere — they are just sent to some kind of communication bus which other services can monitor for things they might need to do. For example if “run instance” fails for some reason, the thing running the instance needs to communicate that failure to the UX or an API so the user knows it didn’t work… and internally, other services like the one managing virtual hard drives might need to know it should clean up any broken/incomplete/unused resources.

Service teams decide what events their service creates, and in AWS many of these are accessible to users (us) via the AWS event service called EventBridge.

I like to think of EventBridge as a stream, and events as salmon. When a service creates an event, it’s like the salmon jumping into the air. It’s only there for a moment, and if you don’t capture it at that point in time, then it’s gone for good. You can’t replay the salmon jump and; sure, another salmon will jump, but it won’t be your salmon. (Since I don’t have a stuffed salmon, in the video I use a LEGO X-Wing instead — it’s far more amusing).

Back to security. Some of these events are useful to us. Especially ones created by security tools like Security Hub. They enable us to act in near-real-time on the things Security Hub sees (including events from all the other AWS security services). And there are operational events we might also want. For example, I can build threat detectors that use CloudTrail events (which are real-time) instead of waiting for those events to get saved down to log files, where I analyze them (which takes 5-15 minutes).

How do we do that? We become the bear!

Let’s be real — in all those salmon videos, there’s always a bear sitting on top, snagging salmon out of the air. We can do the same with events by creating an EventBridge Rule. A rule combines a pattern with an action. The pattern is “Which event(s) am I looking for?” and the action is “What do I do when I match an event?”

We can save events to a log. We can forward them to an external security tool. We can trigger a Lambda function. We can even just email them somewhere.

Understanding the difference between events and logs is important. In datacenters we historically handled everything using log files. In cloud, a lot of really important information is generated as events, but never saved to log files. Or it’s saved someplace hard to use. That’s where EventBridge comes in: it enables us to tap into this stream of information, filter our what we want, and then send that off where it’s useful for us for security.

Which is our belly. Because we are the bear, and you should keep your security team well fed. Salmon is quite a healthy option and… I did it again. Sorry, I really do like analogies too much some days.

Lesson Key Points

• All cloud providers support events, which are the back-end signals published by various services to keep the management plane up to date on what’s going on. Events are essential for making the cloud work.

• EventBridge enables us as cloud customers to tap into this stream of activity, and even publish our own activities.

• Events are ephemeral and disappear instantly. They are not saved. but we can use EventBridge rules to match certain patterns, and then “do something” with those events. We can, for example, email an event or send it to a log file.

The Lab

This lab is a three-step process:

• Create a new Simple Notification Service topic for sending our security alerts.

• Subscribe using your email.

• Create an EventBridge rule to send all new alerts (findings) from Security Hub to the SNS topic.

Note that this is not what you should do in a real enterprise environment, unless you really like emails. You would send the events to your security monitoring tool or someplace else better for handling the data. Our labs will be very constrained, for now, so email is fine — we’ll change it up in future labs when we need to.

Also, we are sending everything from Security Hub. This is a choice, for now, due to the constraints of these lessons. You could instead filter to only send findings from some services (e.g., GuardDuty) or of particular criticality (e.g., HIGH/CRITICAL). You could send CRITICAL alerts to a shared email for the security team, and everything else to a security monitoring tool, or… both… or… you get it. It’s very flexible is what I’m saying, and we’ll keep it simple for now.

Video Walkthrough

Step-by-Step

For once we won’t use our management account at all! Sign into your portal and use your SecurityFullAdmin role in the SecurityAudit account.

First, make sure you are in Oregon. Events are region-specific and only appear in the region where they are created! We set up our home region for Security Hub to Oregon, so that’s where all the events will appear. As a reminder, Security Hub performs a little magic to aggregate events from all accounts and regions — most AWS services don’t.

Then go to Simple Notification Service (SNS):

We want a standard order and the name will already be filled in; use Security Hub Alerts as the display name. Then scroll down and click Create topic.

Now we need to subscribe. This should be easy since you’ve done it before. Click the Create Subscription button, choose Email for the protocol, then enter your address and click Create Subscription on the second page:

Check your email and click Confirm subscription:

Great, now it’s time to go to EventBridge:

And Create rule:

Name it SecurityHubFindings and give it the description in the screenshot. Make sure it’s set for Rule with an event pattern, since the other option is basically just a cron (time-based) trigger. Then click Next:

Leave the default event source at AWS events and scroll past the Sample event section. I don’t use that section much — I tend to not need it when writing event patterns — but if you ever do have a sample event, such as a specific CloudTrail entry, you can paste that in and use it to help build and test your pattern. Ignore it for now:

Instead of using the builder (the default), we will copy and paste a specific pattern directly from the AWS documentation. This pattern will match whenever a new finding is imported into Security Hub. That’s how this works on the back end: all the other security services create their events, and Security Hub imports them to the home region (based on how we have things set up). I think this might be how Security Hub performs centralization, since it has both an event creation function (if you use the Security Standards we disabled, so it creates its own findings) and an import function which collects from everything else. Events are created only in one region, but can be imported centrally. At least that’s my educated guess (for certain values of “educated”).

Here’s the pattern to copy and paste. Notice that the source is Security Hub and the ‘detail-type’ is an imported finding. It takes a little time to learn which patterns to use when to capture different events. It can be tricky — fortunately many of the things you will want to do, someone else has already figured out and posted someplace. Sometimes they even get it correct!

{
  "source": ["aws.securityhub"],
  "detail-type": ["Security Hub Findings - Imported"]
}

Select Custom pattern, then paste in the JSON, then Next:

The next step is to set the target. The first part of this process captures the events we want, like the Death Star and the Millennium Falcon. The second part sets the target: where to send the event. In our case we want to pick AWS service and SNS topic, and then the topic we created — which you may recall is SecurityHubAlerts. Yeah, I had to look it up too.

The next page lets us tag the rule. We’re skipping that, so just click Next. I hope you aren’t too disappointed that I’m skipping the screenshot.

The next page is a summary of what we just set up. Double-check that it looks correct and Create rule:

And thus we come to the end of another weekly lab. Now you have Security Hub set up so any findings/events from any of the security tools (we’ve only set up GuardDuty so far) will send you an email. This is not what you want to do at work! Our labs are constrained and you shouldn’t get any emails for a while. If you do get an email alert outside of a lab where I tell you to expect it, feel free to email me, and I can help investigate.

At some point we will be increasing the potential volume of Security Hub events. When we get there we will modify our EventBridge rule/SNS topic to send notifications someplace else, and unsubscribe your email. That’s one of the cool things about SNS: it’s easy to add and remove subscriptions.

As always, hit me up if you have any questions, and enjoy the rest of your week.

Lab Key Points

• Security Hub consolidates all events into the home region we set last week.

• We created an EventBridge rule to match the pattern when a finding is imported into Security Hub. This is the “trick” which enables us to tap into nearly all AWS security tool events with a simple rule.

• That import function is very fast, so this adds no appreciable delay to notifications, event though we are working across regions.

-Rich