Prerequisites

• You must have completed Create a (Forensic) Snapshot and Play CSI!

The Lesson

Solving crimes is hard work, but with a little grit and a strong stomach, we can all get through this together.

(Work with me, people; it’s Friday after work as I write this, and I can’t go to happy hour until I finish).

Last week we learned about Elastic Block Store (EBS) Volumes and Snapshots. We:

• Launched an instance (via CloudFormation)

• Identified the associated storage volume

• Created a snapshot

• Shared that snapshot with our SecurityAudit account (which is pretending to be our forensics analyst account)

This is such a standard process that it’s part of the very first cloud security automation code I ever wrote. Code that’s so old I used… Ruby… and, well, it’s old. (That code did lead to building and successfully selling a startup, which used to be called DisruptOps and is now FireMon Cloud Defense).

As a reminder, we call this process forensics acquisition, and our objective is to obtain a forensically valid copy of the data. In other words, we need the ability to prove the source hasn’t been tampered with or modified. That isn’t all that important in most cases, but if you ever investigate something that might end up in court (which includes HR issues, not just outside attacks) it’s a good practice.

In larger environments you might do this a lot, and it helps to have a standard setup. Here’s an example architecture (stolen from my incident response class) very similar to what we are doing:

This shows the split between monitoring and operations, automation with cross-account access, and use of a temporary forensics system for analysis. Today we will… perform that analysis.

We are going to build on last week’s lessons, and learn how to:

• Create an EBS volume from a snapshot

• Attach a volume as an additional drive for analysis

• Access the drive from our analysis system

If you ever worked in a cyber forensics lab you had all sorts of special hardware to copy and connect drives without mucking up the evidence. We get to… run an instance and connect a storage volume. It doesn’t matter if we mess up that volume (drive) because we can always create a new one from our immutable snapshot. We can delete the snapshot but can’t modify the data on it, which keeps the lawyers happy.

A SecOps team would have some pre-configured AMIs set up with their analysis tools. In our case we’ll just use a regular Amazon Linux image, and you’ll poke around by hand to see if you can find anything interesting in our ‘compromised’ instance.

Key Lesson Points

• You can pre-load new storage volumes by creating them from a snapshot

  • This does not modify the snapshot in any way

• This is an important capability for backing up and sharing data, and we can it for forensic analysis

• Analysis should be performed in a different account than where the incident happened. You never know if the attacker is lurking in there like China in the US federal wiretap system (I wonder how many years we’ll get out of that incident)

The Lab

I largely described the lab already in the lesson. We will create a volume from the snapshot, attach it, and explore it.

I planted some evidence in the instance we snapshotted last week. This lab will get you to the point where you can hunt for it, but I won’t spoil the surprise. Email me if you get stuck! I want to ensure everyone gets to enjoy the fun.

Video Walkthrough

Step-by-Step

First set up our environment using our little friend, CloudFormation. Log into Sign-in portal > SecurityAudit > AdministratorAccess > CloudFormation > Create stack, and use these details:

• S3 URL: https://cloudslaw.s3.us-west-2.amazonaws.com/lab42.template

• Name: Forensics

Once the stack shows Create Complete, go to EC2 > Snapshots: 

You will likely see enough public snapshots to outlast the heat death of the universe. Seriously, these things have their own gravitation field due to the mass of the electrons. There are three filters here:

• Public: Anything anyone has ever made public. Jimmy Hoffa’s body is here somewhere. At least that’s my theory.

• Owned by me: Snapshots in this account.

• Private: Snapshots shared with this account, like the one we shared last week.

Pick Private snapshots:

Awesome. You should see that CIS-incident-1 snapshot. Click the blue snapshot ID to see its details. From here we can directly create a volume from the snapshot. This is also possible using the CLI/API with the CreateVolume call. In my experience, I get better results in the console creating the volume from the snapshot screen, since it sets the size and type correctly.

Then Actions > Create volume from snapshot:

First a quick note on process. EBS Volumes need to be created in the same availability zone as the instance they connect to! Think way back to when I first described AZs as essentially their own datacenters within regions. It’s hard to plug in a hard drive across buildings, and that’s what we are (virtually) doing. In reality the volume storage needs a very high speed network connection to the virtual machine (instance), and thus has to be physically close.

In this lab I have things set in CloudFormation so the instance should be in us-west-2a. But if you ever run your own stuff it might not be. (And hopefully my CloudFormation is working the way I think, or I’ll get a lot of angry email).

In terms of settings, the defaults are good. Scroll down and add a Tag with a Key: Name and Value: CSI. Then click Submit:

This is so small it will create within seconds. By the time you get to the next screen it should be good, but be aware that in other circumstances you will want to make sure it shows Status: Available.

Now go to Volumes, click the checkbox next to CSI, then go Actions > Attach volume:

We kept things simple with only one analysis instance, so click Instance and select Forensics. This is way easier than the old days, when they wouldn’t show the instance name and you had to write down all the IDs.

Next we need to pick a mount point. Basically, where we are plugging in the virtual drive. The (safe) usable addresses start at /dev/sdf. That’s “device/SCSI Device F” (I think sd means SCSI, but I’m too lazy to validate [Confirmed. —Ed]). Now here’s a weirdness of cloud and virtualization: the operating system knows it’s a virtualized storage device, not a physical one, so it basically says “FU, I’m calling it /dev/xvdf”.

For the record, the night this change pushed was, like, the night before one of my first training classes, and the labs broke and I… cried (and figured it out before class, but it was rough). Then Attach volume (you don’t need to cry if you don’t want to — this should work even without saltwater sacrifice).

It attaches quickly. Now we log into the instance to finish setup and start our autopsy. Click Forensics > Connect > Session Manager like we’ve been doing for the past couple months.

Okay, from the Session Manager window we need a few command lines.

First, let’s make sure it’s mounted correctly. This command translates roughly as “list block devices”:

lsblk

You should see this:

Right. xvdf as expected, because Linux.

Okay, next up we need to create a directory where we will mount this new drive so we can look at it. This translates to “superuser do [escalate my privileges]: create a directory named /forensics”

sudo mkdir /forensics

Now we need to mount our volume at that location. This one is tricker because we are using the eXtensible File System (XFS) and there are multiple partitions on the storage volume. This command line says “mount the xvdf1 partition at the /forensics location”

sudo mount -t xfs /dev/xvdf1 /forensics

And let’s look at it using cd for change directory and ls for list contents:

cd /forensics
ls

Okay, that looks like the root of an Amazon Linux instance!

Here’s where I give you some hints and send you out to hunt. I left something hopefully fun for you to find and it isn’t too hard, but if you don’t know Linux you’ll need these commands:

• cd <path to directory> : e.g. cd /forensics/home

• ls : list the contents of a directory. ls -a includes hidden files.

• cat <filename> : show the contents of a text file

• sudo su : this basically sets you as the root user. You might need this to see parts of the file system you don’t otherwise have access to.

I didn’t hide things too hard. Think about what might be left if someone used stolen credentials to log in and leave something. Then click your heels three times.

And email me at [email protected] if you get stuck and need the answer.

Lab Key Points

• You can create an EBS Volume from an EBS Snapshot by specifying so when you use the create command.

• Volumes and instances need to be in the same region.

• You can connect multiple volumes to an instance, at different mount points.

• Just attaching isn’t enough — in the operating system you also need to make sure it sees the drive, and the OS is set up to let you read it. This is different on different operating systems, but is mostly the same across Linux versions.

-Rich