Practical DevOps

Breaking in to a EC2 instance

Niclas Gustafsson

Published in

ITNEXT

6 min readJul 13, 2020

Have you ever misplaced your private keys?

TL;DR;

A quick and easy way to recover your data even though you have lost your private keys.

How did we get here?

Let’s face it, sometimes you mess up… But sometimes you just inherit some part of the infrastructure that has been running for god knows how long and no-one knows the login/password combinations. This was true back in the physical day when everything was running on solid dedicated hardware. Although, at that time, special consideration was sometimes taken for critical machines to which login / passwords were lost. It was not unheard of placing peculiar restrictions / graphic signs in data centers to keep people on their toes as not to make a bad situation worse by acidentally triggering a reboot by mistake or something worse. This problem is very much around still today, maybe so even more working with the cloud. Especially if you are not using some form of system for keeping your private keys secure and getting all your team members to use the same system.

💡Use a password manager with team capabilities!

So I inherited an EC2 Instance that had been running without a hiccup for roughly 4 years, no one knew where the private keys were. We looked where they should be, and where we thought they could be with no success. No one in the tech team had logged in to the system and the person responsible for the setup left years ago.

Sounds familiar?

…If the software running inside the hardened layers of infrastructure decides to give up, you’re out of luck.

This particular server had a simple task: relaying application metrics to an external service, Librato. It was a small server (t2.micro) that was running a Node.js application, Statsd. Or so we believed, not being able to log in and verify this made this particular service score quite high up on my impending-doom high score list. If that server would break down, it would be quite a process to reinstall and recover. Of course, in the world of clouds, various methods exist to handle some of the problems that can occur. Snapshots, backups, creating AMI images and more. But if the software running inside the hardened layers of infrastructure decides to give up, you’re out of luck.

Encryption, In-transit and at-rest.

An important thing to understand regarding encryption of information is when and where it happens. There are two major areas that are solved independently. Encryption on the wire, i.e. encrypting the traffic that is in transit between two servers/entities. This encryption of information as it flows from point A to B is referred to as encryption in-transit and usually done by using some sort of asymmetric encryption scheme. AWS is currently using 2048-bit SSH-2 RSA keys to guarantee that no-one is eavesdropping on the communication between you and your EC2 server. This key has two parts (all asymmetric keys do), a private and a public part. The private part is kept secret by you. And the public part is stored on the server.

For me, it was this private key that had gone missing and therefore we were unable to gain access to the server.

Encryption at-rest

The other side of the coin, encryption at-rest handles how your information is stored, once it has arrived at its destination. Regardless of the format the information is stored, a row in a relational database, a plain text file on a hard drive, encryption at-rest tells you how and if the information is securely stored. This is regardless how it’s resting, on a hard drive, magnetic tape or some other media.

So when to encrypt in-transit and in-rest? Well, nowadays there’s practically no good reason not to encrypt all traffic in-transit outside your controlled infrastructure. The reason is easy to understand, the traffic flows through unknown elements on the internet before it reaches its destination, all of which could eavesdrop on the payload, whether it be a private conversation or financial transactions. The only reason not to do it is that it will incur some additional payload and processing cost. In most practical applications these can be considered negligible, but there are some uses when low-latency traffic needs might provide.

Encrypting at-rest however is not as cut and dried and often left unused. When using public cloud services one often assumes that the protection layer of the provider is sufficient. Some regulations might require that all information is securely stored and encrypted at-rest.

(There is also an additional layer of complexity when dealing with cloud providers that implement storage services, and that is handling the transferring of data between the server and the storage service. The storage service is responsible to store the data to disks. This introduces an additional layer of in-transit of the blocks between the EC2-server and the EBS-service. (which could of course )

Going back to my EC2 instance that I needed to break into. The file system (volume) was not encrypted. And using AWS, even if the volume would have been encrypted, chances would have been good that I would still have access to the key used to encrypt it since this is a managed service (KMS). So as long as you have a) an unencrypted volume or b) an encrypted volume with access to the keys, this method will work.

We can recreate the server and copy the volume to the new server and use this new one instead.

Now that we understand that the PEM keys used to connect to the EC2-instance often get lost due to the simple truth that we are responsible to safekeep them ourselves, will impact the communication with the server but not the contents on the server, we can recreate the server and copy the volume to the new server and use this new one instead.

AWS Break-and-entry 101

1. Create a snapshot of the attached volume that is in-use by the EC2 instance

💡 Please note that if you are snapshotting a busy volume, data loss and/or corruption may occur. However that might be a fair price to pay to gain access to the system again.

2. Spin up a new EC2 instance with this snapshot.
In the Launch instance wizard -> On the third step “Add Storage” select “Add New Volume” and enter the snapshot ID that you got in step 1, either by ID or search by description in the snapshot field.