Dump the Raft logs in Docker Swarm to get secrets and more!

When docker installations are running in Swarm mode, manager nodes implement the Raft Consensus Algorithm to manage the cluster global state. Docker Swarm mode uses a consensus algorithm to make sure that all the manager nodes, in charge of managing and scheduling tasks in the cluster, are storing the same consistent state.

Each manager node must have an identical copy of the logs, which is located under the following path: /var/lib/docker/swarm/raft/.

These logs are encrypted as they store sensitive data such as the docker secrets (credentials, TLS certificates, etc) used by the services running in the Swarm cluster. However, alongside the logs is located the key used to encrypt them! Located under this path: /var/lib/docker/swarm/certificates/swarm-node.key.

This key is also used for encrypting the communications between the Swarm nodes. This means that if one of the manager nodes has been compromised, then it is possible to decrypt and read the Raft logs to obtain the docker secrets amongst the other sensitive information.

Using Swarm-Rafttool to decrypt the logs

The method to decrypt and read the Raft logs using the Swarm-Rafttool is explained below.

Step 1: Get the Swarm-Rafttool

The swarm-rafttool from Swarmkit is available here: https://github.com/docker/swarmkit/tree/master/cmd/swarm-rafttool. As this tool is written in Go, the first step is to compile it.

Once compiled, it can be run:

Step 2: Make a copy of the Raft logs

If this step is not taken, then swarm-rafttool will show an error, since the logs that are being used by the cluster cannot be accessed by the tool.

Step 3: Decrypt and show the Raft logs

Run the following command:

After that all the entries in the Raft logs can be seen, showing the sensitive data which includes the secrets used by the services running in the cluster.

Swarm Autolock

Docker is aware that storing the key for encryption/decryption of the logs, next to the logs, is not a good idea. That's why it provides a feature (disabled by default) to encrypt this key as well. It's named "autolock" and can be enabled by running a simple command on the Swarm cluster. Once enabled, an "autolock key" is generated to encrypt the key that encrypts the Raft logs (yes, the key of the key!).

As observed, now the swarm-node.key is encrypted.

if autolock is enabled, then an error will be received when trying to dump and read the Raft logs with swarm-rafttool since the autolock key must be provided. How can this be obtained? If by compromising the node, the privileges of the user who runs the Docker daemon (most likely) or root were obtained, then the following command can be executed to obtain the autolock key:

After that, the swarm-rafttool with the --unlock-key parameter can be run as follows:

Such a command will decrypt the key and then the Raft logs.

Another option is to disable the autolock, it is possible to do that by executing the following command:

Looking at defense, it is important to protect the manager nodes from attacks. For this the security patches must be kept up to date, unnecessary exposure of the docker.sock minimised and remote non-secure access to the docker API prevented. 

Do you want to learn more?

https://dreamlab.net/en/education/trainings-schedule/

References:

• https://docs.docker.com/engine/swarm/raft/
• https://docs.docker.com/engine/swarm/swarm_manager_locking/

Sheila A. Berta
Head of Research at Dreamlab Technologies