When docker installations are running in Swarm mode, manager nodes implement the Raft Consensus Algorithm to manage the cluster global state. Docker Swarm mode uses a consensus algorithm to make sure that all the manager nodes, in charge of managing and scheduling tasks in the cluster, are storing the same consistent state.
Each manager node must have an identical copy of the logs, which is located under the following path: /var/lib/docker/swarm/raft/.
These logs are encrypted as they store sensitive data such as the docker secrets (credentials, TLS certificates, etc) used by the services running in the Swarm cluster. However, alongside the logs is located the key used to encrypt them! Located under this path: /var/lib/docker/swarm/certificates/swarm-node.key.
This key is also used for encrypting the communications between the Swarm nodes. This means that if one of the manager nodes has been compromised, then it is possible to decrypt and read the Raft logs to obtain the docker secrets amongst the other sensitive information.
Using Swarm-Rafttool to decrypt the logs
The method to decrypt and read the Raft logs using the Swarm-Rafttool is explained below.
Step 1: Get the Swarm-Rafttool
The swarm-rafttool from Swarmkit is available here: https://github.com/docker/swarmkit/tree/master/cmd/swarm-rafttool. As this tool is written in Go, the first step is to compile it.
$ add-apt-repository ppa:longsleep/golang-backports$ apt-get update && apt-get install golang-go$ export GOPATH=$HOME/go$ go get github.com/docker/swarmkit/cmd/swarm-rafttool |
Once compiled, it can be run:
$ $GOPATH/bin/swarm-rafttool |
Step 2: Make a copy of the Raft logs
If this step is not taken, then swarm-rafttool will show an error, since the logs that are being used by the cluster cannot be accessed by the tool.
$ cp -r /var/lib/docker/swarm $HOME/raftlogs-copy |
Step 3: Decrypt and show the Raft logs
Run the following command:
$ $GOPATH/bin/swarm-rafttool -d $HOME/raftlogs-copy dump-wal |
After that all the entries in the Raft logs can be seen, showing the sensitive data which includes the secrets used by the services running in the cluster.
Swarm Autolock
Docker is aware that storing the key for encryption/decryption of the logs, next to the logs, is not a good idea. That's why it provides a feature (disabled by default) to encrypt this key as well. It's named "autolock" and can be enabled by running a simple command on the Swarm cluster. Once enabled, an "autolock key" is generated to encrypt the key that encrypts the Raft logs (yes, the key of the key!).
As observed, now the swarm-node.key is encrypted.
if autolock is enabled, then an error will be received when trying to dump and read the Raft logs with swarm-rafttool since the autolock key must be provided. How can this be obtained? If by compromising the node, the privileges of the user who runs the Docker daemon (most likely) or root were obtained, then the following command can be executed to obtain the autolock key:
$ docker swarm unlock-key |
After that, the swarm-rafttool with the --unlock-key parameter can be run as follows:
$ $GOPATH/bin/swarm-rafttool -d $HOME/raftlogs-copy --unlock-key SWMKEY-1-XXXXX dump-wal |
Such a command will decrypt the key and then the Raft logs.
Another option is to disable the autolock, it is possible to do that by executing the following command:
$ docker swarm update --autolock=false |
Looking at defense, it is important to protect the manager nodes from attacks. For this the security patches must be kept up to date, unnecessary exposure of the docker.sock minimised and remote non-secure access to the docker API prevented.
Do you want to learn more?
https://dreamlab.net/en/education/trainings-schedule/
References:
- https://docs.docker.com/engine/swarm/raft/
- https://docs.docker.com/engine/swarm/swarm_manager_locking/
Sheila A. Berta
Head of Research at Dreamlab Technologies
