Booting an unattended / headless full disk encrypted server – Ubuntu server 16.04 setup

I wrote this article as a reminder for myself. I’m using Ubuntu 16.04 server on both sides.

I have a server, let’s call it Box1, with full (root file system) LVM encryption. It’s great as it protects my data but the problem is that it needs someone to input the password at boot to decrypt the LVM volume.
If the server reboots and is unattended, it stays stuck at the decrypt password prompt and doesn’t boot. The same when having a headless server running remotely in a data center.

The solution? Mandos!

Box1 will run a small mandos client program in the initial RAM disk environment which will communicate over the local network with the mandos server, let’s call it MandosServer1. All network communications are encrypted using TLS.

Box1 will be identified by MandosServer1 using an OpenPGP key; each client (in this case Box1 but I could have many) has one unique key.

MandosServer1 sends Box1 an encrypted password. The encrypted password is decrypted by Box1 using the same OpenPGP key, and the password is then used to unlock the root file system, whereupon the computers can continue booting normally.

This basically works with MandosServer1 located in the same local network as Box1.
You still can find a WAN setup later in this article.

WARNING
At the time of writing this article the mandos packages provided with Ubuntu 16.04 are buggy (1.7.1-2build1 500). You need to either recompile a newer version or you can download the ones I compiled here:
mandos_1.7.12-1_all.deb
mandos-client_1.7.12-1_amd64.deb

UPDATE (June 5th 2017), my latest build:
mandos_1.7.15-1_all.deb
mandos-client_1.7.15-1_amd64.deb

Server Installation

We will start by installing mandos on MandosServer1:

 sudo dpkg -i mandos_1.7.15-1_all.deb

It will throw errors related to dependencies. To solve that do:

sudo apt-get -f install

Then we start mandos and make it startable at boot:

sudo service dbus reloadsudo service mandos startsystemctl enable mandos.service

Client Installation

We will install the mandos client on Box1.
The following is mainly useful to increase the quantity of entropy in kernel to make /dev/random faster:

sudo apt-get install rng-toolssudo rngd -r /dev/urandom

Installing the client (it may take a while…):

sudo dpkg -i mandos-client_1.7.15-1_amd64.debIt will throw errors related to dependencies. To solve that do:sudo apt-get -f install

Now we will generate the client certificate. It will ask for a password. Be sure to enter here the very same password as the one  you use to unlock the client (Box1):

sudo mandos-keygen --password --type RSA --force

Copy the above command output to the end of the file /etc/mandos/clients.conf server side and restart mandos:

/etc/init.d/mandos restart

Voila!

Now restart your client (Box1). It should boot, pause few seconds at the decrypt password prompt to receive the password from the server MandosServer1 and then continue to boot without interruption.

Using Mandos in a WAN environment

Mandos Server side

First, you need to modify the /etc/mandos/mandos.conf on mandos server. Set up the port to something convenient for you, set use_ipv6 = false (unless you are using IPv6), set zeroconf = False.

Here is my mandos.conf:

# This file must have exactly one section named "DEFAULT".
[DEFAULT]
 
# These are the default values for the server, uncomment and change
# them if needed.
 
# If "interface" is set, the server will only listen to a specific
# network interface.
;interface =
 
# If "address" is set, the server will only listen to a specific
# address. This must currently be an IPv6 address; an IPv4 address
# can be specified using the "::FFFF:192.0.2.3" syntax. Also, if this
# is a link-local address, an interface should be set above.
;address =
 
# If "port" is set, the server to bind to that port. By default, the
# server will listen to an arbitrary port.
port = 9601
 
# If "debug" is true, the server will run in the foreground and print
# a lot of debugging information.
debug = False
 
# GnuTLS priority for the TLS handshake. See gnutls_priority_init(3).
;priority = SECURE256:!CTYPE-X.509:+CTYPE-OPENPGP:!RSA:+SIGN-DSA-SHA256
 
# Zeroconf service name. You need to change this if you for some
# reason want to run more than one server on the same *host*.
# If there are name collisions on the same *network*, the server will
# rename itself to "Mandos #2", etc.
;servicename = Mandos
 
# Whether to provide a D-Bus system bus interface or not
;use_dbus = True
 
# Whether to use IPv6. (Changing this is NOT recommended.)
use_ipv6 = False
 
# Whether to restore saved state on startup
;restore = True
 
# The directory where state is saved
;statedir = /var/lib/mandos
 
# Whether to run in the foreground
;foreground = False
 
# File descriptor number to use for network socket
;socket =
 
# Whether to use ZeroConf; if false, requires port or socket
zeroconf = False

Adjust the timeout parameters on the client.conf file. Adjust so that it will not disable the client by mistake. This is my client.conf file (I removed the client config):

# Default settings for all clients. These values are the default
# values, so uncomment and change them if you want different ones.
[DEFAULT]
 
# How long until a client is disabled and not be allowed to get the
# data this server holds.
timeout = PT5M
 
# How often to run the checker to confirm that a client is still up.
# Note: a new checker will not be started if an old one is still
# running. The server will wait for a checker to complete until the
# above "timeout" occurs, at which time the client will be disabled,
# and any running checker killed.
interval = PT2M
 
# Extended timeout is an added timeout that is given once after a
# password has been sent sucessfully to a client. This allows for
# additional delays caused by file system checks and quota checks.
extended_timeout = PT15M
 
# What command to run as "the checker".
checker = fping -q -- %%(host)s
 
# Whether to approve a client by default after the approval delay.
approved_by_default = True
 
# How long to wait for approval.
approval_delay = PT0S
 
# How long one approval will last.
approval_duration = PT1S
 
# Whether this client is enabled by default
enabled = True

Client side

Then go on the client and modify /etc/mandos/plugin-runner.conf. It should look like this:

## This is the configuration file for plugin-runner(8mandos). This
## file should be installed as "/etc/mandos/plugin-runner.conf", and
## will be copied to "/conf/conf.d/mandos/plugin-runner.conf" in the
## initrd.img file.
##
## After editing this file, the initrd image file must be updated for
## the changes to take effect!
--options-for=mandos-client:--connect=<ServerIP>:<ServerPort>

This will make the client contact a distant server to get the key instead of trying to contact a local network one.

At the moment of the boot process when the mandos client will try to reach the mandos server for the key, the network is not up. You will need to bring it up to be able to reach the server. For this, we need to add a network hook  by creating a file in /etc/mandos/network-hooks.d containing the necessary commands to bring the network up. This is a very simple example with a DHCP configuration:

#!/bin/sh
#
# This is an example of a Mandos client network hook. This hook
# brings up an interface as specified in a separate
# configuration file. To be used, this file and any needed
# configuration file(s) should be copied into the
# /etc/mandos/network-hooks.d directory.
# 
# Copying and distribution of this file, with or without modification,
# are permitted in any medium without royalty provided the copyright
# notice and this notice are preserved. This file is offered as-is,
# without any warranty.
 
# Exit immediately if a command exits with a non-zero status
set -e
 
do_start(){
ip link set enp0s3 up
sleep 10
ipconfig -c dhcp -d enp0s3 
sleep 10
}
 
do_stop(){
ip link set enp0s3 down
}
 
case "${MODE:-$1}" in
start|stop)
do_"${MODE:-$1}"
;;
esac

We need to update initramfs:

sudo update-initramfs -u

Before rebooting to see if this is working, test that the client effectively is able to decrypt the password by receiving it from the server with this:

sudo /usr/lib/x86_64-linux-gnu/mandos/plugins.d/mandos-client --pubkey=/etc/keys/mandos/pubkey.txt --seckey=/etc/keys/mandos/seckey.txt --connect=<ServerIP>:<ServerPort>; echo

This should output the password.

Reboot and check that it works!

Debug

If it doesn’t work as expected, server side you can set debug = True in /etc/mandos/mandos.conf and watch the syslog.
On the client side you can try this command and watch the verbose output:

For a client and a server on the same network:

sudo /usr/lib/x86_64-linux-gnu/mandos/plugins.d/mandos-client --pubkey=/etc/keys/mandos/pubkey.txt --seckey=/etc/keys/mandos/seckey.txt --debug; echo

If you are in a WAN configuration:

sudo /usr/lib/x86_64-linux-gnu/mandos/plugins.d/mandos-client --pubkey=/etc/keys/mandos/pubkey.txt --seckey=/etc/keys/mandos/seckey.txt --connect=<serverIP>:<port>; echo

If everything is fine, issuing this command without the –debug flag should output the password you use to unlock the encrypted file system.

You can also check the config details of a client:

sudo mandos-ctl --dump-json box1

If it shows your client is disabled, check that the mandos server can reach your client via ports 22 and ping. If a client is off or unreachable for some time, mandos disables it (configurable in the client.conf file server side as shown previously).

You can also type this to check if your client is enabled:

# mandos-ctl 
Name      Enabled       Timeout       Last Successful Check 
box1      Yes           00:05:00      2016-10-11T18:17:47.147904

To Enable all clients:

sudo mandos-ctl --enable --all

To enable a specific client:

mandos-ctl --enable box1