I would like to setup an as simple as possible encryption at rest solution for a remote dedicated server. The host I am using will provide physical disks and there is no off the shelf solution for encrypted block level storage.
I was thinking of the following setup. Keep the primary server partition unencrypted so the server can boot without a key being provided to it. Create an encrypted volume via LUKs as an additional partition, encrypt it with a key, and then encrypt that key with a passphrase and store it on the server. Then I'de author a script to mount the volume given the passphrase via SSH (using nohup or screen to ensure the mount persists after the SSH session is closed), keeping the decrypted key in memory at all times. The encrypted volume would contain all the sensitive data as well as the software needed to run my services.
Now the next thing which worries me is swap, if its enabled then all of those secrets could be cached on the root partition unencrypted, so one solution would be to disable swap but it is useful to me as my services are memory hungry and could temporarily spike memory usage beyond what is available physically.
Before I have used encrypted volumes and swap files on encrypted volumes to deal with this but this was on servers I had physical access to so could leverage either manually entering luks passwords or using usb keys, neither is a possible solution in my case.
I was wondering what is the behaviour of having a swap file on the encrypted volume or an entire encryted swap volume that is not mounted at boot. What happens when the server reboots and cannot find this swap file / volume. I'de think it would prevent it booting but is there some fstab setting (like nofail or noauto) which would enable the server to boot without swap and then enable it with my mount script (have it turn swap on post decryption and mount) but in a way such that if that swap file is unavailable during boot it still boots but with swap off. I hope this makes sense.
