HashiCorp Vault with Docker Compose: Secure Secrets Management

Files

First, create a directory to store the configuration files. I will use /opt/vault.

Create the docker-compose.yml:

YAML

94 lines

services:
  init_vault:
    image: alpine:latest
    container_name: init_vault
    group_add:
      - 3001
    volumes:
      - vault-data:/vault/data:rw
      - certs:/tmp/certs/:rw
      - secrets:/secrets:rw
    command: >
      /bin/sh -c "apk add --no-cache openssl && \
      openssl req -x509 -newkey rsa:4096 -sha256 -days 365 \
      -nodes -keyout /tmp/certs/vault-key.pem -out /tmp/certs/vault-cert.pem \
      -subj '/CN=vault' \
      -addext 'subjectAltName=DNS:vault,IP:127.0.0.1' && \
      chown -R 100:3001 /vault/ /tmp/certs /secrets && \
      chmod -R 0750 /secrets/ && \
      exit 0"

  vault:
    image: "hashicorp/vault:latest"
    restart: unless-stopped
    container_name: vault
    ports:
      - "8201:8201"
      - "8200:8200"
    privileged: true
    depends_on:
      init_vault:
        condition: service_completed_successfully
    environment:
      VAULT_ADDR: "https://vault:8200"
      VAULT_CACERT: "/vault/certs/vault-cert.pem"
      VAULT_SKIP_VERIFY: true
    cap_add:
      - IPC_LOCK
    volumes:
      - vault-data:/vault/data:rw
      - ./agent-policies.hcl:/policies/agent-policies.hcl
      - ./vault-config.hcl:/vault/config/vault-config.hcl:ro
      - certs:/vault/certs:ro
    healthcheck:
      test:
        [
          "CMD-SHELL",
          'vault status -format=json 2>/dev/null | grep -q ''"sealed": false''',
        ]
      interval: 10s
      timeout: 5s
      retries: 1000
      start_period: 20s
    command: vault server -config=/vault/config/vault-config.hcl -log-level="info"

    # run docker exec vault vault status
    # then
    # run docker exec vault vault operator init -key-shares=3 -key-threshold=2

  vault-agent:
    image: "hashicorp/vault:latest"
    restart: unless-stopped
    container_name: vault-agent
    depends_on:
      vault:
        condition: service_healthy
    environment:
      VAULT_ADDR: "https://vault:8200"
      VAULT_CACERT: "/vault/certs/vault-cert.pem"
      VAULT_SKIP_VERIFY: true
    cap_add:
      - IPC_LOCK
    volumes:
      - certs:/vault/certs:ro
      - secrets:/secrets:rw
      - ./role-id.txt:/role-id:ro
      - ./secret-id.txt:/secret-id:ro
      - ./agent-config.hcl:/vault/config/agent-config.hcl:rw
      - ./templates:/templates:ro
    command: vault agent -config=/vault/config/agent-config.hcl -log-level="info"

volumes:
  certs:
  vault-data:
  secrets:
    driver: local
    driver_opts:
      type: none
      device: /path/to/mount/secrets/in/host
      o: bind

networks:
  default:
    name: network
    driver: bridge

The role-id.txt and secret-id.txt files will be created later, and you need to specify the device path for the secrets volume.

The group_add value 3001 is the GID of my Docker group. I use this GID for Docker. If you don't add it, you won't be able to access the environment files when they are created.

Create the vault-config.hcl:

HCL

22 lines

api_addr                = "https://vault:8200"
cluster_addr            = "https://vault:8201"
cluster_name            = "vault_cluster"
disable_mlock           = true
ui                      = true

max_lease_ttl       = "2h"
default_lease_ttl   = "20m"

raw_storage_endpoint = "true"
disable_printable_check = "true"

listener "tcp" {
  address       = "0.0.0.0:8200"
  tls_cert_file = "/vault/certs/vault-cert.pem"
  tls_key_file  = "/vault/certs/vault-key.pem"
}

backend "raft" {
  path    = "/vault/data"
  node_id = "vault_1"
}

The agent-config.hcl contains all the Vault Agent configuration:

HCL

37 lines

auto_auth {
  method {
    type = "approle"
    config = {
      role_id_file_path = "/role-id",
      secret_id_file_path = "/secret-id",
      remove_secret_id_file_after_reading = false
    }
  }

  sink {
    type = "file",
    config = {
      path = "/tmp/token"
    }
  }
}

template_config {
  static_secret_render_interval = "1m"
  exit_on_retry_failure         = true
  max_connections_per_host      = 10
}

vault {
  address = "https://vault:8200"
  ca_cert = "/vault/certs/vault-cert.pem"
  tls_server_name = "vault"
  tls_skip_verify = true
}

template {
  source = "/path/to/source"
  destination = "path/to/destination"
  perms = "0750"
  error_on_missing_key = true
}

Finally, create agent-policies.hcl:

HCL

3 lines

path "kv/*" {
  capabilities = ["read"]
}

You can see here how to create your policy file.

I recommend setting the owner with chown so Vault can read and write the files inside the container:

BASH

1 line

sudo chown -R 100:100 /path/to/vault/files

Running

Now run the following command to start the services:

BASH

1 line

docker compose up -d

After the Vault container has started, we need to execute some commands to initialize and unseal Vault, enable AppRole, and create the KV secrets we need.

Init

To initialize the operator, run:

BASH

1 line

docker exec vault vault operator init -key-shares=<number> -key-threshold=<number>

key-shares is the number of unseal keys that will be generated.
key-threshold is how many keys are required to unseal Vault. Every time Vault is restarted, it will need to be unsealed again.

Store these keys in a secure place. Without them, you cannot unseal Vault.

To unseal Vault, run the following command with one of the unseal keys the required number of times:

BASH

1 line

docker exec vault vault operator unseal <key>

You will also receive the Root Token. This is required to log in to Vault and make configuration changes.

After initializing the operator, log in using your root token:

BASH

1 line

docker exec vault vault login <token>

KV

The first thing we need to do is create our secrets. First, enable KV with:

BASH

1 line

docker exec vault vault secrets enable -path=<path/to/secrets> kv-v2

You can create one KV mount for each service you have, or a single one for all services.

For example, you can create mounts named web-app, backend, or envs, and store each service as a path inside envs. Choose the structure that works best for you.

I recommend adding a prefix to all your KV paths, such as kv/your-path. This makes creating policies easier later.

After creating the KV mount, you can add your secrets to it using the UI or with this command:

BASH

1 line

docker exec vault vault kv put -mount=path/to/secrets <name> key1=value1 key2=value2

If you want, I have a script that converts a .env file into a command for creating all the secrets in KV here.

AppRole

This is necessary to allow your Vault Agent to authenticate with Vault.

First, write the policy we created earlier (change my-policy):

BASH

1 line

docker exec vault vault policy write my-policy /policies/agent-policies.hcl

Then enable the AppRole authentication method:

BASH

1 line

docker exec vault vault auth enable approle

Next, create the role that your agent will use. You can choose any name for the role (change my-role):

BASH

5 lines

docker exec vault vault write auth/approle/role/my-role \
    token_type=batch \
    token_ttl=20m \
    token_max_ttl=2h \
    token_policies="my-policy"

Use this command to save the role ID to a file that will be used by the agent. Remember to change the path to your Vault files:

BASH

1 line

docker exec vault vault read -field=role_id auth/approle/role/agent/role-id > /opt/vault/role-id.txt

The other thing we need is the secret ID. Again, remember to change the path to your Vault files:

BASH

1 line

docker exec vault vault write -field=secret_id -f auth/approle/role/agent/secret-id > /opt/vault/secret-id.txt

Agent Template

If you want to convert your secrets into a .env file, for example, you need to create a template using Consul Template markup.

You can learn more here.

After creating your templates, put them in a folder and configure Docker Compose to mount the templates into the Vault Agent. They use the .ctmpl extension and can look like this:

CTMPL

3 lines

{{ with secret "kv/data/envs" -}}
KEY={{ .Data.data.KEY }}
{{- end }}

After doing this, recreate the vault-agent service in Docker.

Now the .env file will be generated inside a folder that we can access through Docker. If you mount the /secrets volume to /secrets, you can access the generated files on the host machine under /secrets.

You may need to change the permissions to read the file. Test whether the user running Docker can access the .env file:

BASH

1 line

cat /path/to/env

When I made this tutorial, I had some problems with the UI where the secrets were not showing up. To solve this, I created a new token and applied the default policies. Remember to edit the policies and add kv/*. They should contain these lines:

HCL

3 lines

path "kv/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}

You can generate a new token with:

BASH

1 line

docker exec vault vault token create -policy=default -ttl=2h