Secrets with Vault

This document describes how to set up Spinnaker secrets in Hashicorp’s Vault. In this example, we’ll be using the default KV secret engine called secret and will be storing GitHub credentials, a kubeconfig file and a Java keystore for SAML SSO.

Authorization

We currently support two methods of authentication with Vault servers.

You’ll need to configure Vault to authenticate with Kubernetes per our Vault Configuration Guide or Hashicorp’s documentation.

Note: If multiple clusters need to access the same Vault server, you’ll need to use the -path flag and give each cluster a different path name. This becomes <cluster auth path> in the example below. If using just one cluster, you can use the default vault auth enable kubernetes command, in which case your path will be kubernetes.

After configuring authentication on the Vault side, use the following hal commands to enable Vault secrets in Spinnaker:

hal armory secrets vault enable
hal armory secrets vault edit \
    --auth-method KUBERNETES \
    --url <Vault server URL>:<port, if required> \
    --role <k8s role with access to Vault> \
    --path <k8s cluster path> (*optional*, default is 'kubernetes')

2. Token authentication

This method is not recommended but it is supported if you choose. We recommend this for testing and development purposes only. For token authentication, you’ll need to have a VAULT_TOKEN environment variable set for halyard and each of the services.

Use the following hal commands to enable Vault secrets using token auth:

hal armory secrets vault enable
hal armory secrets vault edit \
    --auth-method TOKEN \
    --url <Vault server URL>:<port, if required>

Configuring Halyard to use Vault secrets

Note: You’ll need to be running Armory Halyard version 1.5.1 or later.

sudo update-halyard --version 1.5.1

Halyard will need access to the Vault server in order to decrypt secrets for validation and deployment. While the Spinnaker services are configured through ~/.hal/config, the Halyard daemon has its own configuration file found at /opt/spinnaker/config/halyard.yml. The contents of your file may look different than this example, but just make sure to add the secrets block somewhere at the root level.

If you’re running Halyard locally, you can use Token auth method. Set your VAULT_TOKEN environment variable and add the secrets block to halyard.yml like so:

halyard:
  halconfig:
    ...

spinnaker:
  artifacts:
    ...

secrets:
  vault:
    enabled: true
    url: <Vault server URL>
    authMethod: TOKEN

Or if you’re running Halyard in Kubernetes, you can have Halyard use Kubernetes auth:

halyard:
  halconfig:
    ...

spinnaker:
  artifacts:
    ...

secrets:
  vault:
    enabled: true
    url: <Vault server URL>
    authMethod: KUBERNETES
    role: <k8s role>
    path: <k8s cluster path>

Then restart the daemon (you’ll only need to do this the first time):

hal shutdown

Your next hal command will automatically bring the daemon back up if you’re running Halyard locally. If it’s running within a docker container, you’ll need to mount the volume containing the updated halyard.yml and restart the container. And if Halyard is in Kubernetes, you’ll need to restart the pod.

Storing secrets

To store a file, simply prepend the file path with @. It will accept relative paths but cannot resolve ~:

vault kv put secret/spinnaker/kubernetes config=@path/to/kube/config

The command above stores a single key-value pair at the secret/spinnaker/kubernetes path. Any updates to that path will replace the existing values even if using a different key! In order to store multiple secrets at the same path, it must be done in a single command, like so:

vault kv put secret/spinnaker/github password=<password> token=<token>

Otherwise, just store different secrets at different paths, like we’re doing in these examples.

Make sure to base64 encode any binary files:

base64 -i saml.jks -o saml.b64
vault kv put secret/spinnaker/saml base64keystore=@saml.b64

Referencing secrets

Now that secrets are safely stored in Vault, you’ll reference them in config files with the general syntax below. The format for referencing string values and files is the same, with the exception of the additional b:true parameter for a base64 encoded file.

encrypted:vault!e:<secret engine>!n:<namespace>!k:<key>!b:<is base64 encoded?>

Note: The specific parameters, e.g. e:<engine>, k:<key>, etc, can be in any order. Only b:true is optional.

For example, to reference the GitHub password:

encrypted:vault!e:secret!n:spinnaker/github!k:password

And the same for referencing a file:

encrypted:vault!e:secret!n:spinnaker/kubernetes!k:config

Binary files

Important! Spinnaker needs to base64 decode binary files on the way out of Vault, so make sure to add the b:true to the encrypted syntax. For example, to reference the Java keystore:

encrypted:vault!e:secret!n:spinnaker/saml!k:base64keystore!b:true

Supported Versions

Vault secrets are supported end-to-end in Armory Spinnaker 2.4.0 and with partial service support in 2.3.7.