Spinnaker HA Setup

The information below was written for a previous version of Armory Spinnaker (v1.13 and earlier). Please look here for documentation on the latest version.

Setting up Polling and Nonpolling

For certain components you’ll only want a single instance running on an ASG on “polling” mode. Namely Igor and Echo which need to run on a single instance so that multiple trigger events are not sent to Spinnaker and issuing multiple events for the same build.

To set this up, edit your Armory Spinnaker deploy pipeline so it looks like this.

Note, the cloud_details set to nonpolling is required. On boot, Armory Spinnaker will use /etc/default/server-env to determine which group this instance belongs to.

“Disable Poller” stage

Armory includes a service (Lighthouse) that will wait for existing jobs to complete before shutting itself down. If Fiat is enabled, make sure that you have configured a service account.

“Deploy to Prod nonpolling”

Setup your deploy stage to deploy a nonpolling cluster. You can scale this cluster up to as needed.

“Deploy to Prod Polling”

Deploy only 1 instance.

Using a isolated Redis for each subservice

Spinnaker subservices can share a single Redis. However here are some reasons why you may want to have separate Redis’s:

  • Subservice isolation
  • Redis performance issues, which allows you to scale each Redis individually

Here’s an example for Clouddriver, it’s the same pattern for other subservices:

$ cat /opt/spinnaker/env/prod.env

$ cat /opt/spinnaker/config/clouddriver-redis.yml
  connection: ${CLOUDDRIVER_REDIS_URL:redis://localhost:6379}

Validate the setup is correct

Look through your logs for to make sure the nonpolling instances don’t poll any resources. Here’s some examples:

For igor, only the polling instance should have log lines like:

INFO 1 --- [readScheduler-2] c.n.s.igor.jenkins.JenkinsBuildMonitor   : Polling cycle started: Fri Jun 08 18:15:35 GMT 2018
INFO 1 --- [readScheduler-2] c.n.s.igor.jenkins.JenkinsBuildMonitor   : Polling cycle done in 1359ms

For echo, only the polling instance should have this line:

INFO 1 --- [pool-5-thread-1] .s.e.s.a.p.i.PipelineConfigsPollingAgent : Running the pipeline configs polling agent...