Clouddriver with RDBMS

Since version 2.5.x (OSS 1.14.x), Clouddriver can store its data (task, infrastructure, etc) in a MySQL compatible database. Similar to Orca, the main advantage of doing this is to improve performance and remove Redis as a single point of failure.

Base Configuration

You can find a complete description of the options in the open source documentation. SQL is not currently supported in Halyard’s main configuration but can be setup in <HALYARD>/<DEPLOYMENT>/profiles/clouddriver-local.yml.

Database Setup

You can skip this step if you create the database during provisioning - for instance with Terraform.

Once you’ve provisioned your RDBMS and ensured connectivity with Spinnaker, you need to create the database:

CREATE SCHEMA `clouddriver` DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;

Then we’ll grant authorization to the clouddriver_service and clouddriver_migrate users:


  GRANT
    SELECT, INSERT, UPDATE, DELETE, CREATE, EXECUTE, SHOW VIEW
  ON `clouddriver`.*
  TO 'clouddriver_service'@'%';

  GRANT
    SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, REFERENCES, INDEX, ALTER, LOCK TABLES, EXECUTE, SHOW VIEW
  ON `orca`.*
  TO `clouddriver_migrate'@'%';

The above configuration grants authorization from any host. You can restrict it to the cluster in which Spinnaker runs by replacing the % with the IP address of Clouddriver pods from MySQL.

Deployment

You have two options for deploying Clouddriver with MySQL: a simpler deployment, which involves downtime, or a three-step method that avoids downtime. Pick the method that best fits your requirements.

Simple Deployment

If you are not worried about downtime or if Spinnaker is not currently executing any pipelines, you can run a simply deployment by applying the configuration in <HALYARD>/<DEPLOYMENT>/profiles/clouddriver-local.yml:

sql:
  enabled: true
  taskRepository:
    enabled: true
  cache:
    enabled: true
    # These parameters were determined to be optimal via benchmark comparisons
    # in the Netflix production environment with Aurora. Setting these too low
    # or high may negatively impact performance. These values may be sub-optimal
    # in some environments.
    readBatchSize: 500
    writeBatchSize: 300
  scheduler:
    enabled: true
  connectionPools:
    default:
      # additional connection pool parameters are available here,
      # for more detail and to view defaults, see:
      # https://github.com/spinnaker/kork/blob/master/kork-sql/src/main/kotlin/com/netflix/spinnaker/kork/sql/config/ConnectionPoolProperties.kt
      default: true
      jdbcUrl: jdbc:mysql://your.database:3306/clouddriver
      user: clouddriver_service
      # password: depending on db auth and how spinnaker secrets are managed
    # The following tasks connection pool is optional. At Netflix, clouddriver
    # instances pointed to Aurora read replicas have a tasks pool pointed at the
    # master. Instances where the default pool is pointed to the master omit a
    # separate tasks pool.
    tasks:
      user: clouddriver_service
      jdbcUrl: jdbc:mysql://your.database:3306/clouddriver
  migration:
    user: clouddriver_migrate
    jdbcUrl: jdbc:mysql://your.database:3306/clouddriver

redis:
  enabled: false
  cache:
    enabled: false
  scheduler:
    enabled: false
  taskRepository:
    enabled: false

No Downtime Deployment

To avoid downtime for your deployment, use the following three steps:

Step 1: Warm up the cache

The first step is to start a Clouddriver that is not accessible from other services to validate the installation and warm up the cache.

You can do it manually or by using the following script. Make sure tables are properly created and being populated by these instances of Clouddriver.

Step 2: Use MySQL to back new tasks

After waiting a few minutes (from 2 to 10 minutes depending on how many accounts are connected), we’ll update Spinnaker to use MySQL but remain aware of task statuses in Redis.

We’re deploying Spinnaker with the following configuration in clouddriver-local.yml:

sql:
  enabled: true
  taskRepository:
    enabled: true
  cache:
    enabled: true
    # These parameters were determined to be optimal via benchmark comparisons
    # in the Netflix production environment with Aurora. Setting these too low
    # or high may negatively impact performance. These values may be sub-optimal
    # in some environments.
    readBatchSize: 500
    writeBatchSize: 300
  scheduler:
    enabled: true
  connectionPools:
    default:
      # additional connection pool parameters are available here,
      # for more detail and to view defaults, see:
      # https://github.com/spinnaker/kork/blob/master/kork-sql/src/main/kotlin/com/netflix/spinnaker/kork/sql/config/ConnectionPoolProperties.kt
      default: true
      jdbcUrl: jdbc:mysql://your.database:3306/clouddriver
      user: clouddriver_service
      # password: depending on db auth and how spinnaker secrets are managed
    # The following tasks connection pool is optional. At Netflix, clouddriver
    # instances pointed to Aurora read replicas have a tasks pool pointed at the
    # master. Instances where the default pool is pointed to the master omit a
    # separate tasks pool.
    tasks:
      user: clouddriver_service
      jdbcUrl: jdbc:mysql://your.database:3306/clouddriver
  migration:
    user: clouddriver_migrate
    jdbcUrl: jdbc:mysql://your.database:3306/clouddriver

redis:
  cache:
    enabled: false
  scheduler:
    enabled: false

dualTaskRepository:
  enabled: true
  primaryClass: com.netflix.spinnaker.clouddriver.sql.SqlTaskRepository
  previousClass: com.netflix.spinnaker.clouddriver.data.task.jedis.RedisTaskRepository

Note: At this point, you can stop the pods you created in step 1. If you used the script above, just delete the spin-clouddriver-sql deployment.

Step 3: Remove Redis

After waiting a few minutes so that Redis tasks are no longer relevant, we finish by removing Redis entirely:

sql:
  enabled: true
  taskRepository:
    enabled: true
  cache:
    enabled: true
    # These parameters were determined to be optimal via benchmark comparisons
    # in the Netflix production environment with Aurora. Setting these too low
    # or high may negatively impact performance. These values may be sub-optimal
    # in some environments.
    readBatchSize: 500
    writeBatchSize: 300
  scheduler:
    enabled: true
  connectionPools:
    default:
      # additional connection pool parameters are available here,
      # for more detail and to view defaults, see:
      # https://github.com/spinnaker/kork/blob/master/kork-sql/src/main/kotlin/com/netflix/spinnaker/kork/sql/config/ConnectionPoolProperties.kt
      default: true
      jdbcUrl: jdbc:mysql://your.database:3306/clouddriver
      user: clouddriver_service
      # password: depending on db auth and how spinnaker secrets are managed
    # The following tasks connection pool is optional. At Netflix, clouddriver
    # instances pointed to Aurora read replicas have a tasks pool pointed at the
    # master. Instances where the default pool is pointed to the master omit a
    # separate tasks pool.
    tasks:
      user: clouddriver_service
      jdbcUrl: jdbc:mysql://your.database:3306/clouddriver
  migration:
    user: clouddriver_migrate
    jdbcUrl: jdbc:mysql://your.database:3306/clouddriver

redis:
  enabled: false
  cache:
    enabled: false
  scheduler:
    enabled: false
  taskRepository:
    enabled: false