v2.25.0 Armory Continuous Deployment Release (Spinnaker™ v1.25.3)

Release notes for Armory Continuous Deployment

2021/03/25 Release Notes

Note: If you’re experiencing production issues after upgrading Spinnaker, rollback to a previous working version and please report issues to http://go.armory.io/support.

For information about what Armory supports for this version, see the Armory Continuous Deployment v2.25 compatibility matrix.

Required Halyard or Operator version

To install, upgrade, or configure Armory 2.25.0, use one of the following tools:

  • Armory-extended Halyard 1.12 or later

  • Armory Operator 1.2.6 or later

    For information about upgrading, Operator, see Upgrade the Operator.

Security

Armory scans the codebase as we develop and release software. For information about CVE scans for this release, see the Support Portal. Note that you must be logged in to the portal to see the information.

Breaking changes

Breaking changes are kept in this list for 3 minor versions from when the change is introduced. For example, a breaking change introduced in 2.21.0 appears in the list up to and including the 2.24.x releases. It would not appear on 2.25.x release notes.

Suffix no longer added to jobs created by Kubernetes Run Job stage

Spinnaker no longer automatically appends a unique suffix to the name of jobs created by the Kubernetes Run Job stage. Prior to this release, if you specified metadata.name: my-job, Spinnaker updates the name to my-job-[random-string] before deploying the job to Kubernetes. As of this release, the job’s name will be passed through to Kubernetes exactly as supplied.

To continue having a random suffix added to the job name, set the metadata.generateName field instead of metadata.name, which causes the Kubernetes API to append a random suffix to the name.

This change is particularly important for users who are using the preconfigured job stage for Kubernetes or are sharing job stages among different pipelines. In these cases, jobs often running concurrently, and it is important that each job have a unique name. In order to retain the previous behavior, manually update your Kubernetes job manifests to use the generateName field.

Previously, this behavior was opt-in.

Impact

As of Armory 2.22, this behavior is the default. Users can still opt out of the new behavior by setting kubernetes.jobs.append-suffix: true in clouddriver-local.yml. This causes Spinnaker to continue to append a suffix to the name of jobs as in prior releases.

The ability to opt out of the new behavior will be removed in Armory 2.23 (OSS 1.23). The above setting will have no effect, and Spinnaker will no longer append a suffix to job names. We recommended that 2.22 users note which jobs are using the old behavior and prepare to remove the setting before upgrading to Armory 2.23 in the future.

Introduced in: Armory 2.22

Zombie Executions

Starting in Spinnaker 2.23.0, ManifestForceCacheRefreshTask was removed, as Kubernetes manifest related stages now do live lookups. While upgrading to Spinnaker 2.23.0 or later, if there is a running pipeline that contains a Kubernetes manifest related stage, it becomes a zombie execution. This causes Orca, Spinnaker’s orchestration service, to fail to complete any Kubernetes manifest related stage in that pipeline.

Workarounds:

To resolve the issue, cancel any zombie executions. For information about how to cancel them, see the Orca Zombie Execution runbook.

Affected versions: 2.23.0 and later

ManifestForceCacheRefreshTask removed from Orca

When you upgrade to 2.23.0 or later, you might encounter the following error:

2021-01-29 23:57:19.691 ERROR 1 --- [    scheduler-2] c.netflix.spinnaker.q.redis.RedisQueue   : Failed to read message 8f072714f1df6dbf3af93a4f4fe4cae2, requeuing...
com.fasterxml.jackson.databind.JsonMappingException: No task found for 'com.netflix.spinnaker.orca.clouddriver.tasks.manifest.ManifestForceCacheRefreshTask' (through reference chain: com.netflix.spinnaker.orca.q.RunTask["taskType"])

The ManifestForceCacheRefreshTask task is no longer a required task when deploying a manifest. In earlier releases, forcing the cache to refresh was part of the deployment process for manifests. Because of this change, if a task was running or retried before the upgrade, the error shows up in logs as an exception.

Workaround

Before starting, make sure that you have access to the Redis instance that Orca uses.

To resolve this issue, delete the message from the queue:

  1. Verify that there are pipeline execution failure messages that contain ManifestForceCacheRefreshTask:

    Redis

    hgetall orca.task.queue.messages

    The command returns information similar to the following:

    1) "93ac65e03399a4cfd3678e1355936ab2"
2) "{\"kind\":\"runTask\",\"executionType\":\"PIPELINE\",\"executionId\":\"01EVFCCDG3Q2209E0Z1QTNC0FS\",   \"application\":\"armoryhellodeploy\",\"stageId\":\"01EVFCCDG3TJ7AFPYEJT1N8RDJ\",\"taskId\":\"5\",\"taskType\":\"com.netflix.spinnaker.   orca.clouddriver.tasks.manifest.ManifestForceCacheRefreshTask\",\"attributes\":[{\"kind\":\"attempts\",\"attempts\":1}],   \"ackTimeoutMs\":600000}"

  2. Delete the message(s):

    Redis

    hdel orca.task.queue.messages

    The command returns information similar to the following:

    93ac65e03399a4cfd3678e1355936ab2
(integer) 1

Known issues

Bake failures

The Packer version included with Rosco disregards package overrides that use the -var-file= option. This may cause bakes to fail.

Affected versions: 2.22.2 and later

Lambda UI issue

There is a UI bug related to the caching agent that prevents Lambda functions from being displayed in the UI when there are no other clusters associated with the Application. In other words, in order for the function to show up in “Functions” tab, there needs to be a cluster (such as an AWS ASG/EC2 instance) deployed for that application.

Affected versions: 2.23.0 (1.23.0) - 2.26.2 Fixed version: 2.26.3

pipelineID for Pipelines-as-Code

There is a known issue where the pipelineID function does not work the first time, which causes pipelines to not be updated. Subsequent changes to the dinghyfile update the pipelines correctly.

Affected versions: 2.21.1 - 2.26.0 Fixed version: 2.26.1

Git repo artifact provider cannot checkout SHAs

Only branches are currently supported. For more information, see 6363.

Server groups

There is a known issue where you cannot edit AWS server groups with the Edit button in the UI. The edit window closes immediately after you open it.

Workaround: To make changes to your server groups, edit the stage JSON directly by clicking on the Edit stage as JSON button.

Fixed issues

  • Fixed an issue where Pipelines-as-Code fails unexpectedly when updating modules.
  • Fixed an issue where usernames were not deduplicated properly because of case sensitivity.
  • Fixed a start up issue that occurs when spinnaker.base-url.www is not set
  • Fixed an issue that occurs when a user enters a value for the Manifest field and then deletes it. This caused the resulting stage config to use "manifest":"". The manifest now defaults to spinnaker.yml.
  • Fixed an issue where a Jenkins stage stalls with a “Wait For Jenkins Job Start” message if the stage is canceled and restarted within a certain time frame.

Highlighted updates

Artifacts

Improved performance for Git repo artifacts

Armory now uses the git binary instead of jgit. This change adds support for shallow clones, allowing for faster downloads. No changes to existing accounts configuration are needed. Note that there is a known issue related to this change: it does not currently support checkouts using a SHA.

Filter for caching Docker registries

Docker accounts now support a new configuration parameter: repositoriesRegex. When used, only the repositories that match the pattern get cached. This is useful for Docker accounts that have a large number of repositories or accounts where repositories are added frequently. As repositories are added, you do not need to update the list of repositories that you configured.

This feature requires either Armory Operator 1.2.6 (or later) or Armory-extended Halyard 1.12 (or later). For more information, see Docker Artifacts.

Authz

Access to a service account is now granted if any authorization predicate allows it. The previous behavior

Baking images

The Packer version that is bundled with Rosco, the baking service, has been updated. Rosco now supports the AWS parameter assume_role in Packer templates.

CI system

Jenkins

The integration between Armory and Jenkins has been improved:

  • You can configure the orchestration service Orca so that it updates the description of a running Jenkins build and generates a backlink. For more information, see Enabling Backlinks from Jenkins to Spinnaker.
  • You can now update the job description of a running or completed Jenkins job through the API.

Clouddriver

Improved the performance of health checks for the Clouddriver pod by running them in different threads. Previously, the Clouddriver pod did not report a Ready state until after all Kubernetes accounts verified connectivity through a kubectl get namespaces call.

Deployment targets

Cloud Foundry

The following improvements have been made to the Cloud Foundry provider:

  • Armory now supports the ability to deploy Docker images to Cloud Foundry like any other application.
  • There is a new stage that supports more granular service bindings. More specifically, if a service needs additional information in order to make a binding successful, you can now customize that information in the Service Binding Stage.
  • Applications during deployment will now bind to their services before completely building the droplet. This allows the deployments to avoid an additional costly restage operation. Not all applications require restages after binding, but this change improves performance for applications that do require restages.

SpEL

Armory now supports the following for SpEL expressions:

  • resolvedArtifacts, which returns all the resolved stage artifacts
  • canaryConfigNameToId, which takes a canary config name and converts it to an ID

Spinnaker Community Contributions

There have also been numerous enhancements, fixes, and features across all of Spinnaker’s other services. See the Spinnaker v1.25.3 changelog for details.

Detailed updates

Bill Of Materials (BOM)

Here’s the BOM for this version.

Expand
version: 2.25.0
timestamp: "2021-03-25 09:28:32"
services:
    clouddriver:
        commit: de3aa3f0
        version: 2.25.3
    deck:
        commit: 516bcf0a
        version: 2.25.3
    dinghy:
        commit: 522e67e5
        version: 2.25.1
    echo:
        commit: 3a098acc
        version: 2.25.2
    fiat:
        commit: ca75f0d0
        version: 2.25.3
    front50:
        commit: 502b753e
        version: 2.25.2
    gate:
        commit: "47352833"
        version: 2.25.5
    igor:
        commit: 252dbd5c
        version: 2.25.2
    kayenta:
        commit: "72616529"
        version: 2.25.2
    monitoring-daemon:
        version: 2.25.0
    monitoring-third-party:
        version: 2.25.0
    orca:
        commit: 53f48823
        version: 2.25.2
    rosco:
        commit: 272f4f82
        version: 2.25.2
    terraformer:
        commit: 5dcae243
        version: 2.25.0
dependencies:
    redis:
        version: 2:2.8.4-2
artifactSources:
    dockerRegistry: docker.io/armory

Armory

Armory Deck - 2.24.3…2.25.3

  • feat(terraformer): hide sections when output is selected (#712)
  • chore(deps): bump to new OSS 1.24.2 bom (#713)
  • feat(envuuid): send env uuid in header for dinghy events (#715)
  • feat(dinghy): add pull request URL (#718)
  • Fix(Terraform): Console Output UX (#723)
  • fix(terraformer): console logs output enhancement (#725)
  • ES-581: Fixes hamburger menu alignment (#738)
  • feat(dinghy): add new content tab to show processed dinghyfiles. (#744)
  • feat(dinghyEvents): add author information, refactor code (#745)
  • fix(terraformer): ensure Ignore Locking sets the correct json value (#749)
  • chore(build): update to 1.25.3 (#761) (#762)

Armory Clouddriver - 2.24.23…2.25.3

  • feat(docker): adding ecr utility to get an ecr token (#235)
  • fix(dep/springboot): downgrade spring-boot to SR4 (#249)
  • fix(config): reverted spring.profile sytax to SR4 syntax (#251)
  • fix(dependencies): downgrade resilience4j to match OSS (#253)
  • fix(dep/springboot): revert downgrade spring-boot to SR4 (#249) (#257)
  • chore(build): use Armory commons BOM, remove some unused constraints (#271)
  • chore(dependencies): exclude tencent, huawei, oracle, yandex, move junit/logback to bom (#274)
  • chore(build): bump google sdk to fix CVE-2019-17638 and CVE-2020-27216 (#280)
  • feat(git): Added git binaries to docker images (#311) (#312)

Armory Orca - 2.24.13…2.25.2

  • chore(build): use armory commons BOM (#208)
  • chore(dependencies): use armory commons bom (#212)

Armory Rosco - 2.24.12…2.25.2

  • fix(fargate-job-executor): ensure that aws credentials are refreshed … (#164)
  • fix(fargate): add default to awsIamRole (#167)
  • chore(build): use armory commons bom (#185)
  • feat(tests): add coveralls test coverage reporting (#174)
  • chore(dependencies): use armory commons bom (#192)
  • fix(test): fix integration tests (#194)
  • chore(fargate): remove caller from remote jobs and update int test infra to us… (#196)
  • fix(build): revert change to gradle GHA build file (#198)
  • fix(fargate-job-executor): Initialize logs to a default message when fargate jobs are initing, fall back to stored logs when there is a failure to get logs from CW (#197)
  • fix(packer-command-factory): Adding a Fargate specific PackerCommandF… (#200)
  • chore(dependencies): removes webjars. (#206)

Dinghy™ - 2.24.10…2.25.1

  • feat(sqlsupport): full sql support (#300)
  • feat(parser): sprig functions support (#311)
  • fix(liquibase): liquibase will be executed by dinghy since it has the secrets (#314)
  • fix(validations): pr validations fixes (#317)
  • feat(pullrequest-url): add pullrequest url for dinghy events. (#323)
  • feat(appmetadata): appmetadata in spec will add those fields to applications (#324)
  • fix(liquibase): Dinghy 2.24 not working in SQL mode (#325)
  • fix(pipelines): Add parameter for SpEl Expression version (#340)
  • feat(Post_multiple_comments_for_lengthy_logs): Breaks up long log messages into multiple GitHub comments to prevent a 422 (#347)
  • chore(infrastructure): manage infrastructure for centralized Dinghy with terraform. (#350)
  • feat(add_new_relic_support): Added the New Relic agent to dinghy and … (#352)
  • chore(build): bump go-yaml-tools (#343)
  • fix(crash_on_module_updates): adds check to verify a 200 status code on comment post prior to postin the reaction (#358)

Armory Gate - 2.24.14…2.25.5

  • chore(dependencies): use armory commons bom (#223)
  • chore(dependencies): exclude test libraries from runtime (#232)
  • test(web): Test health format, remove groovy, spock (#234)
  • fix(test/url): Add urls to tests Spinnaker bump version1615369808 (#257)
  • fix(keel): add default keel endpoints. (#258) (#259)

Terraformer™ - 2.24.4…2.25.0

  • chore(metrics): Migrate to Prometheus metrics with go-metrics (#301)
  • fix(cve): remove scp until OpenSSH_8.4+ is available (#307)
  • feat(metrics): Add Prometheus metrics for artifacts fetched, artifact latency, artif… (#308)
  • Revert “feat(metrics): Add Prometheus metrics for artifacts fetched, artifact latency, artif… (#308)” (#315)
  • fix(metrics): Fix panic caused by disabled metrics in Terraformer config (#317)
  • feat(metrics): add redis persistence metrics (#313)
  • feat(metrics): add persistence loop and job executor metrics (#322)
  • chore(gitrepo): add more tests for tar writing from clouddriver (#310)
  • chore(int-tests): update clouddriver version (#320)
  • feat(tests): add coveralls test coverage (#312)
  • feat(metrics): add redis key miss metrics and fix redis latency metric (#324)
  • fix(tests): gitignore profile.cov file (#323)
  • feat(metrics): Improve executor’s oldest job accuracy (#326)
  • chore(metrics): update go-spec to 0.0.3 so gauge metrics won’t be prefixed with the host name (#327)
  • fix(cve): Alpine > 3.13 to fix glib and openssh cves (#328)
  • fix(profiles): support passing vars via TF_VAR (#339)
  • chore(logs): go-spec JSON logger refactor (#343)
  • fix(tests): fix code coverage badge on readme (#344)
  • chore(build): bump go-yaml-tools version (#350)
  • fix(build): kick off build (#370)

Armory Igor - 2.24.9…2.25.2

  • chore(build): use armory commons bom (#177)
  • feat(tests): add coveralls test coverage reporting (#164)
  • chore(build): rely on armory-commons (#183)
  • chore(build): update scan action version (#195)

Armory Echo - 2.24.11…2.25.2

  • feat(dinghy): send environment uuid to dinghy (#263)
  • chore(build): use armory-commons BOM (#282)
  • chore(dependencies): move junit/logback to bom (#285)

Armory Fiat - 2.24.12…2.25.3

  • chore(build): use armory commons BOM (#156)
  • chore(dependencies): use armory commons bom (#162)
  • Update gradle.properties (#193)

Armory Front50 - 2.24.12…2.25.2

  • chore(build): use armory commons BOM (#202)
  • feat(tests): add coveralls test coverage reporting (#193)
  • chore(dependencies): use armory commons bom (#208)
  • chore(build): upgrade scan action version (#228)

Armory Kayenta - 2.24.14…2.25.2

  • fix(cve): fix 2020-5408 force security-core to 5.2.4 (#163)
  • Update gradle.yml
  • fix(test): re adding dynatrace on integration tests (#171)
  • chore(build): use armory commons BOM (#183)
  • chore(dependencies): use armory commons bom (#189)
  • fix(dynatrace): fix parse of datapoints values (#204)
  • fix(dynatrace): add test to verify values conversion (#209)
  • feat(dynatrace): adding support for resolution parameter (#224) (#227)

Last modified May 26, 2023: (a7d5a9eb)