Air-Gapped with the Armory Operator
This guide details how you can host the Armory Enterprise Bill of Materials (BOM) and Docker images, as well as the Armory Operator Docker images, in your air-gapped environment. The steps at a high level are:
- Clone the
spinnaker-kustomize-patchesrepo, which contains helper scripts as well as Kustomize patches.
- Deploy S3-compatible MinIO to store the BOM.
- Download the BOM.
- Copy the BOM to your MinIO bucket.
- Host Armory Enterprise Docker images in your private Docker registry.
- Download the Armory Operator.
- Host Armory Operator Docker images in your private Docker registry.
- Update Armory Operator configuration.
- Deploy the Armory Operator in your Kubernetes cluster.
Before you begin
- You are familiar with the Armory Operator and configuring Armory Enterprise using Kustomize patches.
- You have read the introduction to air-gapped environments.
- You have public internet access.
- You have administrator access to your Kubernetes cluster.
- You have created two namespaces in Kubernetes:
- You have access to a private Docker registry with credentials to push images.
- You have installed the AWS CLI.
- You have installed
yqversion 4+. This is used by helper scripts.
Armory maintains the
spinnakaker-kustomize-patches repo, which contains common configuration options for Armory Enterprise or Spinnaker as well as helper scripts. The patches in this repo give you a reliable starting point when adding and removing features.
Configuration in this repository is meant for Armory Continuous Delivery. To make it compatible with Spinnaker instead, apply the
To start, create your own copy of the
by clicking the
Use this template button:
If you intend to update your copy from upstream, use Fork instead. See Creating a repository from a template for the difference between Use this template and Fork.
Once created, clone this repository to your local machine.
Deploy MinIO for storage
Now that you have cloned the
spinnaker-kustomize-patches repo, you need to
create a storage bucket to host the BOM. MinIO is a good
choice for the bucket since it’s S3 compatible and runs as a pod in Kubernetes.
A ready to use MinIO component can be referenced in
# Your kustomization.yml file resources: - path/to/spinnaker-kustomize-patches/core/persistence/in-cluster/minio.yml
When you look at the content of the
minio.yml manifest, you see that MinIO
needs a secret key called
env: # MinIO access key and secret key - name: MINIO_ACCESS_KEY value: "minio" - name: MINIO_SECRET_KEY valueFrom: secretKeyRef: name: minio-secret-key key: minioAccessKey
minioAccessKey is stored in a Kubernetes secret called
You create the secret by adding a generator to your
secretGenerator: - name: minio-secret-key options: disableNameSuffixHash: true literals: - minioAccessKey=MyAccessKeyValue
Consult the kustomize documentation for more ways to configure secrets.
Deploy MinIO with your infrastructure:
kubectl apply -k .
Download the BOM
Decide which Armory Enterprise version you want to deploy. Check Armory Release Notes for the latest supported versions.
spinnaker-kustomize-patches/utilities/airgap directory contains helper scripts for air-gapped environments. Use
bomdownloader.sh to download the version of the Armory Enterprise BOM that you require.
bomdownloader.sh takes two command line parameters in the following order:
- Armory Enterprise version; for example, 2.27.0.
- The name of your Docker registry; for example,
The script creates a
halconfig folder, downloads the necessary files, and updates the BOM to use the Docker registry you specified. To download the BOM:
Switch to the
bomdownloader.sh <armory-version> <docker-registry>. For example, if you want to download the 2.25.0 BOM and your registry is
./utilities/airgap/bomdownloader.sh 2.25.0 my.jfrog.io/myteam/armory
Output is similar to:
download: s3://halconfig/versions.yml to halconfig/versions.yml download: s3://halconfig/bom/2.25.0.yml to halconfig/bom/2.25.0.yml download: s3://halconfig/profiles/clouddriver/2.25.3/clouddriver.yml to halconfig/profiles/clouddriver/2.25.3/clouddriver.yml ... Version 2.25.0 is ready to be uploaded to your private bucket. For example, with "aws cp --recursive" or "gsutil cp -m -r ..."
Inspecting your file system, you should see the new
halconfig folder. For example, if you specified Armory Enterprise v2.25.0, your file system should be:
halconfig ┣ bom ┃ ┗ 2.25.0.yml ┣ profiles ┃ ┣ clouddriver ┃ ┣ dinghy ┃ ┣ echo ┃ ┣ fiat ┃ ┣ front50 ┃ ┣ gate ┃ ┣ igor ┃ ┣ kayenta ┃ ┣ monitoring-daemon ┃ ┣ orca ┃ ┣ rosco ┃ ┗ terraformer ┗ versions.yml
Each subdirectory in the
profiles directory contains a
<service-name>.yml profile file.
If you need to change your Docker registry, you can manually edit the
<armory-version>.yml file located under
halconfig/bom. Update the value for the key
version: 2.25.0 timestamp: "2021-03-25 09:28:32" services: clouddriver: commit: de3aa3f0 version: 2.25.3 deck: commit: 516bcf0a version: 2.25.3 ... terraformer: commit: 5dcae243 version: 2.25.0 dependencies: redis: version: 2:2.8.4-2 artifactSources: dockerRegistry: my.jfrog.io/myteam/armory
Copy the BOM
With the MinIO pod running, copy your local BOM into your MinIO bucket.
Set environment variables:
minioAccessKeyvalue you created in the Deploy MinIO to host the BOM section.
export AWS_ACCESS_KEY_ID=minio export AWS_SECRET_ACCESS_KEY=<minioAccessKey>
Port forward the MinIO service:
kubectl port-forward svc/minio 9000 -n spinnaker
Copy your local BOM to MinIO:
aws s3 mb s3://halconfig --endpoint=http://localhost:9000 aws s3 cp --recursive halconfig s3://halconfig --endpoint=http://localhost:9000
Host the Armory Enterprise Docker images
There are two options for hosting the Docker images: 1) configure your Docker registry as a proxy for
docker.io/armory; or 2) download the images and push them to your private Docker registry.
docker.io/armory as a remote repository within your private Docker registry. If you are using JFrog Artifactory, you can follow the instructions in the Remote Docker Repositories section of the JFrog docs.
You can use the
imagedownloader.sh helper script in the
spinnaker-kustomize-patches/utilities/airgap directory to download and push the images to your private Docker registry.
The execution format is:
<armory-version> is the version you specified in the Download the BOM section.
When you run
imagedownloader.sh from the
spinnaker-patches-repository directory, the script looks for the downloaded BOM and proceeds to download, tag, and push the images for that particular version to the private Docker registry you specified when you ran
Download the Armory Operator
Download and unpack the latest Armory Operator release into the
spinnaker-kustomize-patches/operator folder. Run the following command from your
bash -c 'curl -L https://github.com/armory-io/spinnaker-operator/releases/latest/download/manifests.tgz | tar -xz'
You should see the following directory structure in the
operator ┣ deploy ┃ ┣ crds ┃ ┣ openshift ┃ ┣ operator ┃ ┗ spinnaker ┣ halyard-local.yml ┣ kustomization.yml ┣ patch-config.yaml ┗ patch-validations.yaml
Host the Armory Operator Docker images
You can find the
operatorimageupdate.sh script in
spinnaker-kustomize-patches/utilities/airgap. The script does the following:
- Downloads the Armory Operator Docker images and updates their names.
- Pushes the images to the Docker registry you specify in the command line.
- Updates the Armory Operator’s
kustomization.ymlwith the new image names.
spinnaker-kustomize-patches/operator folder, execute the
Update Armory Operator configuration
Update MinIO secret access key
You also need to update Armory Operator configuration to include the secret access key for MinIO.
spinnaker-kustomize-patches/operator/patch-config.yml and update the
AWS_SECRET_ACCESS_KEY value with the
minioAccessKey value you created in the Deploy MinIO to host the BOM section.
apiVersion: apps/v1 kind: Deployment metadata: name: spinnaker-operator spec: template: spec: containers: - name: spinnaker-operator env: - name: AWS_ACCESS_KEY_ID # you can choose to use a secret for these values value: minio - name: AWS_SECRET_ACCESS_KEY # you can choose to use a secret for these values value: changeme volumeMounts: - mountPath: /opt/spinnaker/config/halyard.yml name: operator-config subPath: halyard-local.yml - name: halyard env: - name: AWS_ACCESS_KEY_ID # you can choose to use a secret for these values value: minio - name: AWS_SECRET_ACCESS_KEY # you can choose to use a secret for these values value: changeme volumeMounts: - mountPath: /opt/spinnaker/config/halyard-local.yml name: operator-config subPath: halyard-local.yml volumes: - configMap: defaultMode: 420 name: operator-config name: operator-config
Update Halyard configuration
The Armory Operator uses its own Halyard installation to deploy and manage Armory Enterprise. You need to configure the new BOM location in
spinnaker-kustomize-patches/operator/halyard-local.yml. Update your
halyard-local.yml to match the content of the highlighted lines in the following example:
halyard: halconfig: directory: /home/spinnaker/.hal spinnaker: config: # This section is used in air-gapped environments to specify an alternate location for the Bill Of Materials (BOM). input: gcs: enabled: false # If the BOM is stored in a GCS bucket, switch this to true. bucket: halconfig # Name of the bucket where the BOM is located. #region: us-west-2 # Bucket region; region does not matter for MinIO. enablePathStyleAccess: true # If you are using a platform that does not support PathStyleAccess, such as MinIO, switch this to true (https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html#access-bucket-intro). endpoint: http://minio.spinnaker:9000 anonymousAccess: false
Deploy the Armory Operator
Now that you have updated
patch-config.yml, you can deploy the Armory Operator using its
kustomization.yml file. By default, the Armory Operator deploys to the
spinnaker-operator namespace. From the
spinnaker-kustomize-patches/operator directory, execute:
kubectl apply -k .
After the Armory Operator pod is running, verify that the S3-compatible MinIO bucket is properly configured and that the bucket contains the BOM.
The following example uses version 2.25.0 and a private Docker registry called
Run the following to access the Halyard container running in the
kubectl exec -ti deploy/spinnaker-operator -c halyard -- bash
Verify that the bucket is properly configured by running:
aws s3 ls --endpoint=http://minio.spinnaker:9000 s3://halconfig/
Inspect the contents of the BOM for the version you downloaded in Download the BOM:
hal version bom <version-number>
If you run the command passing in
2.25.0, output is similar to:
+ Get BOM for 2.25.0 Success version: 2.25.0 timestamp: '2021-03-25 09:28:32' services: echo: version: 2.25.2 commit: 3a098acc clouddriver: version: 2.25.3 commit: de3aa3f0 ... dependencies: redis: version: 2:2.8.4-2 artifactSources: dockerRegistry: my.jfrog.io/myteam/armory
Was this page helpful?
Thank you for letting us know!
Sorry to hear that. Please tell us how we can improve.
Last modified June 15, 2022: (5239aaae)