Deploying to AWS from Spinnaker (using IAM instance roles)

Once you have (OSS or Armory) Spinnaker up and running in Kubernetes, you’ll want to start adding deployment targets. (This document assumes Spinnaker was installed with halyard, that you have access to your current halconfig (and a way to operate hal against it), and that you have a way to create AWS permissions, users, and roles

Overview

This document will guide you through the following:

  • Understanding AWS deployment from Spinnaker

  • Configuring Spinnaker to access AWS using an IAM User (with an Access Key and Secret Access Key)

    • Creating a Managed Account IAM Role in each your target AWS Accounts
    • Creating the default BaseIamRole for use when deploying EC2 instances
    • Creating a Managing Account IAM Policy in your primary AWS Account
    • Creating a Managing Account IAM User with access to the Managing Account Policy
    • Configuring the Managed Accounts to trust the Managing Account IAM User
    • Adding the Managing Account User and Managed Accounts to Spinnaker via Halyard
    • Adding/Enabling the AWS Cloudprovider to Spinnaker

Background: Understanding AWS Deployment from Spinnaker

Even though Spinnaker is installed in Kubernetes, it can be used to deploy to other cloud environments, such as AWS. Rather than granting Spinnaker direct access to each of the target AWS accounts, Spinnaker will assume a role in each of the target accounts.

Deploying

Spinnaker is able to deploy EC2 instances (via ASGs).

  • Spinnaker’s Clouddriver Pod should be able to assume a Managed Account Role in each deployment target AWS account, and use that role to perform any AWS actions. This may include one or more of the following:
    • Create AWS Launch Configurations and Auto Scaling Groups to deploy AWS EC2 instances
    • Run ECS Containers
    • Run AWS Lambda Actions (alpha/beta as of the time of this document)
    • Create AWS CloudFormation Stacks (alpha/beta as of the time of this document)
  • Clouddriver is configured with direct access to a “Managing Account” Policy (it may be helpful to think of this as the Master or Source Policy), which is accomplished on one of two ways:
    • If Spinnaker is running in AWS (either in AWS EKS, or with Kubernetes nodes running in AWS EC2), the Managing Account Policy can be made available to Spinnaker by adding it to the AWS nodes (EC2 instances) where the Spinnaker Clouddriver pod(s) are running.
      • (You can also use Kube2IAM or similar capabilities, but this is not covered in this document)
    • An IAM User with access to the Managing Account Policy can be passed directly to Spinnaker via an Access Key and Secret Access Key, configured via Halyard
  • For each AWS account that you want Spinnaker to be able to deploy to, Spinnaker needs a “Managed Account” Role in that AWS account, with permissions to do the things you want Spinnaker to be able to do (it may be helpful to think of this as a Target Role)
  • The Managing Account Role (Source/Master Role) should be able to assume each of the Managed Account Roles (Target Roles). This requires two things:
    • The Managing Account Role needs a permission string for each Managed Account it needs to be able to assume. It may be helpful to think of this as an outbound permission.
    • Each Managed Account needs to have a trust relationship with the Managing Account User or Role to allow the Managing Account User or Role to assume it. It may be helpful to think of this as an inbound permission.

In addition, if you are deploying EC2 instances with AWS, you will need to provide an IAM role for each instance. If you do not specify a role, Spinnaker will attempt to use a role called BaseIAMRole. So you should create a BaseIAMRole (potentially with no permissions).

Example

Here’s an example situation:

  • We would like Spinnaker to deploy to three AWS accounts, with account IDs 111111111111, 222222222222, and 333333333333. Each of these is a Managed Account
  • Choose one account (111111111111), that Spinnaker will log into directly. This is the Managing Account
  • We will end up with four IAM entities:
    • A Managing Account Policy in account 111111111111 (arn:aws:iam::111111111111:user/managingAccount)
    • A Managed Account Role in account 111111111111 (arn:aws:iam::111111111111:role/spinnakerManaged)
    • A Managed Account Role in account 222222222222 (arn:aws:iam::222222222222:role/spinnakerManaged)
    • A Managed Account Role in account 333333333333 (arn:aws:iam::333333333333:role/spinnakerManaged)
  • The Managing Account Policy needs these:
    • The sts:AssumeRole permission for each of the Managed Account Roles
    • The ec2:DescribeAvailabilityZones permission
    • The ec2:DescribeRegions permission
    • It should be attached to the IAM Instance Role where Spinnaker is running
  • Each Managed Account Role needs these:
    • PowerUserAccess
    • The iam:PassRole permission for roles that will be assigned to EC2 instances that are being deployed
    • A trust relationship with the IAM Instance Role attached to the EC2 instances where Spinnaker is running (to allow Spinnaker to assume the Managed Account Role)

Baking

Spinnaker is able to use Packer to bake AMIs in AWS. If you’re using IAM Instance Roles, you need to provide credentials to Spinnaker to use to Bake by adding additional policies to the EC2 instances where Spinnaker is running. The AWS account that you’re baking in must also be configured as a Managing Account, and that Managing Account (Role) must be configured as the primary AWS account within Spinnaker.

These policies must have all permissions necessary to bake (for example, PowerUserAccess and associated PassRoles)

Spinnaker will always bake with the EC2 instance role (unless you specify explicit baking creds). If you need to deploy to other accounts, update your Packer template to support sharing the baked image with other accounts. For example, add this to your builder configuration in your packer template (and add the custom packer template following the instructions in the Spinnaker Packer documentation):

    "ami_users": ["222222222222","333333333333"]

Configuration

Here’s a sample halconfig aws YAML block that supports the above:

    aws:
      enabled: true
      accounts:
      - name: aws-1
        requiredGroupMembership: []
        providerVersion: V1
        permissions: {}
        accountId: '111111111111'
        regions:
        - name: us-east-1
        - name: us-west-2
        assumeRole: role/spinnakerManaged
      - name: aws-2
        requiredGroupMembership: []
        providerVersion: V1
        permissions: {}
        accountId: '222222222222'
        regions:
        - name: us-east-1
        - name: us-west-2
        assumeRole: role/spinnakerManaged
      - name: aws-3
        requiredGroupMembership: []
        providerVersion: V1
        permissions: {}
        accountId: '333333333333'
        regions:
        - name: us-east-1
        - name: us-west-2
        assumeRole: role/spinnakerManaged
      # Because we're baking in 111111111111, this must match the accountName that is associated with 111111111111
      primaryAccount: aws-1
      bakeryDefaults:
        templateFile: aws-ebs-shared.json
        baseImages: []
        awsAssociatePublicIpAddress: true
        defaultVirtualizationType: hvm
      defaultKeyPairTemplate: '-keypair'
      defaultRegions:
      - name: us-west-2
      defaults:
        iamRole: BaseIAMRole

Prerequisites

This document assumes the following:

  • Your Spinnaker is up and running
  • Your Spinnaker was installed and configured via Halyard
  • You have access to Halyard
  • You have permissions to create IAM roles using IAM policies and permissions, in all relevant AWS accounts
    • You should also be able to set up cross-account trust relationships between IAM roles.
  • If you want to add the IAM Role to Spinnaker via an Access Key/Secret Access Key, you have permissions to create an IAM User
  • If you want to add the IAM Role to Spinnaker via IAM instance profiles/policies, you have permissions to modify the IAM instance

All configuration with AWS in this document will be handled via the browser-based AWS Console. All configurations could alternately be configured via the aws CLI, but this is not currently covered in this document.

Also - we will be granting AWS Power User Access to each of the Managed Account Roles. You could optionally grant fewer permisisons, but those more limited permissions are not covered in this document.

Configuring Spinnaker to use AWS IAM Instance Roles

If you are running Spinnaker on AWS (either via AWS EKS or installed directly on EC2 instances), you can use AWS IAM roles to allow Clouddriver to interact with the various AWS APIs across multiple AWS Accounts.

Instance Role Part 1: Creating a Managed Account IAM Role in each your target AWS Accounts

In each account that you want Spinnaker to deploy to, you should create an IAM role for Spinnaker to assume.

For each account you want to deploy to, perform the following:

  1. Log into the browser-based AWS Console
  2. Navigate to the IAM page (click on “Services” at the top, then on “IAM” under “Security, Identity, & Compliance”)
  3. Click on “Roles” on the left hand side
  4. Click on “Create role”
  5. For now, for the “Choose the service that will use this role”, select “EC2”. We will change this later, because we want to specify an explicit consumer of this role later on.
  6. Click on “Next: Permissions”
  7. Search for “PowerUserAccess” in the search filter, and select the Policy called “PowerUserAcces”
  8. Click “Next: Tags”
  9. Optionally, add tags that will identify this role.
  10. Click “Next: Review”
  11. Enter a Role Name. For example, “DevSpinnakerManagedRole”. Optionally, add a description, such as “Allows Spinnaker Dev Cluster to perform actions in this account.”
  12. Click “Create Role”
  13. In the list of Roles, click on your new Role (you may have to scroll down or filter for it).
  14. Click on “Add inline policy” (on the right).
  15. Click on the “JSON” tab, and paste in this:

    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Action": "iam:PassRole",
                "Resource": [
                    "*"
                ],
                "Effect": "Allow"
            }
        ]
    }
    
  16. Click “Review Policy”
  17. Call it “PassRole”, and click “Create Policy”
  18. Copy the Role ARN and save it. It should look something like this: arn:aws:iam::123456789012:role/DevSpinnakerManagedRole. This will be used in the section “Instance Role Part 3”, and in the Halyard section, “Instance Role Part 6”

You will end up with a Role ARN for each Managed / Target account. The Role names do not have to be the same (although it is a bit cleaner if they are). For example, you may end up with roles that look like this:

  • arn:aws:iam::123456789012:role/DevSpinnakerManagedRole
  • arn:aws:iam::123456789013:role/DevSpinnakerManagedRole
  • arn:aws:iam::123456789014:role/DevSpinnakerManaged

Instance Role Part 2: Creating the BaseIAMRole for EC2 instances

When deploying EC2 instances, Spinnaker currently requires that you attach a role for each instance (even if you don’t want to grant the instance any special permissions. If you do not specify an instance role, Spinnaker will default to a role called BaseIAMRole, and it will throw an error if this does not exist. Therefore, you should at a minimum create an empty role called BaseIAMRole.

  1. Log into the browser-based AWS Console
  2. Navigate to the IAM page (click on “Services” at the top, then on “IAM” under “Security, Identity, & Compliance”)
  3. Click on “Roles” on the left side
  4. Click “Create role”
  5. Select “EC2”, and click “Next: Permissions”
  6. Click “Next: Tags”
  7. Optionally, add tags if required by your organization. Then, click “Next: Review”.
  8. Specify the Role Name as “BaseIAMRole”

Instance Role Part 3: Creating a Managing Account IAM Policy in your primary AWS Account

In the account that Spinnaker lives in (i.e., the AWS account that owns the EKS cluster where Spinnaker is installed), create an IAM Policy with permissions to assume all of your Managed Roles.

  1. Log into the AWS account where Spinnaker lives, into the browser-based AWS Console
  2. Navigate to the IAM page (click on “Services” at the top, then on “IAM” under “Security, Identity, & Compliance”)
  3. Click on “Policies” on the left hand side
  4. Click on “Create Policy”
  5. Click on the “JSON” tab, and paste in this:

    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Action": [
                    "ec2:DescribeAvailabilityZones",
                    "ec2:DescribeRegions"
                ],
                "Resource": [
                    "*"
                ]
            },
            {
                "Action": "sts:AssumeRole",
                "Resource": [
                    "arn:aws:iam::123456789012:role/DevSpinnakerManagedRole",
                    "arn:aws:iam::123456789013:role/spinnakerManaged",
                    "arn:aws:iam::123456789014:role/DevSpinnakerManaged"
                ],
                "Effect": "Allow"
            }
        ]
    }
    
  6. Update the sts:AssumeRole block with the list of Managed Roles you created in Instance Role Part 1.
  7. Click on “Review Policy”
  8. Create a name for your policy, such as “DevSpinnakerManagingPolicy”. Optionally, add a descriptive description. Copy the name of the policy. This will be used in the next section, “Instance Role Part 4”
  9. On the list policies, click your newly-created Policy.

(This policy could also be attached inline directly to the IAM Instance Role, rather than creating a standalone policy)

Instance Role Part 4: Adding the Managing Account IAM Policy to the existing IAM Instance Role on the AWS nodes

  1. Log into the AWS account where Spinnaker lives, into the browser-based AWS Console
  2. Navigate to the EC2 page (click on “Services” at the top, then on “EC2” under “Compute”)
  3. Click on “Running Instances”
  4. Find one of the nodes which is part of your EKS or other Kubernetes cluster, and select it.
  5. In the Instance details section of the screen (in the lower half), find the “IAM Role” and click on it to go to the Role page.
  6. Click on “Attach Policies”
  7. Search for the Policy that you created, and select it.
  8. Click “Attach Policy”
  9. Back on the screen for the Role, copy the node role ARN. It should look something like this: arn:aws:iam::123456789010:role/node-role. This will be used in the next section, “Instance Role Part 5”

Instance Role Part 5: Configuring the Managed Accounts IAM Roles to trust the IAM Instance Role from the AWS nodes

Now that we know what role will be assuming each of the Managed Roles, we must configure the Managed Roles (Target Roles) to trust and allow the Managing (Assuming) Role to assume them. This is called a “Trust Relationship” and is configured each of the Managed Roles (Target Roles).

For each account you want to deploy to, perform the following:

  1. Log into the browser-based AWS Console
  2. Navigate to the IAM page (click on “Services” at the top, then on “IAM” under “Security, Identity, & Compliance”)
  3. Click on “Roles” on the left hand side
  4. Find the Managed Role that you created earlier in this account, and click on the Role Name to edit the role.
  5. Click on the “Trust relationships” tab.
  6. Click on “Edit trust relationship”
  7. Replace the Policy Document with this (Update the ARN with the node role ARN from “Instance Role Part 4”)

    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Principal": {
            "AWS": [
              "arn:aws:iam::123456789010:role/node-role"
            ]
          },
          "Action": "sts:AssumeRole"
        }
      ]
    }
    
  8. Click “Update Trust Policy”, in the bottom right.

Instance Role Part 6: Adding the Managed Accounts to Spinnaker via Halyard

The Clouddriver pod(s) should be now able to assume each of the Managed Roles (Target Roles) in each of your Deployment Target accounts. We need to configure Spinnaker to be aware of the accounts and roles its allowed to consume. This is done via Halyard.

For each of the Managed (Target) accounts you want to deploy to, perform the following from your Halyard instance:

  1. Run this command, updating fields as follows:
    • AWS_ACCOUNT_NAME should be a unique name which is used in the Spinnaker UI and API to identify the deployment target. For example, aws-dev-1 or aws-dev-2
    • ACCOUNT_ID should be the account ID for the Managed Role (Target Role) you are assuming. For example, if the role ARN is arn:aws:iam::123456789012:role/ DevSpinnakerManagedRole, then ACCOUNT_ID would be 123456789012
    • ROLE_NAME should be the full role name within the account, including the type of object (role). For example, if the role ARN is arn:aws:iam::123456789012:role/ DevSpinnakerManagedRole, then ROLE_NAME would be role/DevSpinnakerManagedRole
    # Enter the account name you want Spinnaker to use to identify the deployment target,  the account ID, and the role name.
    export AWS_ACCOUNT_NAME=aws-dev-1
    export ACCOUNT_ID=123456789012
    export ROLE_NAME=role/DevSpinnakerManagedRole
     
    hal config provider aws account add ${AWS_ACCOUNT_NAME} \
        --account-id ${ACCOUNT_ID} \
        --assume-role ${ROLE_NAME}
    
  2. Optionally, edit the account with additional options such as those indicated in the halyard documentation. For example, to set the regions that you can deploy to:

    export AWS_ACCOUNT_NAME=aws-dev-1
    hal config provider aws account edit ${AWS_ACCOUNT_NAME} \
        --regions us-east-1,us-west-2
    

Instance Role Part 7: Adding/Enabling the AWS CloudProvider configuration to Spinnaker

Once you’ve added all of the Managed (Target) accounts, run these commands to set up and enable the AWS cloudprovider setting as whole (this can be run multiple times with no ill effects):

  1. Enable the AWS Provider

    hal config provider aws enable
    
  2. Apply all Spinnaker changes:

    # Apply changes
    hal deploy apply