Start by installing the AWS CLI tool on your machine, if necessary.
Simplifying AWS Credentials
You can make future AWS CLI commands easier by creating AWS profiles, which will add | configuration and credentials to the local ~/.aws/credentials file. In some cases you'll | be using two different accounts/profiles, so you can add the credentials for multiple | accounts to ~/.aws/credentials by using aws configure --profile commands.
BASH
aws configure --profile spinnaker-developer
AWS Access Key ID [None]:
AWS Secret Access Key [None]:
Default region name [None]: us-west-2
Default output format [None]: text
aws configure --profile primary
AWS Access Key ID [None]:
AWS Secret Access Key [None]:
Default region name [None]: us-west-2
Default output format [None]: text |`
In the future, simply add the <span class="code-class-custom">--profile <profile-name></span> flag to any AWS CLI command to | force AWS CLI to use that account.
How to Install Halyard
Halyard is the CLI tool that manages Spinnaker deployments and is typically the first step to any manual Spinnaker setup
Install Halyard with the InstallHalyard.sh script. If prompted, default options are usually just fine.
BASH
sudo bash InstallHalyard.sh
Source your recently-modified .bashrc file (or ~/.bash_profile).
BASH
. ~/.bashrc
Verify Halyard was installed by checking the version.
BASH
hal -v
# OUTPUT
1.9.1-20180830145737
That's the basics! Halyard is now ready to be configured and used for manual or quick start Spinnaker deployments.
How to Install Spinnaker
This section walks you through the most basic Spinnaker installation process, suitable for simple Spinnaker deployments.
Use the hal version list command to view the current Spinnaker version list.
BASH
hal version list
Configure Halyard to use the latest version of Spinnaker.
BASH
hal config version edit --version 1.9.2
(Optional) Enable Chaos Monkey in the Halyard config.
BASH
hal config features edit --chaos true
Tell Halyard what type of environment you're deploying Spinnaker to. Most production setups will want to use Kubernetes or another distributed solution, but the default deployment is a local installation. The hal config deploy edit --type flag can be used to change the environment.
BASH
hal config deploy edit --type localdebian
Halyard requires some form of persistent storage, so we'll use AWS S3 for simplicity. Modify the Halyard config and be sure to pass an AWS ACCESS_KEY_ID and SECRET_ACCESS_KEY with privileges to create and use S3 buckets.
BASH
hal config storage s3 edit --access-key-id <AWS_ACCESS_KEY_ID> --secret-access-key --region us-west-2
Configure Halyard to use the s3 storage type.
BASH
hal config storage edit --type s3
Now use sudo hal deploy apply to deploy Spinnaker to the local machine.
BASH
sudo hal deploy apply
After deployment finishes you should have a functioning Spinnaker installation! If you've installed on your local machine you can navigate to the Spinnaker Deck UI at localhost:9000 to see it in action. If Spinnaker was deployed on a remote machine use the hal deploy connect command to quickly establish SSH tunnels and connect.
Manually deploying Spinnaker with the help of Halyard is the best way to have the utmost control over your Spinnaker installation, and is ideal for advanced deployments to EC2 instances, EKS/Kubernetes clusters, and the like. Choose one of the three options depending on your needs.
In AWS navigate to CloudFormation and click Create Stack.
Download thismanaging.yaml file to your local machine.
Under Choose a template click the Choose File button under Upload a template to Amazon S3 and select the downloaded managing.yaml.
Click Next.
Input spinnaker-managing-infrastructure-setup into the Stack name field.
Select false in the UseAccessKeyForAuthentication dropdown.
Click Next.
On the Options screen leave defaults and click Next.
Check the I acknowledge that AWS CloudFormation might create IAM resources with custom names. checkbox and click Create to generate the stack.
If your AWS account already contains the BaseIAMRole AWS::IAM::ROLE you may have to delete it before this template will succeed.
Once the spinnaker-managing-infrastructure-setup stack has a CREATE_COMPLETEStatus, select the Outputs tab and copy all key/value pairs there somewhere convenient. They'll look something like the following.
Now we'll use AWS CLI to create the spinnaker-managing-infrastructure stack via CloudFormation. We want to use the primary or managing account for this, so we'll specify the --profile primary, which will grab the appropriate credentials, region, and so forth.
Error: Unresolved resource dependency for SpinnakerInstanceProfile
If you receive the above error while creating the spinnaker-managing-infrastructure stack you may need to edit the `managing.yaml` file and comment out the two SpinnakerInstanceProfileArn related lines under the Outputs block.
Once the spinnaker-managing-infrastructure stack has been created open the AWS console, navigate to the CloudFormation service, select the Outputs tab of the spinnaker-managing-infrastructure stack. We'll be using the ManagingAccountId and AuthArn values in the next step, which look something like the following.
Now enter the following command to create the companion spinnaker-managed-infrastructure stack in CloudFormation. Be sure to specify the profile value and paste the appropriate ManagingAccountId and AuthArn values from above.
Spinnaker uses accounts added via the Halyard hal config provider aws account API to handle all actions performed within the specified provider (such as AWS, in this case). For this example we'll just be using our primary managing account, but you can freely add more accounts as needed.
If you need to configure the stack to your own particular needs you can easily edit the template YAML as necessary. For example, in this guide we're only using a single managing account to handle Spinnaker/Kubernetes in AWS, but if you need to also include additional managed accounts you'll want to add their respective AWS ARN strings to the managing.yaml file around this line.
Now we'll use AWS CLI to issue a cloudformation deploy command to create a new spinnaker-managing-infrastructure-setup stack using the managing.yaml
template. From here on out this guide will use explicit names where applicable, but feel free to customize options as you see fit (such as the stack name, EksClusterName, and so forth).
The step above takes 10 - 15 minutes to complete, but once complete issue the following commands, which will use the AWS CLI to assign some environment variables values from the spinnaker-managing-infrastructure-setup stack we just created. We'll be using these values throughout the remainder of this guide.
You can easily output the value of an exported variable with echo $VARIABLE_NAME. However, remember that unless you export these values they only temporarily exist in the console in which you issued the commands. You may need to reissue the above commands later in the guide if you change terminal windows, so keep them handy.
Download this managed.yaml template. This template will create the spinnakerManaged AWS::IAM::Role that Spinnaker can use.
Execute this secondary CloudFormation deployment using the managed.yaml. Notice that this command (and many following commands) use some of the environmental variables we assigned previously, so the first stack deployment will need to be complete first.
If the second step of deploying spinnaker-managing-infrastructure-setup hasn't completed yet, feel free to skip this step for the time being and proceed with installing kubectl and AWS IAM Authenticator below. Just return to this step before moving past that point.
This guide will walk you through the entire process of setting up a Kubernetes cluster via AWS EKS, attaching some worker nodes (i.e. EC2 instances), deploying Spinnaker to manage the Kubernetes cluster, and then using Chaos Monkey and other Simian Army tools on it! If you're looking for a simpler Spinnaker installation, you might be interested in our Spinnaker AWS Quick Start guide.
We'll need to install the <span class="code-class-custom">kubectl</span> client and the <span class="code-class-custom">AWS IAM Authenticator for Kubernetes</span>, which will allow Amazon EKS to use IAM for authentication to our Kubernetes cluster.
Download the appropriate kubectl binary. Note: For the remainder of this guide we'll be using Linux examples, but most everything applies to other environments.
With everything setup you can now edit the <span class="code-class-custom">kubectl</span> configuration files, which will inform <span class="code-class-custom">kubectl</span> how to connect to your Kubernetes/EKS cluster.
Copy and paste the following into the ~/.kube/config file.
This guide assumes you're just configuring kubectl to handle a single Kubernetes cluster, but if you need to configure and handle multiple clusters the convention for doing so is to create a unique config file for each cluster. Simply name each config file ~/.kube/config-<cluster-name> , where <cluster-name> is replaced by the name of the Kubernetes cluster you already created.
Replace the <...> placeholder strings with the following EKS_CLUSTER_ environmental variable values from earlier:
<endpoint-url>: $EKS_CLUSTER_ENDPOINT
<base64-encoded-ca-cert>: $EKS_CLUSTER_CA_DATA
<cluster-name>: $EKS_CLUSTER_NAME
Using a Specific AWS::IAM::Role
If you need to have the AWS IAM Authenticator and kubectl use a specific Role then uncomment the - "-r" and - "<role-arn>" lines and paste the AWS::IAM::Role ARN in place of <role-arn>.
Your ~/.kube/config file should now look something like this:
Save your config file and export the KUBECONFIG variable to include the new config location.
BASH
export KUBECONFIG=$KUBECONFIG:~/.kube/config
Verify that kubectl is able to use your credentials to connect to your cluster with kubectl get svc:
BASH
kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 172.25.0.1 <none> 443/TCP 31m
Create AWS Accounts and Roles
We now need to apply service accounts and roles to the <span class="code-class-custom">kubectl spinnaker</span> namespace. In Kubernetes, a namespace is a virtual cluster, which can join other virtual clusters and all be housed within the same physical cluster.
Issue the following commands to create the spinnaker namespace, spinnaker-service-account service account, and spinnaker-admin binding. The apply command in kubectl applies a configuration based on the passed resource (YAML files, in this case).
The next step is to add Kubernetes as a provider to Halyard/Spinnaker. A provider is just an interface to one of many virtual resources Spinnaker will utilize. AWS, Azure, Docker, and many more are all considered providers, and are managed by Spinnaker via <span class="code-class-custom">accounts</span>.
Start by enabling the Kubernetes provider in Halyard.4
Enable the artifacts and chaos features of Halyard. Artifacts in Spinnaker merely reference any external resource, such as a remote file, a binary blob, and image, and so forth. The chaos feature allows us to utilize Chaos Monkey, the base form of which is built into Spinnaker by default.
BASH
hal config features edit --artifacts true
hal config features edit --chaos true
Add AWS Provider to Halyard
Now we need to also add AWS as another provider and account. Be sure to replace <span class="code-class-custom"<AWS_ACCOUNT_ID></span> with the primary/managing account ID of your AWS account.
BASH
hal config provider aws account add aws-primary --account-id --assume-role role/spinnakerManaged
hal config provider aws enable
Add ECS Provider to Halyard
The last provider to enable is ECS. We'll add the <span class="code-class-custom">ecs-primary</span> account to Halyard and associate it with the <span class="code-class-custom">aws-primary</span> AWS account added above:
BASH
hal config provider ecs account add ecs-primary --aws-account aws-primary
hal config provider ecs enable
Use Distributed Deployment
We also need to ensure Halyard deploys Spinnaker in a distributed fashion among our Kubernetes cluster. Without this step, the default configuration is to deploy Spinnaker onto the local machine.
BASH
hal config deploy edit --type distributed --account-name kubernetes-master
Error: kubectl`not installed, or can't be found by Halyard.
If you get such an error when issuing the distributed deployment command above, it likely means Halyard just needs to be restarted. Simply issue the hal shutdown command to stop the Halyard daemon, then retry the deployment edit command again, which will automatically restart Halyard before executing.
Use S3 for Persistent Storage
Let's tell Spinnaker to use AWS S3 for storing persistent data (in this case, creating a small S3 bucket). Issue the following command by replacing <span class="code-class-custom"> <AWS_ACCESS_KEY_ID> </span> with any AWS access key that has full S3 service privileges.
BASH
hal config storage s3 edit --access-key-id --secret-access-key --region us-west-2
hal config storage edit --type s3
Create Kubernetes Worker Nodes
Now we'll launch some AWS EC2 instances which will be our worker nodes for our Kubernetes cluster to manage.
The default template creates an auto-balancing collection of up to 3 worker nodes (instances). Additionally, the deployment command we'll be using below specifies t2.large instance types. As always, feel free to modify the amazon-eks-nodegroup.yaml or instance types to meet your needs.
Issue the following command to use the template and create your worker node collection.
To connect up our newly-launched worker instances with the Spinnaker cluster we previously deployed we need to create a new ~/.kube/aws-auth-cm.yaml file. Paste the following text into aws-auth-cm.yaml, replacing with the AUTH_ARN variable created previously (Remember, you can use echo $AUTH_ARN to print to console).
Apply this newly created role mapping by issuing the following command.
BASH
kubectl apply -f ~/.kube/aws-auth-cm.yaml
Check the status of your Kubernetes nodes with kubectl get nodes. The --watch flag can be added to perform constant updates. Once all nodes have a Ready STATUS you're all set to deploy Spinnaker.
BASH
kubectl get nodes
NAME STATUS ROLES AGE VERSION
ip-10-100-10-178.us-west-2.compute.internal Ready 2m v1.10.3
ip-10-100-10-210.us-west-2.compute.internal Ready 2m v1.10.3
ip-10-100-11-239.us-west-2.compute.internal Ready 2m v1.10.3
Deploy Spinnaker
Start by listing the current Spinnaker versions with hal version list:
BASH
hal version list
Specify the version you wish to install with the --version flag below. We'll be using the latest at the time of writing, 1.9.2.
BASH
hal config version edit --version 1.9.2
Now use hal deploy apply to deploy Spinnaker using all the configuration settings we've previously applied. This will go about distributing Spinnaker in your EKS/Kubernetes cluster.
BASH
hal deploy apply
Handling hal deploy apply Errors
In some cases you may experience a deployment error, particularly when trying your first Spinnaker deployment. Often the console output is quite vague, so the best course of action is to check your Spinnaker/Halyard log files. Typically these are located in /var/log/spinnaker and /var/log/spinnaker/halyard. Since Halyard runs on Java, logs let you see the entire Java error stack trace, rather than the basic (and often useless) error name.
Profile-related IndexOutOfBoundsException
With recent Halyard/Spinnaker versions there's a known bug that you may experience in which an IndexOutOfBoundsException occurs during deployment when using the AWS provider. The cause usually seems to be that Halyard is assuming an explicit region value in the YAML configuration file for the AWS account being used. Even though the aws block in the config has a defaultRegions key, that seems to be ignored, which can cause this error. The current solution is to manually edit the primary AWS account and explicitly set the region value, which should solve the issue and allow you to run a Spinnaker deployment.
BASH
hal config provider aws account edit aws-primary --add-region us-west-2
That's it, you should now have a Spinnaker deployment up and running on a Kubernetes cluster, using EKS and EC2 worker node instances! Issue the <span class="code-class-custom">hal deploy connect</span> command to provide port forwarding on your local machine to the Kubernetes cluster running Spinnaker, then open http://localhost:9000 to make sure everything is up and running.
Select the <span class="code-class-custom">spinnaker</span> app and you should see your <span class="code-class-custom">aws-primary</span> account with a <span class="code-class-custom">spinnaker-eks-nodes-NodeGroup</span> containing your three EC2 worker node instances.
