Spinnaker Installation, Deployments, Questions & Answers
A General Review of Spinnaker
The focus of the article will be on end-to-end testing – everything about Spinnaker.
INSTALLATION
Very briefly, on the topic of installing Spinnaker at AWS:
– Follow their documentation
– There are a series of steps regarding IAM Roles covered in https://www.spinnaker.io/setup/providers/aws/ . These are a bit tricky, and if you don’t implement them all of them exactly correctly, Spinnaker will break.
– The VPC should preferably be a “Default VPC”. Check this.
– The VPC should be given a Name tag that doesn’t include a . character (e.g. myvpc)
– The subnets should each be given a Name tag that follows vpcname.subnetpurpose.availability-zone – e.g. myvpc.internal.us-west-2a
– Add a number of AWS Regions, even including (yet again) the default region. Such as
hal config provider aws account edit my-aws-account –add-region us-west-2
hal config provider aws account edit my-aws-account –add-region us-west-1
hal deploy apply
– After launching both Spinnaker and Halyard, make sure there are up to 8 java processes running. They are:
rosco
orca
igor
gate
front50
clouddriver
echo
halyard
Here are ways to check:
ps -ef ps -ef | grep java | grep -v grep | wc
That makes 8 java processes. If fewer are running, observe the log files:
cd /var/log/spinnaker ls -alR
Find the log file of the process which has stopped, and observe error messages. Fix the problems and restart spinnaker. Make sure all the processes are running. Later on, it will also be helpful to find errors in these logs.
CONNECTING TO THE SPINNAKER SERVER
Spinnaker only listens on the localhost by default.
Earlier official documentation had discussed tunneling, however it may have been removed. Here are the steps. On your dev workstation,
~/.ssh/config :
Host spinnaker-start HostName 1.2.3.4 IdentityFile /path/to/my-aws-account-keypair.pem ControlMaster yes ControlPath ~/.ssh/spinnaker-tunnel.ctl RequestTTY no LocalForward 9000 127.0.0.1:9000 LocalForward 8084 127.0.0.1:8084 LocalForward 8087 127.0.0.1:8087 User ubuntu Host spinnaker-stop HostName 1.2.3.4 IdentityFile /path/to/my-aws-account-keypair.pem ControlPath ~/.ssh/spinnaker-tunnel.ctl RequestTTY no Host spinnaker HostName 1.2.3.4 IdentityFile /path/to/my-aws-account-keypair.pem User ubuntu
replace HostName and IdentityFile.
spinnaker-tunnel.sh :
#!/usr/bin/bash
socket=$HOME/.ssh/spinnaker-tunnel.ctl
if [ "$1" == "" ]; then
echo "Please run either ./spinnaker-tunnel.sh start or ./spinnaker-tunnel.sh stop"
fi
if [ "$1" == "start" ]; then
if [ ! \( -e ${socket} \) ]; then
echo "Starting tunnel to Spinnaker..."
ssh -f -N spinnaker-start && echo "Done."
else
echo "Tunnel to Spinnaker running."
fi
fi
if [ "$1" == "stop" ]; then
if [ \( -e ${socket} \) ]; then
echo "Stopping tunnel to Spinnaker..."
ssh -O "exit" spinnaker-stop && echo "Done."
else
echo "Tunnel to Spinnaker stopped."
fi
fi
chmod 755 spinnaker-tunnel.sh
ssh-start.sh :
#!/bin/bash eval `ssh-agent` ssh-add /path/to/my-aws-account-keypair.pem
chmod 755 ssh-start.sh
Then, in order to connect
. ssh-start.sh #only once
./spinnaker-tunnel.sh start #whenever the tunnel should be started
./spinnaker-tunnel.sh stop #whenever the tunnel should be stopped
In order to customize the configurations of the various services, there are many config files such as
/opt/spinnaker/config/spinnaker.yml /opt/rosco/config/rosco.yml
and many more. I have written another post dedicated only to understanding the config files: Spinnaker Configuration File Precedence
DEPLOYMENTS
Once you have Spinnaker open in a browser, the basic steps are:
– Create an app
– Create a pipeline
– Add “bake” and “deploy” stages for the pipeline
– Run this, and debug the problems
– Observe the clusters which are generated
– Try destroying and recreating clusters with Spinnaker
Rather than walk through all those steps in this article, which would perhaps be boring, another strategy is to say: just go ahead and try those steps, on your own. You may run into problems or questions. Let’s try to answer those now.
Questions and Answers:
Q: The bake stage has many options, and it’s generally confusing. How is that set up?
A: See Spinnaker, Ansible and Packer
Q: If you bake in multiple regions, how do you deploy in multiple regions? one stage? multiple stages?
A: For the “Deploy Configuration”, you can add multiple server groups. They each are assigned the correct AMI from the bake stage. This magically “just works”. It keeps track of the AMI’s.
Q: The restriction of only using Ubuntu Trusty instead of Ubuntu Xenial (as of 2017-10 and earlier), is that for Spinnaker server itself, or target images, or both?
A: Just the Spinnaker server itself. The target image may be Xenial.
Q: How can I add more images, such as new versions of Ubuntu, to the deployment pipeline?
A: Here is an example. Modify as necessary.
vi /home/ubuntu/.hal/default/profiles/rosco/rosco.yml
- baseImage:
id: xenial
shortDescription: v16.04
detailedDescription: Ubuntu Xenial Xerus v16.04
packageType: deb
virtualizationSettings:
- region: eu-west-2
virtualizationType: hvm
instanceType: t2.micro
sourceAmi: ami-996372fd
sshUserName: ubuntu
- region: ap-northeast-2
virtualizationType: hvm
instanceType: t2.micro
sourceAmi: ami-d28a53bc
sshUserName: ubuntu
- region: ap-south-1
virtualizationType: hvm
instanceType: t2.micro
sourceAmi: ami-099fe766
sshUserName: ubuntu
hal deploy apply hal config provider aws account edit my-aws-account --add-region eu-west-2 hal config provider aws account edit my-aws-account --add-region ap-northeast-2 hal config provider aws account edit my-aws-account --remove-region ap-northeast-1 hal config provider aws account edit my-aws-account --add-region ap-south-1 hal deploy apply
Q: Do you have an example app that I could deploy with spinnaker.
A: Yes, https://github.com/sdarwin/ansible-samplewebsite . That is covered in Spinnaker, Ansible and Packer
Q: Why aren’t spinnaker services starting on boot, or restarting on reboot?
A: For Ubuntu 14.04, create and run this script to solve the problem.
# upstarter.sh
#!/bin/bash
for i in clouddriver echo front50 gate igor orca rosco
do
echo "manual" > /etc/init/$i.override
done
chmod 755 upstarter.sh
./upstarter.sh
Q: I am having difficulties setting up email notifications.
A: Instead of Gmail, try Amazon SES, or another dedicated mail relay service. Or, a local installation of Exim or Postfix. The following steps will send mail via the local machine.
vi /home/ubuntu/.hal/default/profiles/echo-local.yml
management.health.mail.enabled: true mail: enabled: true from: [email protected] host: localhost fromAddress: [email protected] spring: mail: host: localhost
hal deploy apply
Install Postfix or Exim on the server itself. Check that is working. Send mail from the command-line:
mail [email protected]
Q: The Spinnaker auth configuration looks complicated.
A: Yes. Try SSH tunnels instead, explained above.
Q: What is the secret of spinnaker?
A: Here is an observation about how pipelines work:
One apparently implicit and important fact, is that a generated image in one stage will be sent to the next stage for deployment.
– Bake an image
– Deploy an image
You don’t have to reference it. The pipeline understands it should be the image from the last stage. This would then explain the mysterious “Find an Image”, or “Tag an Image”, which are optional stages. You may think something like “which one? which image?” It must be this implicit image, which is getting passed on from stage to stage in the pipeline.
Q: How do I upgrade Hal and Spinnaker?
A: Example:
sudo apt-get update && sudo apt-get upgrade spinnaker-halyard hal version list hal config version edit --version 1.4.2 hal deploy apply
Q: How do you add a scaling policy for an autoscaling group?
A: After deploying clusters in Spinnaker, add the scaling policy in AWS, rather than the Spinnaker interface.
Q: Must you recreate such scaling policies every time, in AWS?
A: Subsequent deployments of the same pipeline will use the previous scaling policy.
Q: If something changes in AWS, does Spinnaker pick it up? Is there a two-way street between AWS configs and Spinnaker configs?
A: Yes
Q: How do min, max, and desired levels work for autoscaling groups?
A: The “desired” level will fluctuate during autoscaling. If the instances are under load, this setting increases, up until the “max” level. If the instances are not under load, this setting will decrease, down to the “min” level. You should configure “min” and “max”, and let autoscaling do the rest.
Q: What does disabling a cluster accomplish?
A: This removes the instances from the Load Balancer, without deleting them.