Spinnaker, Ansible and Packer

The main focus of the article will be specifically the Spinnaker Bake cycle. This is the intersection of Spinnaker, Ansible and Packer.

For a bit of advice about installation, which is actually a prerequisite, refer to Spinnaker Installation, Deployments, Questions & Answers

Packer and Ansible

On the Spinnaker server, the packer templates are located in /opt/rosco/config/packer/ . However, there is a slightly roundabout way of customizing these templates. Place your new templates in ~/.hal/default/profiles/rosco/packer/ instead. Then run “hal deploy apply” to deploy them over to /opt/rosco/config/packer/ .
As far as customizing rosco.yml, a similar process applies. Create and edit ~/.hal/default/profiles/rosco.yml , and run “hal deploy apply”
Note: the official documentation says to place customized packer configs in ~/.hal/$DEPLOYMENT/profiles/rosco/packer/ . The “~” is from

/opt/spinnaker/config/halyard-user

In this case, ubuntu. Therefore, “~” means /home/ubuntu. Therefore, the path is actually /home/ubuntu/.hal/default/profiles/rosco/packer/

The Packer Template

/opt/rosco/config/packer/aws-ebs.json is a good starting point. Let’s create our own customized packer template based on this, called website.json

cp /opt/rosco/config/packer/aws-ebs.json /home/ubuntu/.hal/default/profiles/rosco/packer/website.json
chown ubuntu:ubuntu /home/ubuntu/.hal/default/profiles/rosco/packer/website.json

and now edit /home/ubuntu/.hal/default/profiles/rosco/packer/website.json

Notice the provisioners section near the end

  "provisioners": [{
    "type": "shell",
    "script": "{{user `configDir`}}/install_packages.sh",
    "environment_vars": [
      "repository={{user `repository`}}",
      "package_type={{user `package_type`}}",
      "packages={{user `packages`}}",
      "upgrade={{user `upgrade`}}"
    ],
    "pause_before": "30s"
  }]

Let’s add a section for ansible, so it now becomes:

  "provisioners": [{
    "type": "shell",
    "script": "{{user `configDir`}}/install_packages.sh",
    "environment_vars": [
      "repository={{user `repository`}}",
      "package_type={{user `package_type`}}",
      "packages={{user `packages`}}",
      "upgrade={{user `upgrade`}}"
    ],
    "pause_before": "30s"
  },
{
    "type": "shell",
    "inline": [
      "sudo apt-get update",
      "sudo apt-get install -y software-properties-common",
      "sudo apt-get install -y git",
      "sudo apt-add-repository -y ppa:ansible/ansible",
      "sudo apt-get update",
      "sudo apt-get install -y ansible",
      "sudo apt-get install -y python-pip",
      "sudo pip install pyopenssl ndg-httpsclient pyasn1"
    ]
  },
{
  "type": "ansible-local",
  "inventory_groups": "web",
  "playbook_file": "/etc/ansible/website.yml",
  "playbook_dir": "/etc/ansible/"
}
]

Remove aws_enhanced_networking from the .json file. Not supported any more.

After any changes, run:

hal deploy apply

The first part of that (“type”: “shell”) installs Ansible itself. Add any additional steps you require. This section is relatively self-explanatory, with the main code really being just “apt-get install ansible”.

The second part of that (“type”: “ansible-local”) runs Ansible in order to provision the target machine, and requires a bit more explanation. But first, let’s continue the setup.

On the spinnaker host:

mkdir -p /etc/ansible/roles
cd /etc/ansible/roles
git clone https://github.com/sdarwin/ansible-samplewebsite sdarwin.samplewebsite
cp /etc/ansible/roles/sdarwin.samplewebsite/default.yml /etc/ansible/website.yml
ansible-galaxy install -r /etc/ansible/roles/sdarwin.samplewebsite/requirements.yml

In other words, /etc/ansible on the Spinnaker machine should represent a standard Ansible directory structure. /etc/ansible/roles should have any roles you will be installing. Place top-level playbooks in /etc/ansible itself.

Returning to the packer template, let’s review the configuration again.

“playbook_dir”: “/etc/ansible/”
This magical directive will copy “/etc/ansible/” from the local Spinnaker host over to the target host.

“playbook_file”: “/etc/ansible/website.yml”
And this one will run the playbook website.yml, which is now on the target host.

You may take a look at the example Ansible role at https://github.com/sdarwin/ansible-samplewebsite , which is an absolutely minimal “Hello World” website.

Why the ansible-local provisioner instead of the ansible provisioner? At the time of this writing, there had appeared to be a bug regarding “become” (i.e. sudo), although it may now have been corrected. Additionally, ansible-local was written two years earlier, which translates into better long-term support and bug fixes for the module.

Next,

Create a pipeline in Spinnaker.
Add a Bake Stage.

Package: htop #which is meaningless, since the provisioning will be done by ansible rather than a package.
Base OS: trusty
(advanced)
Template File name: website.json #had been placed in /home/ubuntu/.hal/default/profiles/rosco/packer

Just choose any package to install, because that field is not relevant for ansible.

Detour: Mentioning current open bugs encountered regarding Packer will make this article less timeless, since soon they will be fixed. Nevertheless here are some:
– Try upgrading packer. Unknown configuration key: “enhanced_networking”. Must modify all templates and remove “enhanced_networking”.
– Errors appearing in log files about “iam:ListServerCertificates”
Add this in AWS IAM to the Role being applied

{
“Effect”: “Allow”,
“Action”: “iam:ListServerCertificates”,
“Resource”: “*”
}
– If errors appear in the log files about (Service: AmazonEC2; Status Code: 400; Error Code: UnsupportedOperation;)
Review the IAM steps from Spinnaker Installation. They all must be followed completely.

Test and re-test a Red/Black deployment strategy by updating /etc/ansible/roles/sdarwin.samplewebsite/templates/index.html.j2 so that “Hello World” changes to “Hello World foobar etc etc”. Re-deploy. Observe the updates.

Since this article is focused on understanding the Bake phase of Spinnaker, let’s carefully review each of the options on the Bake Configuration Page. First, just a quick listing of all the options that appear, and then again in more detail.

Bake Configuration (basic list of options)
– Regions
– Package
– Base OS
– VM Type
– Rebake
– Template File Name
– Extended Attributes
– Var File Name
– Base Name
– Base AMI
– AMI Name
– AMI Suffix

Bake Configuration (in more detail)

– Regions

Of course, the regions where the bake step will run. To add more regions on this list, for example ap-northeast-1

hal config provider aws account edit my-aws-account --add-region ap-northeast-1
hal deploy apply

Out of curiosity, what files are changed. The first step “hal config …” adds ap-northeast-1 into regions in this file:
/home/ubuntu/.hal/config

The second step “hal deploy apply” adds ap-northeast-1 into regions these files:
/opt/spinnaker/config/rosco.yml
/opt/spinnaker/config/clouddriver.yml

– Package “The name of the package you want installed (without any version identifiers).”

The package which will be installed. However, if the instance is provisioned in other ways, such as Ansible, the package is sort of meaningless. Enter a package, in any case. How about “htop”.

– Base OS

Choose from a drop-down list of OS choices. The list originates from /opt/rosco/config/rosco.yml and can be customized by editing /home/ubuntu/.hal/default/profiles/rosco/rosco.yml and then “hal deploy apply”

– VM Type “HVM PV”

Amazon supports two types of VM’s. Whichever is chosen must match the AMI type in the previous step Base OS.

– Rebake “Rebake image without regard to the status of any existing bake”

Since the “package” isn’t changing, but Ansible is changing, it may be good to always Rebake. Therefore choose this option. It can also be manually selected when running the pipeline.

– Template File Name “(Optional) The explicit packer template to use, instead of resolving one from rosco’s configuration.”

/opt/rosco/config/rosco.yml contains some defaults for the template file. Specifically templateFile: aws-ebs.json
/opt/rosco/config/rosco.yml can be customized, see the “Base OS” step. Rather than customize rosco.yml though, just set the Template File Name here.
The templates are located in /opt/rosco/config/packer/
The procedure though is to create them in /home/ubuntu/.hal/default/staging/rosco/packer/ and run “hal deploy apply” to provision them to /opt/rosco/config/packer/
Since aws-ebs.json is the default, copy that as a starting point, if you will customize templates.

– Extended Attributes “(Optional) Any additional attributes that you want to pass on to rosco, which will be injected into your packer runtime variables.”

Let’s say we create a key var123 and a value value123. When packer is run, the following additional command line option appears: -var var123=value123 . More details https://www.packer.io/docs/templates/user-variables.html

– Var File Name “(Optional) The name of a json file containing key/value pairs to add to the packer command.”

It’s looking for a file in this basic location /opt/rosco/config/packer/_example_var_file_.json (replace _example_var_file_)
The packer command is run with the option -var-file=/opt/rosco/config/packer/_example_var_file_.json
The contents of _example_var_file_.json should be such as
{
“foo”: “bar”,
“abc”: “xyz”
}
The topic of how packer handles this is covered in https://www.packer.io/docs/templates/user-variables.html

– Base Name

uncertain. not documented.

– Base AMI “(Optional) ami-????????”

This overrides the AMI value, which was set for Base OS. A problem would be deploying in more than one region simultaneously. You’d need to somehow set more than one AMI. For that reason, it would be better to configure the Base OS instead of setting this Base AMI. See above.

When packer ran it specified the command-line option -var aws_source_ami=ami-66051402

– AMI Name “(Optional) Default = $package-$arch-$ami_suffix-$store_type”

As an example, when the package was htop, the generated AMI was called htop-all-20171005134156-xenial, When packer ran it specified the command-line option -var aws_target_ami=htop-all-20171005134156-xenial
In order to customize the package name, set the value to _example_-${arch}-${ami_suffix}-${store_type} , replacing _example_.

– AMI Suffix “(Optional) String of date in format YYYYMMDDHHmm, default is calculated from timestamp,”

As seen in the previous step, the date is part of the AMI name. Change the date format if you’d like, although there is little reason to do so.