Home Salt-Formulas Development Documentation

Testing Salt Formulas

Each formula contains Makefile with at least test target. Under tests directory are located resources for test execution.

Test target executes “smoke test” implemented by tests/run_tests.sh capable to fetch dependencies in python virtual environment by executing salt-call state.show_sls with provided tests/pillar data.

The purpose of the smoke test is to find syntax, typo issues and verify example pillar data against the formula.

Initial content of tests folder contains test pillars and a run_tests.sh as generated by cookiecutter.

tests
├── pillar
│   └── client_single.sls
│   └── server_single.sls
└── run_tests.sh

Create or update pillars in tests/pillar/\*.sls with test data.

Generate Test Structures in Formula

There is and salt-formulas cookiecutter template. to generate initial repository structure for new formula.

For existing formulas there is an convenient script capable to generate initial structures from available content. For more details follow the README in the above linked repository. To simply generate test structures according specification stated in this document simply run kitchen-init.sh.

tl;dr:

curl -skL "https://raw.githubusercontent.com/salt-formulas/salt-formulas/master/cookiecutter/salt-formula/kitchen-init.sh" | bash -s --

Formula Testing with Test Kitchen

Test Kitchen with forked kitchen-salt provisioner plugin may be used for local development as well as CI scenario.

Test Kitchen is a test harness tool to execute your configured code on one or more platforms in isolation. There is a .kitchen.yml in main directory that defines platforms to be tested and suites to execute on them.

Kitchen CI can spin instances locally or remote, based on used driver. For example .kitchen.yml may define a docker or vagrant driver.

For more, explore it’s rich ecosystem of supported drivers/provisioners/verifiers/…

Using Test Kitchen

A listing of scenarios to be executed:

$ kitchen list

Instance                    Driver   Provisioner  Verifier  Transport  Last Action

client-single-ubuntu-1404   Docker   SaltSolo     Inspec    Ssh        <Not Created>
client-single-ubuntu-1604   Docker   SaltSolo     Inspec    Ssh        <Not Created>
client-single-centos-71     Docker   SaltSolo     Inspec    Ssh        <Not Created>

The Busser Verifier is used to setup and run tests implementated in <repo>/test/integration. It installs the particular driver to tested instance (Serverspec, InSpec, Shell, Bats, …) prior the verification is executed.

Example workflow:

# list instances and status
kitchen list

# manually execute integration tests
kitchen [test || [create|converge|verify|exec|login|destroy|...]] [instance] -t tests/integration

# use with provided Makefile (ie: within CI pipeline)
make kitchen

How it Works

Kitchen spin an instances in Docker, Vagrant, OpenStack environment, etc. based on configured driver. Instance is configured as salt minion, where the configuration is defined by .kitchen.yml and tests/pillar/*.sls

Override your specific needs with .kitchen.<backend|local>.yml that you may load as: KITCHEN_LOCAL_YAML=.kitchen.<driver>.yml kitchen <action> <suite>.

Example: KITCHEN_LOCAL_YAML=.kitchen.local kitchen verify server-ubuntu-1404 -t tests/integration.

Test Kitchen then allows you execute several action to perform your testing under configured conditions:

  1. create, provision an test instance (VM, container)
  2. converge, run a provisioner (shell script, kitchen-salt)
  3. verify, run a verification (inspec, other may be added)
  4. destroy

Verifying Deployment

There is couple of verifier plugins that are shipped with Test Kitchen. They allow to run simple bash scripts and checking it’s exit codes to run specific purpose based frameworks.

The Busser Verifier goes with test-kitchen by default. It is used to setup and run tests implemented in <repo>/test/integration. It guess and installs the particular driver to tested instance. By default InSpec is expected.

You may avoid to install busser framework if you configure specific verifier in .kitchen.yml:

verifier:
        name: inspec

For default Inspec Verifier implement your scripts directly in <repo>/test/integration/<suite> directory with _spec.rb suffix.

If you would to write another verification scripts than InSpec store them in <repo>/tests/integration/<suite>/<verifier>. Busser <https://github.com/test-kitchen/busser> is a test setup and execution framework under test kitchen.

Implement integration tests under <repo>/tests/integration/<suite>/<verifier> directory with _spec.<verifier suffix> filename suffix.

InSpec

InSpec is native validation framework for Test Kitchen and as such don’t require usage of <verifier> folder. Thus the tests may by stored directly under <repo>/tests/integration/<suite>

Additional resources.

Example verification scripts under tests/integration folder of the formula:

tests
├── integration
│   ├── default
│   │   └── default_testcase_spec.rb  # Written in InSpec
│   ├── backupmx
│   │   └── serverspec                # <Verifier framework>
│   │       └── backupmx_spec.rb      # Written in ServerSpec
│   ├── helpers
│   │   └── serverspec
│   │       └── spec_helper.rb
│   ├── relay
│   │   └── serverspec
│   │       └── relay_spec.rb
│   └── server
│       └── serverspec
│           ├── aliases_spec.rb
│           └── server_spec.rb
├── pillar
│   ├── backupmx.sls
│   ├── relay.sls
│   └── server.sls
└── run_tests.sh

Requirements

Use latest stable kitchen-salt <https://github.com/saltstack/kitchen-salt> and kitchen-test.

TL;DR

First you have to install ruby package manager gem.

Install required gems:

# Ruby side:
gem install <gem name from the list below>

# Isolated w/Bundler
gem install bundler

cat > Gemfile <-EOF
              source 'https://rubygems.org'

              gem 'rake'
              gem 'test-kitchen'
              gem 'kitchen-docker'
              gem 'kitchen-inspec'
              gem 'inspec'
              gem 'kitchen-salt', :git => 'https://github.com/salt-formulas/kitchen-salt.git'
      EOF

bundle install [--path $PWD/.vendor/bundle]

# use with preffix 'bundle kitchen':
# bundle exec kitchen list

Create aliases:

cat > ~/.${SHELL}rc <-EOF
              alias bk='nocorrect bundle exec kitchen'
              alias kl='nocorrect bundle exec kitchen list'
EOF

See http://kitchen.ci/ for more details.

Install procedure

One may be satisfied installing ruby and gems system-wide right from OS package manager.

If you are an ruby/chef developer you will probably want to use ChefDK <https://downloads.chef.io/chefdk>.

For advanced users or the sake of complex environments you may use rbenv for user side ruby installation.

An example steps to install user side ruby and prerequisites:

# Use package manager to install rbenv and ruby-build
sudo apt-get install rbenv ruby-build

# list all available versions:
rbenv install -l

# install a Ruby version of your choice or pick latest
rbenv install $(rbenv install -l|grep -E '^[ ]*[0-9]\.[0-9]+'|tail -1)

# activate
rbenv local 2.4.0

# it's usually a good idea to update rubygems first
rbenv exec gem update --system

# install test kitchen
rbenv exec gem install bundler
rbenv exec gem install test-kitchen

Continue with the optional Gemfile in the formula main directory to fetch fine tuned dependencies. If you use Gemfile and Bundler for local dependencies prepend all command with rbenv exec bundler exec and possibly set an alias in your ~/.bashrc, etc.

cat >> ~/.${SHELL}rc <<-EOF
              alias rk="rbenv exec kitchen"
              alias bk="rbenv exec bundler exec kitchen"
EOF

With such alias set, you should be able to execute rbenv exec bundler exec make kitchen and see test results.

Sample Configurations

For advanced configs have a look at .kitchen*.yml examples in cookiecutter template <https://github.com/salt-formulas/salt-formulas/tree/master/cookiecutter/salt-formula/%7B%7Bcookiecutter.service_name%7D%7D>_.

.kitchen.yml

---
driver:
  name: docker
  hostname: opencontrail
  use_sudo: true

provisioner:
  name: salt_solo
  salt_install: bootstrap
  salt_bootstrap_url: https://bootstrap.saltstack.com
  salt_version: latest
  require_chef: false
  log_level: error
  formula: opencontrail
  grains:
    noservices: True
  dependencies:
    - name: linux
      repo: git
      source: https://github.com/salt-formulas/salt-formula-linux
  state_top:
    base:
      "*":
        - linux
        - opencontrail
  pillars:
    top.sls:
      base:
        "*":
          - linux_repo_openstack
          - linux_repo_cassandra
          - linux_repo_opencontrail
          - linux_repo_mos
          - linux
          - opencontrail
          - opencontrail_juniper
    linux.sls:
      linux:
        system:
          enabled: true
          name: opencontrail
    opencontrail_juniper.sls: {}
  pillars-from-files:
    linux_repo_mos.sls: tests/pillar/repo_mos8.sls
    linux_repo_cassandra.sls: tests/pillar/repo_cassandra.sls
    linux_repo_openstack.sls: tests/pillar/repo_openstack.sls
    linux_repo_opencontrail.sls: tests/pillar/repo_opencontrail.sls

verifier:
  name: inspec
  sudo: true

platforms:
  - name: <%= ENV['PLATFORM'] || 'ubuntu-xenial' %>
    driver_config:
      image: <%= ENV['PLATFORM'] || 'trevorj/salty-whales:xenial' %>
      platform: ubuntu

suites:

  - name: <%= ENV['SUITE'] || 'single' %>
    provisioner:
      pillars-from-files:
        opencontrail.sls: tests/pillar/<%= ENV['SUITE'] || 'single' %>.sls

  - name: cluster
    provisioner:
      pillars-from-files:
        opencontrail.sls: tests/pillar/cluster.sls

  - name: analytics
    provisioner:
      pillars-from-files:
        opencontrail.sls: tests/pillar/analytics.sls

  - name: control
    provisioner:
      pillars-from-files:
        opencontrail.sls: tests/pillar/control.sls

  - name: vendor-juniper
    provisioner:
      vendor_repo:
        - type: apt
          url: http://aptly.local/contrail
          key_url: http://aptly.local/public.gpg
          components: main
          distribution: trusty
      pillars-from-files:
        opencontrail.sls: tests/pillar/control.sls
      pillars:
        opencontrail_juniper.sls:
          opencontrail:
            common:
              vendor: juniper


# vim: ft=yaml sw=2 ts=2 sts=2 tw=125

Continous Integration with Travis

Salt-formulas uses Travis CI to run smoke and integration tests. To generate .travis.yml follow Generate test structures in formula.

Sample Configurations

.travis.yml

sudo: required
services:
  - docker

# PREREQUISITES
install:
  - pip install PyYAML
  - pip install virtualenv
  - |
    test -e Gemfile || cat <<EOF > Gemfile
    source 'https://rubygems.org'
    gem 'rake'
    gem 'test-kitchen'
    gem 'kitchen-docker'
    gem 'kitchen-inspec'
    gem 'inspec'
    gem 'kitchen-salt', :git => 'https://github.com/salt-formulas/kitchen-salt.git
  - bundle install

# BUILD MATRIX
env:
  - PLATFORM=trevorj/salty-whales:trusty
  - PLATFORM=trevorj/salty-whales:xenial
  - PLATFORM=trevorj/salty-whales:xenial-2016.3

# SMOKE TEST
before_script:
  - set -o pipefail
  - make test | tail

# KITCHEN TEST
script:
  - bundle exec kitchen test -t tests/integration

# vim: ft=yaml sw=2 ts=2 sts=2 tw=125

Common Practices

noservices

At some rare cases execution of given state in the formula is not possible or required. For these cases set grain noservices: True and wrap corresponding code as in the example below:

{%- if not grains.get('noservices', False) %}
mysql_database_{{ database_name }}:
  mysql_database.present:
  - name: {{ database_name }}
  - character_set: {{ database.get('encoding', 'utf8') }}
  - connection_user: {{ connection.user }}
  - connection_pass: {{ connection.password }}
  - connection_charset: {{ connection.charset }}
{%- endif %}

As the mysql database might not be available in the given test environment (travis/docker, etc..).

In .kitchen.yml we set grain noservices: True by default.

grains:
  noservices: True

** formula dependencies **

Formula dependencies might be specified in <formula repo>/metadata.yml

name: "galera"
version: "1.0"
source: "https://github.com/salt-formulas/salt-formula-galera"
dependencies:
- name: mysql
  source: "https://github.com/salt-formulas/salt-formula-mysql"

While using test-kitchen formula dependencies must be specified in .kitchen.yml as well. Dependencies may be installed from git, spm or even apt repository.

provisioner::
  dependencies:
    - name: mysql
      repo: git
      source: https://github.com/salt-formulas/salt-formula-mysql.git
    - name: linux
      repo: git
      source: https://github.com/salt-formulas/salt-formula-linux.git

For convenience kitchen-salt will read metadata.yml of these dependencies and install their dependencies in case you omit them in .kitchen.yml.

** build matrix **

To simplify local CI we ship .kitchen.yml with limited number of platforms. (ie: latest ubuntu as a falback option if no ENV variable PLATFORM is specified)

However this is later extended on Travis CI while using ENV variables in build matrix.

.travis.yml snippet:

# BUILD MATRIX
env:
  - PLATFORM=trevorj/salty-whales:trusty
  - PLATFORM=trevorj/salty-whales:xenial

.kitchen.yml snippet:

platforms:
  - name: <%= ENV['PLATFORM'] || 'ubuntu-xenial' %>
    driver_config:
      image: <%= ENV['PLATFORM'] || 'trevorj/salty-whales:xenial' %>
      platform: ubuntu