Home Salt-Formulas Development Documentation
Testing Salt Formulas¶
Each formula contains Makefile with at least test target.
Under tests
directory are located resources for test execution.
Test target executes “smoke test” implemented by tests/run_tests.sh capable to
fetch dependencies in python virtual environment by executing salt-call
state.show_sls with provided tests/pillar
data.
The purpose of the smoke test is to find syntax, typo issues and verify example pillar data against the formula.
Initial content of tests
folder contains test pillars and a run_tests.sh as generated by
cookiecutter.
tests
├── pillar
│ └── client_single.sls
│ └── server_single.sls
└── run_tests.sh
Create or update pillars in tests/pillar/\*.sls
with test data.
Generate Test Structures in Formula¶
There is and salt-formulas cookiecutter template. to generate initial repository structure for new formula.
For existing formulas there is an convenient script capable to generate
initial structures from available content. For more details follow the README
in the above linked repository. To simply generate test structures according
specification stated in this document simply run kitchen-init.sh
.
tl;dr:
curl -skL "https://raw.githubusercontent.com/salt-formulas/salt-formulas/master/cookiecutter/salt-formula/kitchen-init.sh" | bash -s --
Formula Testing with Test Kitchen¶
Test Kitchen with forked kitchen-salt provisioner plugin may be used for local development as well as CI scenario.
Test Kitchen is a test harness tool to execute your configured code on one or
more platforms in isolation. There is a .kitchen.yml
in main directory
that defines platforms to be tested and suites to execute on them.
Kitchen CI can spin instances locally or remote, based on used driver.
For example .kitchen.yml
may define a
docker or
vagrant driver.
For more, explore it’s rich ecosystem of supported drivers/provisioners/verifiers/…
Using Test Kitchen¶
A listing of scenarios to be executed:
$ kitchen list
Instance Driver Provisioner Verifier Transport Last Action
client-single-ubuntu-1404 Docker SaltSolo Inspec Ssh <Not Created>
client-single-ubuntu-1604 Docker SaltSolo Inspec Ssh <Not Created>
client-single-centos-71 Docker SaltSolo Inspec Ssh <Not Created>
The Busser Verifier is used to setup and run tests implementated in <repo>/test/integration. It installs the particular driver to tested instance (Serverspec, InSpec, Shell, Bats, …) prior the verification is executed.
Example workflow:
# list instances and status
kitchen list
# manually execute integration tests
kitchen [test || [create|converge|verify|exec|login|destroy|...]] [instance] -t tests/integration
# use with provided Makefile (ie: within CI pipeline)
make kitchen
How it Works¶
Kitchen spin an instances in Docker, Vagrant, OpenStack environment, etc.
based on configured driver. Instance is configured as salt minion, where the
configuration is defined by .kitchen.yml
and tests/pillar/*.sls
Override your specific needs with .kitchen.<backend|local>.yml
that you may load as:
KITCHEN_LOCAL_YAML=.kitchen.<driver>.yml kitchen <action> <suite>
.
Example: KITCHEN_LOCAL_YAML=.kitchen.local kitchen verify server-ubuntu-1404 -t tests/integration
.
Test Kitchen then allows you execute several action to perform your testing under configured conditions:
- create, provision an test instance (VM, container)
- converge, run a provisioner (shell script, kitchen-salt)
- verify, run a verification (inspec, other may be added)
- destroy
Verifying Deployment¶
There is couple of verifier plugins that are shipped with Test Kitchen. They allow to run simple bash scripts and checking it’s exit codes to run specific purpose based frameworks.
The Busser Verifier goes with test-kitchen by default.
It is used to setup and run tests implemented in <repo>/test/integration
. It guess and installs the particular driver to tested instance.
By default InSpec is expected.
You may avoid to install busser framework if you configure specific verifier in .kitchen.yml
:
verifier:
name: inspec
For default Inspec Verifier implement your scripts directly in <repo>/test/integration/<suite>
directory with _spec.rb
suffix.
If you would to write another verification scripts than InSpec store them in <repo>/tests/integration/<suite>/<verifier>
.
Busser <https://github.com/test-kitchen/busser>
is a test setup and execution framework under test kitchen.
Implement integration tests under <repo>/tests/integration/<suite>/<verifier>
directory with _spec.<verifier suffix>
filename
suffix.
InSpec
InSpec is native validation framework for Test Kitchen and as such don’t require usage of <verifier>
folder. Thus the tests may by
stored directly under <repo>/tests/integration/<suite>
Additional resources.
Example verification scripts under tests/integration folder of the formula:
tests
├── integration
│ ├── default
│ │ └── default_testcase_spec.rb # Written in InSpec
│ ├── backupmx
│ │ └── serverspec # <Verifier framework>
│ │ └── backupmx_spec.rb # Written in ServerSpec
│ ├── helpers
│ │ └── serverspec
│ │ └── spec_helper.rb
│ ├── relay
│ │ └── serverspec
│ │ └── relay_spec.rb
│ └── server
│ └── serverspec
│ ├── aliases_spec.rb
│ └── server_spec.rb
├── pillar
│ ├── backupmx.sls
│ ├── relay.sls
│ └── server.sls
└── run_tests.sh
Requirements¶
Use latest stable kitchen-salt <https://github.com/saltstack/kitchen-salt>
and kitchen-test.
TL;DR
First you have to install ruby package manager gem.
Install required gems:
# Ruby side:
gem install <gem name from the list below>
# Isolated w/Bundler
gem install bundler
cat > Gemfile <-EOF
source 'https://rubygems.org'
gem 'rake'
gem 'test-kitchen'
gem 'kitchen-docker'
gem 'kitchen-inspec'
gem 'inspec'
gem 'kitchen-salt', :git => 'https://github.com/salt-formulas/kitchen-salt.git'
EOF
bundle install [--path $PWD/.vendor/bundle]
# use with preffix 'bundle kitchen':
# bundle exec kitchen list
Create aliases:
cat > ~/.${SHELL}rc <-EOF
alias bk='nocorrect bundle exec kitchen'
alias kl='nocorrect bundle exec kitchen list'
EOF
See http://kitchen.ci/ for more details.
Install procedure
One may be satisfied installing ruby and gems system-wide right from OS package manager.
If you are an ruby/chef developer you will probably want to use ChefDK <https://downloads.chef.io/chefdk>.
For advanced users or the sake of complex environments you may use rbenv for user side ruby installation.
An example steps to install user side ruby and prerequisites:
# Use package manager to install rbenv and ruby-build
sudo apt-get install rbenv ruby-build
# list all available versions:
rbenv install -l
# install a Ruby version of your choice or pick latest
rbenv install $(rbenv install -l|grep -E '^[ ]*[0-9]\.[0-9]+'|tail -1)
# activate
rbenv local 2.4.0
# it's usually a good idea to update rubygems first
rbenv exec gem update --system
# install test kitchen
rbenv exec gem install bundler
rbenv exec gem install test-kitchen
Continue with the optional Gemfile
in the formula main directory to fetch fine tuned dependencies.
If you use Gemfile and Bundler for local dependencies prepend all command with
rbenv exec bundler exec
and possibly set an alias in your ~/.bashrc, etc.
cat >> ~/.${SHELL}rc <<-EOF
alias rk="rbenv exec kitchen"
alias bk="rbenv exec bundler exec kitchen"
EOF
With such alias set, you should be able to execute rbenv exec bundler exec
make kitchen
and see test results.
Sample Configurations¶
For advanced configs have a look at .kitchen*.yml
examples in cookiecutter template <https://github.com/salt-formulas/salt-formulas/tree/master/cookiecutter/salt-formula/%7B%7Bcookiecutter.service_name%7D%7D>_.
.kitchen.yml
---
driver:
name: docker
hostname: opencontrail
use_sudo: true
provisioner:
name: salt_solo
salt_install: bootstrap
salt_bootstrap_url: https://bootstrap.saltstack.com
salt_version: latest
require_chef: false
log_level: error
formula: opencontrail
grains:
noservices: True
dependencies:
- name: linux
repo: git
source: https://github.com/salt-formulas/salt-formula-linux
state_top:
base:
"*":
- linux
- opencontrail
pillars:
top.sls:
base:
"*":
- linux_repo_openstack
- linux_repo_cassandra
- linux_repo_opencontrail
- linux_repo_mos
- linux
- opencontrail
- opencontrail_juniper
linux.sls:
linux:
system:
enabled: true
name: opencontrail
opencontrail_juniper.sls: {}
pillars-from-files:
linux_repo_mos.sls: tests/pillar/repo_mos8.sls
linux_repo_cassandra.sls: tests/pillar/repo_cassandra.sls
linux_repo_openstack.sls: tests/pillar/repo_openstack.sls
linux_repo_opencontrail.sls: tests/pillar/repo_opencontrail.sls
verifier:
name: inspec
sudo: true
platforms:
- name: <%= ENV['PLATFORM'] || 'ubuntu-xenial' %>
driver_config:
image: <%= ENV['PLATFORM'] || 'trevorj/salty-whales:xenial' %>
platform: ubuntu
suites:
- name: <%= ENV['SUITE'] || 'single' %>
provisioner:
pillars-from-files:
opencontrail.sls: tests/pillar/<%= ENV['SUITE'] || 'single' %>.sls
- name: cluster
provisioner:
pillars-from-files:
opencontrail.sls: tests/pillar/cluster.sls
- name: analytics
provisioner:
pillars-from-files:
opencontrail.sls: tests/pillar/analytics.sls
- name: control
provisioner:
pillars-from-files:
opencontrail.sls: tests/pillar/control.sls
- name: vendor-juniper
provisioner:
vendor_repo:
- type: apt
url: http://aptly.local/contrail
key_url: http://aptly.local/public.gpg
components: main
distribution: trusty
pillars-from-files:
opencontrail.sls: tests/pillar/control.sls
pillars:
opencontrail_juniper.sls:
opencontrail:
common:
vendor: juniper
# vim: ft=yaml sw=2 ts=2 sts=2 tw=125
Continous Integration with Travis¶
Salt-formulas uses Travis CI to run smoke and integration tests. To generate
.travis.yml
follow Generate test structures in formula.
Sample Configurations¶
.travis.yml
sudo: required
services:
- docker
# PREREQUISITES
install:
- pip install PyYAML
- pip install virtualenv
- |
test -e Gemfile || cat <<EOF > Gemfile
source 'https://rubygems.org'
gem 'rake'
gem 'test-kitchen'
gem 'kitchen-docker'
gem 'kitchen-inspec'
gem 'inspec'
gem 'kitchen-salt', :git => 'https://github.com/salt-formulas/kitchen-salt.git
- bundle install
# BUILD MATRIX
env:
- PLATFORM=trevorj/salty-whales:trusty
- PLATFORM=trevorj/salty-whales:xenial
- PLATFORM=trevorj/salty-whales:xenial-2016.3
# SMOKE TEST
before_script:
- set -o pipefail
- make test | tail
# KITCHEN TEST
script:
- bundle exec kitchen test -t tests/integration
# vim: ft=yaml sw=2 ts=2 sts=2 tw=125
Common Practices¶
noservices
At some rare cases execution of given state in the formula is not possible or required.
For these cases set grain noservices: True
and wrap corresponding code as in
the example below:
{%- if not grains.get('noservices', False) %}
mysql_database_{{ database_name }}:
mysql_database.present:
- name: {{ database_name }}
- character_set: {{ database.get('encoding', 'utf8') }}
- connection_user: {{ connection.user }}
- connection_pass: {{ connection.password }}
- connection_charset: {{ connection.charset }}
{%- endif %}
As the mysql database might not be available in the given test environment (travis/docker, etc..).
In .kitchen.yml
we set grain noservices: True
by default.
grains:
noservices: True
** formula dependencies **
Formula dependencies might be specified in <formula repo>/metadata.yml
name: "galera"
version: "1.0"
source: "https://github.com/salt-formulas/salt-formula-galera"
dependencies:
- name: mysql
source: "https://github.com/salt-formulas/salt-formula-mysql"
While using test-kitchen formula dependencies must be specified in .kitchen.yml
as well.
Dependencies may be installed from git, spm or even apt repository.
provisioner::
dependencies:
- name: mysql
repo: git
source: https://github.com/salt-formulas/salt-formula-mysql.git
- name: linux
repo: git
source: https://github.com/salt-formulas/salt-formula-linux.git
For convenience kitchen-salt will read metadata.yml
of these dependencies
and install their dependencies in case you omit them in .kitchen.yml
.
** build matrix **
To simplify local CI we ship .kitchen.yml
with limited number of platforms.
(ie: latest ubuntu as a falback option if no ENV
variable PLATFORM
is specified)
However this is later extended on Travis CI while using ENV
variables in build matrix.
.travis.yml
snippet:
# BUILD MATRIX
env:
- PLATFORM=trevorj/salty-whales:trusty
- PLATFORM=trevorj/salty-whales:xenial
.kitchen.yml
snippet:
platforms:
- name: <%= ENV['PLATFORM'] || 'ubuntu-xenial' %>
driver_config:
image: <%= ENV['PLATFORM'] || 'trevorj/salty-whales:xenial' %>
platform: ubuntu