Home SaltStack-Formulas Development Documentation

Formula Authoring Guidelines

Salt formulas encapsulate specific services. This document contains guidelines to salt formula creation and maintenance.

Formula Directory Structure

Formulas follow the same directory structure as Salt official conventions and best practices described in the SaltStack documentation.

Every formula should have the following directory layout:

service-formula
|-- _grains/
|   `-- service.yml
|-- _modules/
|   `-- service.yml
|-- _states/
|   `-- service.yml
|-- service/
|   `-- files/
|       |-- service.conf
|       `-- service-systemd
|   `-- meta/
|       |-- sphinx.yml
|       `-- colletd.yml
|   |-- map.jinja
|   |-- init.sls
|   |-- _common.sls
|   |-- role1.sls
|   `-- role2/
|       |-- init.sls
|       |-- service.sls
|       `-- more.sls
|-- debian/
|   ├── changelog
|   ├── compat
|   ├── control
|   ├── copyright
|   ├── docs
|   ├── install
|   ├── rules
|   └── source
|       └── format
|-- metadata/
|   `-- service/
|       |-- role1/
|       |   |-- deployment1.yml
|       |   `-- deployment2.yml
|       `-- role2/
|           `-- deployment3.yml
|-- CHANGELOG.rst
|-- LICENSE
|-- pillar.example
|-- README.rst
`-- VERSION

Content of the formula directories in more detail.

_grains/
Optional grain modules
_modules/
Optional execution modules
_states/
Optional state modules
service/
Salt state files
service/meta/
Support metadata definitions
debian/
APT Package metadata
metadata/
Reclass metadata

Salt state files

Salt state files are located in service directory.

service/map.jinja

Map file helps to clean the differences among operating systems and provides default values so there’s no need to provide default value in state files.

Following snippet uses YAML to serialize the data and is the recommended way to write map.jinja file as YAML can be easily extended in place.

{%- load_yaml as role1_defaults %}
Debian:
  pkgs:
  - python-psycopg2
  dir:
    base: /srv/service/venv
    home: /var/lib/service
RedHat:
  pkgs:
  - python-psycopg2
  dir:
    base: /srv/service/venv
    home: /var/lib/service
    workspace: /srv/service/workspace
{%- endload %}

{%- set role1 = salt['grains.filter_by'](role1_defaults, merge=salt['pillar.get']('service:role1')) %}

Following snippet uses JSON to serialize the data and was favored in past.

{% set api = salt['grains.filter_by']({
    'Debian': {
        'pkgs': ['salt-api'],
        'service': 'salt-api',
    },
    'RedHat': {
        'pkgs': ['salt-api'],
        'service': 'salt-api',
    },
}, merge=salt['pillar.get']('salt:api')) %}

Following snippet sets different common role parameters according to service:role:source:engine pillar variable of given service role.

{%- set source_engine = salt['pillar.get']('service:role:source:engine') %}

{%- load_yaml as base_defaults %}
{%- if source_engine == 'git' %}
Debian:
  pkgs:
  - python-psycopg2
  dir:
    base: /srv/service/venv
    home: /var/lib/service
    workspace: /srv/service/workspace
{%- else %}
Debian:
  pkgs:
  - helpdesk
  dir:
    base: /usr/lib/service
{%- endif %}
{%- endload %}

service/init.sls

Conditional include of individual service roles. Basically this is essential piece that makes the usage of formulas truly model-driven. You have catalog of services and this determines according to present metadata what roles get started.

Using service/init.sls file allows the service catalog to be role independent.

include:
{% if pillar.service.role1 is defined %}
- service.role1
{% endif %}
{% if pillar.service.role2 is defined %}
- service.role2
{% endif %}

You can use one file as role1.sls for simple roles. For more complex roles handling many resources, use individual directories as role2.

service-formula/
`-- service/
    |-- role1.sls
    `-- role2/
        |-- init.sls
        |-- service.sls
        |-- resource1.sls
        `-- resource2.sls

Then you can verify the full service catalog on node by following command:

root@web01:~# salt-call state.show_top
[INFO    ] Loading fresh modules for state activity
local:
    ----------
    base:
        - linux
        - openssh
        - ntp
        - salt
        - backupninja
        - git
        - sphinx
        - python
        - nginx
        - nodejs
        - postgresql
        - rabbitmq
        - redis
        - ruby

Service metadata are stored also as services grain.

root@web01:~# salt-call grains.item services
local:
    ----------
    services:
        - linux
        - openssh
        - ntp
        - salt
        - backupninja
        - git
        - sphinx
        - python
        - nginx
        - nodejs
        - postgresql
        - rabbitmq
        - redis
        - ruby

And each service roles metadata is stored as detailed roles grain.

root@web01:~# salt-call grains.item roles
local:
    ----------
    roles:
        - git.client
        - postgresql.server
        - nodejs.environment
        - ntp.client
        - linux.storage
        - linux.system
        - linux.network
        - redis.server
        - rabbitmq.server
        - python.environment
        - backupninja.client
        - nginx.server
        - openssh.client
        - openssh.server
        - salt.minion
        - sphinx.server

Note

It is recommended to run state.sls salt prior the state.highstate command as grains may not be generated properly and some configuration parameters may not be set at all.

service/role1.sls

Actual salt state resources that enforce service existence. Common production and recommended pattern is to install packages, setup configuration files and ensure the service is up and running.

{%- from "redis/map.jinja" import server with context %}
{%- if server.enabled %}

redis_packages:
  pkg.installed:
  - names: {{ server.pkgs }}

{{ server.dir.conf }}/redis.conf:
  file.managed:
  - source: salt://redis/files/redis.conf
  - template: jinja
  - user: root
  - group: root
  - mode: 644
  - require:
    - pkg: redis_packages

redis_service:
  service.running:
  - enable: true
  - name: {{ server.service }}
  - watch:
    - file: {{ server.dir.conf }}/redis.conf

{%- endif %}

For development purposes other installation than s

Note

The role for role.enabled condition is to restrict the give service role from execution with default parametes, the single error is thrown instead. You can optionaly add else statement to disable or completely remove given service role.

service/role2/init.sls

This approach is used with more complex roles, it is similar to service/init.sls, but uses conditions to further limit the inclusion of unnecessary files.

For example Linux network role includes conditionally hosts and interfaces.

{%- from "linux/map.jinja" import network with context %}
include:
- linux.network.hostname
{%- if network.host|length > 0 %}
- linux.network.host
{%- endif %}
{%- if network.interface|length > 0 %}
- linux.network.interface
{%- endif %}
- linux.network.proxy

Coding styles for state files

Good styling practices for writing salt state declarations.

Line length above 80 characters

As a ‘standard code width limit’ and for historical reasons - IBM punch card had exactly 80 columns.

Single line declaration

Avoid extending your code by adding single-line declarations. It makes your code much cleaner and easier to parse / grep while searching for those declarations.

The bad example:

python:
  pkg:
    - installed

The correct example:

python:
  pkg.installed

No newline at the end of the file

Each line should be terminated in a newline character, including the last one. Some programs have problems processing the last line of a file if it isn’t newline terminated.

Trailing whitespace characters

Trailing whitespaces take more spaces than necessary, any regexp based searches won’t return lines as a result due to trailing whitespace(s).

Reclass metadata files

Each of these files serve as default metadata set for given deployment. Each service role can have several deployments. For example rabbitmq server role has following deployments:

  • metadata/rabbitmq/server/local.yml
  • metadata/rabbitmq/server/single.yml
  • metadata/rabbitmq/server/cluster.yml

metadata/service/role1/local.yml

applications:
- rabbitmq
parameters:
  _param:
    rabbitmq_admin_user: admin
  rabbitmq:
    server:
      enabled: true
      secret_key: ${_param:rabbitmq_secret_key}
      bind:
        address: 127.0.0.1
        port: 5672
      plugins:
      - amqp_client
      - rabbitmq_management
      admin:
        name: ${_param:rabbitmq_admin_user}
        password: ${_param:rabbitmq_admin_password}

metadata/service/role1/single.yml

applications:
- rabbitmq
parameters:
  _param:
    rabbitmq_admin_user: admin
  rabbitmq:
    server:
      enabled: true
      secret_key: ${_param:rabbitmq_secret_key}
      bind:
        address: 0.0.0.0
        port: 5672
      plugins:
      - amqp_client
      - rabbitmq_management
      admin:
        name: ${_param:rabbitmq_admin_user}
        password: ${_param:rabbitmq_admin_password}

metadata/service/role1/cluster.yml

applications:
- rabbitmq
parameters:
  rabbitmq:
    server:
      enabled: true
      secret_key: ${_param:rabbitmq_secret_key}
      bind:
        address: ${_param:cluster_local_address}
        port: 5672
      plugins:
      - amqp_client
      - rabbitmq_management
      admin:
        name: admin
        password: ${_param:rabbitmq_admin_password}
      host:
        '/openstack':
          enabled: true
          user: openstack
          password: ${_param:rabbitmq_openstack_password}
          policies:
          - name: HA
            pattern: '^(?!amq\.).*'
            definition: '{"ha-mode": "all"}'
    cluster:
      enabled: true
      name: openstack
      role: ${_param:rabbitmq_cluster_role}
      master: ${_param:cluster_node01_hostname}
      mode: disc
      members:
      - name: ${_param:cluster_node01_hostname}
        host: ${_param:cluster_node01_address}
      - name: ${_param:cluster_node02_hostname}
        host: ${_param:cluster_node02_address}
      - name: ${_param:cluster_node03_hostname}
        host: ${_param:cluster_node03_address}

Parameters like ${_param:rabbitmq_secret_key} are interpolation of common parameter passed from higher system or cluster levels.

Debian packaging

Use of debian packaging is preferable way for deploying production salt masters and it’s formulas. Take basic structure of debian directory from some existing formula and modify to suit your formula.

Description of most important files follows.

debian/changelog

salt-formula-salt (0.1) trusty; urgency=medium

  + Initial release

 -- Ales Komarek <ales.komarek@tcpcloud.eu> Thu, 13 Aug 2015 23:23:41 +0200

debian/docs

Files listed here will be available in /usr/share/doc. Don’t put COPYRIGHT or LICENSE files here as they are handled in a different way.

README.rst
CHANGELOG.rst
VERSION

debian/install

Defines what is going to be installed in which location.

salt/*                  /usr/share/salt-formulas/env/salt/
metadata/service/*      /usr/share/salt-formulas/reclass/service/salt/

debian/control

This file keeps metadata of source and binary package.

Source: salt-formula-salt
Maintainer: tcpcloud Packaging Team <pkg-team@tcpcloud.eu>
Section: admin
Priority: optional
Build-Depends: debhelper (>= 9)
Standards-Version: 3.9.6
Homepage: http://www.tcpcloud.eu
Vcs-Browser: https://github.com/tcpcloud/salt-formula-salt
Vcs-Git: https://github.com/tcpcloud/salt-formula-salt.git

Package: salt-formula-salt
Architecture: all
Depends: ${misc:Depends}, salt-master, reclass
Description: Salt salt formula
 Install and configure Salt masters and minions.

Supplemental files

Files that are required to complete information about given formula.

README.rst

A sample skeleton of the README.rst file:

=======
service
=======

Install and configure the Specific service.

.. note::

    See the full `Salt Formulas installation and usage instructions
    <http://docs.saltstack.com/en/latest/topics/development/conventions/formulas.html>`_.

Available states
================

.. contents::
    :local:

``service``
-----------

Install the ``service`` package and enable the service.

``service.role1``
-----------------

Setup individual role.


Available metadata
==================

.. contents::
    :local:

``metadata.service.role.single``
----------------------------------

Setup from system packages.


``metadata.service.role.development`
--------------------------------------

Setup from git repository.


Configuration parameters
========================

.. contents::
    :local:

``service_secret_key``
------------------------------

``rabbitmq_service_password``
-------------------------------------

``postgresql_service_password``
---------------------------------------

If development is setup.

``service_source_revision``
---------------------------

If development is setup.

Example reclass
===============

Production setup

.. code-block:: yaml

    service-single:
      name: service-single
      domain: dev.domain.com
      classes:
      - system.service.server.single
      params:
        rabbitmq_admin_password: cwerfwefzdcdsf
        rabbitmq_secret_key: fsdfwfdsfdsf
        rabbitmq_service_password: fdsf24fsdfsdacadf
        keystone_service_password: fdasfdsafdasfdasfda
        postgresql_service_password: dfdasfdafdsa
        nginx_site_service_host: ${linux:network:fqdn}
        service_secret_key: fda32r

Development setup

.. code-block:: yaml

    service-single:
      name: service-single
      domain: dev.domain.com
      classes:
      - system.service.server.development
      params:
        rabbitmq_admin_password: cwerfwefzdcdsf
        rabbitmq_secret_key: fsdfwfdsfdsf
        rabbitmq_service_password: fdsf24fsdfsdacadf
        keystone_service_password: fdasfdsafdasfdasfda
        postgresql_service_password: dfdasfdafdsa
        nginx_site_service_host: ${linux:network:fqdn}
        service_secret_key: fda32r
        service_source_repository: git@git.tcpcloud.eu:python-apps/service.git
        service_source_revision: feature/243


Example pillar
==============

Install from specific branch of Git

.. code-block:: yaml

   service:
     server:
       source:
         engine: 'git'
         address: 'git@git.tcpcloud.eu:python-apps/service.git'
         revision: 'feature/214'

To enable debug logging for both Django and Gunicorn and raise
number of Gunicorn workers

.. code-block:: yaml

   service:
     server:
       log_level: 'debug'
       workers: 8

To change where Django listens

.. code-block:: yaml

   service:
     server:
       bind:
         address: 'not-localhost'
         port: 9755

Read more
=========

* http://doc.tcpcloud.eu/

LICENSE

Contains license information and terms & conditions how you are allowed to use and distribute the files of the underlying directories.

Copyright (c) 2014-2015 Your name

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

   http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

VERSION

Latest version number, git repository tag, package version as well.

0.0.2

CHANGELOG.rst

The CHANGELOG.rst file should detail the individual versions, their release date and a set of bullet points for each version highlighting the overall changes in a given version of the formula.

A sample skeleton of the CHANGELOG.rst file:

CHANGELOG.rst:

service formula
===============

0.0.2 (2014-01-01)

- Re-organized formula file layout
- Fixed filename used for upstart logger template
- Allow for pillar message to have default if none specified

0.0.1 (2013-01-01)

- Initial formula setup

Versioning

Formula are versioned according to Semantic Versioning, http://semver.org/.

Note

Given a version number MAJOR.MINOR.PATCH, increment the:

  1. MAJOR version when you make incompatible API changes,
  2. MINOR version when you add functionality in a backwards-compatible manner, and
  3. PATCH version when you make backwards-compatible bug fixes.

Additional labels for pre-release and build metadata are available as extensions to the MAJOR.MINOR.PATCH format.

Formula versions are tracked using Git tags as well as the VERSION file in the formula repository. The VERSION file should contain the currently released version of the particular formula.

Formula unit testing

A smoke-test for invalid Jinja, invalid YAML, or an invalid Salt state structure can be performed by with the state.show_sls function:

salt '*' state.show_sls service-name

Salt Formulas can then be tested by running each .sls file via state.sls and checking the output for the success or failure of each state in the Formula. This should be done for each supported platform.

salt '*' state.sls sls-file-name test