Kubernetes Formula¶
Kubernetes is an open-source system for automating deployment, scaling, and management of containerized applications. This formula deploys production ready Kubernetes and generate Kubernetes manifests as well.
You can download kubectl configuration and connect to your cluster. However, keep in mind kubernetes_control_address needs to be accessible from your computer:
mkdir -p ~/.kube
[ -f ~/.kube/config ] && cp -v ~/.kube/config ~/.kube/config-backup
ssh cfg01 "sudo ssh ctl01 /etc/kubernetes/kubeconfig.sh" > ~/.kube/config
kubectl get no
cfg01 is Salt master node and ctl01 is one of Kubernetes masters
Sample Pillars¶
REQUIRED: Define image to use for hyperkube, CNIs and calicoctl image
parameters:
kubernetes:
common:
hyperkube:
image: gcr.io/google_containers/hyperkube:v1.6.5
pool:
network:
calicoctl:
image: calico/ctl
cni:
image: calico/cni
Enable helm-tiller addon
parameters:
kubernetes:
common:
addons:
helm:
enabled: true
Enable calico-policy addon
parameters:
kubernetes:
common:
addons:
calico_policy:
enabled: true
Enable virtlet addon
parameters:
kubernetes:
common:
addons:
virtlet:
enabled: true
namespace: kube-system
image: mirantis/virtlet:v0.8.0
hosts:
- cmp01
- cmp02
Enable netchecker addon
parameters:
kubernetes:
common:
addons:
netchecker:
enabled: true
master:
namespace:
netchecker:
enabled: true
Enable Kubenetes Federation control plane
parameters:
kubernetes:
master:
federation:
enabled: True
name: federation
namespace: federation-system
source: https://dl.k8s.io/v1.6.6/kubernetes-client-linux-amd64.tar.gz
hash: 94b2c9cd29981a8e150c187193bab0d8c0b6e906260f837367feff99860a6376
service_type: NodePort
dns_provider: coredns
childclusters:
- secondcluster.mydomain
- thirdcluster.mydomain
Enable external DNS addon with CoreDNS provider
parameters:
kubernetes:
common:
addons:
coredns:
enabled: True
externaldns:
enabled: True
domain: company.mydomain
provider: coredns
Enable external DNS addon with Designate provider
parameters:
kubernetes:
common:
addons:
externaldns:
enabled: True
domain: company.mydomain
provider: designate
designate_os_options:
OS_AUTH_URL: https://keystone_auth_endpoint:5000
OS_PROJECT_DOMAIN_NAME: default
OS_USER_DOMAIN_NAME: default
OS_PROJECT_NAME: admin
OS_USERNAME: admin
OS_PASSWORD: password
OS_REGION_NAME: RegionOne
Enable external DNS addon with AWS provider
parameters:
kubernetes:
common:
addons:
externaldns:
enabled: True
domain: company.mydomain
provider: aws
aws_options:
AWS_ACCESS_KEY_ID: XXXXXXXXXXXXXXXXXXXX
AWS_SECRET_ACCESS_KEY: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Enable external DNS addon with Google CloudDNS provider
parameters:
kubernetes:
common:
addons:
externaldns:
enabled: True
domain: company.mydomain
provider: google
google_options:
key: ''
project: default-123
key should be exported from google console and processed as cat key.json | tr -d ‘n’
Enable OpenStack cloud provider
parameters:
kubernetes:
common:
cloudprovider:
enabled: True
provider: openstack
params:
auth_url: https://openstack.mydomain:5000/v3
username: nova
password: nova
region: RegionOne
tenant_id: 4bce4162d8744c599e350099cfa22a0a
domain_name: default
subnet_id: 72407854-aca6-4cf1-b873-e9affb09484b
lb_version: v2
Configure service verbosity
parameters:
kubernetes:
master:
verbosity: 2
pool:
verbosity: 2
Set cluster name and domain
parameters:
kubernetes:
common:
kubernetes_cluster_domain: mycluster.domain
cluster_name : mycluster
Enable autoscaler for dns addon. Poll period can be skipped.
kubernetes:
common:
addons:
dns:
domain: cluster.local
enabled: true
replicas: 1
server: 10.254.0.10
autoscaler:
enabled: true
poll-period-seconds: 60
Pass aditional parameters to daemons:
parameters:
kubernetes:
master:
apiserver:
daemon_opts:
storage-backend: pigeon
controller_manager:
daemon_opts:
log-dir: /dev/nulL
pool:
kubelet:
daemon_opts:
max-pods: "6"
Containers on pool definitions in pool.service.local
parameters:
kubernetes:
pool:
service:
local:
enabled: False
service: libvirt
cluster: openstack-compute
namespace: default
role: ${linux:system:name}
type: LoadBalancer
kind: Deployment
apiVersion: extensions/v1beta1
replicas: 1
host_pid: True
nodeSelector:
- key: openstack
value: ${linux:system:name}
hostNetwork: True
container:
libvirt-compute:
privileged: True
image: ${_param:docker_repository}/libvirt-compute
tag: ${_param:openstack_container_tag}
Master definition
kubernetes:
common:
cluster_name: cluster
addons:
dns:
domain: cluster.local
enabled: true
replicas: 1
server: 10.254.0.10
master:
admin:
password: password
username: admin
apiserver:
address: 10.0.175.100
secure_port: 443
insecure_address: 127.0.0.1
insecure_port: 8080
ca: kubernetes
enabled: true
etcd:
host: 127.0.0.1
members:
- host: 10.0.175.100
name: node040
name: node040
token: ca939ec9c2a17b0786f6d411fe019e9b
kubelet:
allow_privileged: true
network:
engine: calico
mtu: 1500
hash: fb5e30ebe6154911a66ec3fb5f1195b2
private_ip_range: 10.150.0.0/16
version: v0.19.0
service_addresses: 10.254.0.0/16
storage:
engine: glusterfs
members:
- host: 10.0.175.101
port: 24007
- host: 10.0.175.102
port: 24007
- host: 10.0.175.103
port: 24007
port: 24007
token:
admin: DFvQ8GJ9JD4fKNfuyEddw3rjnFTkUKsv
controller_manager: EreGh6AnWf8DxH8cYavB2zS029PUi7vx
dns: RAFeVSE4UvsCz4gk3KYReuOI5jsZ1Xt3
kube_proxy: DFvQ8GelB7afH3wClC9romaMPhquyyEe
kubelet: 7bN5hJ9JD4fKjnFTkUKsvVNfuyEddw3r
logging: MJkXKdbgqRmTHSa2ykTaOaMykgO6KcEf
monitoring: hnsj0XqABgrSww7Nqo7UVTSZLJUt2XRd
scheduler: HY1UUxEPpmjW4a1dDLGIANYQp1nZkLDk
version: v1.2.4
kubernetes:
pool:
address: 0.0.0.0
allow_privileged: true
ca: kubernetes
cluster_dns: 10.254.0.10
cluster_domain: cluster.local
enabled: true
kubelet:
allow_privileged: true
config: /etc/kubernetes/manifests
frequency: 5s
master:
apiserver:
members:
- host: 10.0.175.100
etcd:
members:
- host: 10.0.175.100
host: 10.0.175.100
network:
engine: calico
mtu: 1500
hash: fb5e30ebe6154911a66ec3fb5f1195b2
version: v0.19.0
token:
kube_proxy: DFvQ8GelB7afH3wClC9romaMPhquyyEe
kubelet: 7bN5hJ9JD4fKjnFTkUKsvVNfuyEddw3r
version: v1.2.4
Enable basic, token and http authentication, disable ssl auth, create some static users:
kubernetes:
master:
auth:
basic:
enabled: true
user:
jdoe:
password: dummy
groups:
- system:admin
http:
enabled: true
header:
user: X-Remote-User
group: X-Remote-Group
ssl:
enabled: false
token:
enabled: true
user:
jdoe:
token: dummytoken
groups:
- system:admin
Kubernetes with OpenContrail network plugin¶
On Master:
kubernetes:
common:
addons:
contrail_network_controller:
enabled: true
namespace: kube-system
image: yashulyak/contrail-controller:latest
master:
network:
engine: opencontrail
default_domain: default-domain
default_project: default-domain:default-project
public_network: default-domain:default-project:Public
public_ip_range: 185.22.97.128/26
private_ip_range: 10.150.0.0/16
service_cluster_ip_range: 10.254.0.0/16
network_label: name
service_label: uses
cluster_service: kube-system/default
config:
api:
host: 10.0.170.70
On pools:
kubernetes:
pool:
network:
engine: opencontrail
Dashboard public IP must be configured when Contrail network is used:
kubernetes:
common:
addons:
public_ip: 1.1.1.1
Kubernetes control plane running in systemd¶
By default kube-apiserver, kube-scheduler, kube-controllermanager, kube-proxy, etcd running in docker containers through manifests. For stable production environment this should be run in systemd.
kubernetes:
master:
container: false
kubernetes:
pool:
container: false
Because k8s services run under kube user without root privileges, there is need to change secure port for apiserver.
kubernetes:
master:
apiserver:
secure_port: 8081
Kubernetes with Flannel¶
On Master:
kubernetes:
master:
network:
engine: flannel
# If you don't register master as node:
etcd:
members:
- host: 10.0.175.101
port: 4001
- host: 10.0.175.102
port: 4001
- host: 10.0.175.103
port: 4001
common:
network:
engine: flannel
On pools:
kubernetes:
pool:
network:
engine: flannel
etcd:
members:
- host: 10.0.175.101
port: 4001
- host: 10.0.175.102
port: 4001
- host: 10.0.175.103
port: 4001
common:
network:
engine: flannel
Kubernetes with Calico¶
On Master:
kubernetes:
master:
network:
engine: calico
mtu: 1500
# If you don't register master as node:
etcd:
members:
- host: 10.0.175.101
port: 4001
- host: 10.0.175.102
port: 4001
- host: 10.0.175.103
port: 4001
On pools:
kubernetes:
pool:
network:
engine: calico
mtu: 1500
etcd:
members:
- host: 10.0.175.101
port: 4001
- host: 10.0.175.102
port: 4001
- host: 10.0.175.103
port: 4001
Running with secured etcd:
kubernetes:
pool:
network:
engine: calico
mtu: 1500
etcd:
ssl:
enabled: true
master:
network:
engine: calico
etcd:
ssl:
enabled: true
Running with calico-policy controller:
kubernetes:
pool:
network:
engine: calico
mtu: 1500
addons:
calico_policy:
enabled: true
master:
network:
engine: calico
mtu: 1500
addons:
calico_policy:
enabled: true
Enable Prometheus metrics in Felix
kubernetes:
pool:
network:
prometheus:
enabled: true
master:
network:
prometheus:
enabled: true
Post deployment configuration
# set ETCD
export ETCD_AUTHORITY=10.0.111.201:4001
# Set NAT for pods subnet
calicoctl pool add 192.168.0.0/16 --nat-outgoing
# Status commands
calicoctl status
calicoctl node show
Kubernetes with GlusterFS for storage¶
kubernetes:
master:
...
storage:
engine: glusterfs
port: 24007
members:
- host: 10.0.175.101
port: 24007
- host: 10.0.175.102
port: 24007
- host: 10.0.175.103
port: 24007
...
Kubernetes Storage Class¶
AWS EBS storageclass integration. It also requires to create IAM policy and profiles for instances and tag all resources by KubernetesCluster in EC2.
kubernetes:
common:
addons:
storageclass:
aws_slow:
enabled: True
default: True
provisioner: aws-ebs
name: slow
type: gp2
iopspergb: "10"
zones: xxx
nfs_shared:
name: elasti01
enabled: True
provisioner: nfs
spec:
name: elastic_data
nfs:
server: 10.0.0.1
path: /exported_path
Kubernetes namespaces¶
Create namespace:
kubernetes:
master:
...
namespace:
kube-system:
enabled: True
namespace2:
enabled: True
namespace3:
enabled: False
...
Kubernetes labels¶
Label node:
kubernetes:
master:
label:
label01:
value: value01
node: node01
enabled: true
key: key01
...
Pull images from private registries¶
kubernetes:
master:
...
registry:
secret:
registry01:
enabled: True
key: (get from `cat /root/.docker/config.json | base64`)
namespace: default
...
control:
...
service:
service01:
...
image_pull_secretes: registry01
...
Kubernetes Service Definitions in pillars¶
Following samples show how to generate kubernetes manifest as well and provide single tool for complete infrastructure management.
Deployment manifest¶
salt:
control:
enabled: True
hostNetwork: True
service:
memcached:
privileged: True
service: memcached
role: server
type: LoadBalancer
replicas: 3
kind: Deployment
apiVersion: extensions/v1beta1
ports:
- port: 8774
name: nova-api
- port: 8775
name: nova-metadata
volume:
volume_name:
type: hostPath
mount: /certs
path: /etc/certs
container:
memcached:
image: memcached
tag:2
ports:
- port: 8774
name: nova-api
- port: 8775
name: nova-metadata
variables:
- name: HTTP_TLS_CERTIFICATE:
value: /certs/domain.crt
- name: HTTP_TLS_KEY
value: /certs/domain.key
volumes:
- name: /etc/certs
type: hostPath
mount: /certs
path: /etc/certs
PetSet manifest¶
service:
memcached:
apiVersion: apps/v1alpha1
kind: PetSet
service_name: 'memcached'
container:
memcached:
...
Configmap¶
You are able to create configmaps using support layer between formulas.
It works simple, eg. in nova formula there’s file meta/config.yml which
defines config files used by that service and roles.
Kubernetes formula is able to generate these files using custom pillar and grains structure. This way you are able to run docker images built by any way while still re-using your configuration management.
Example pillar:
kubernetes:
control:
config_type: default|kubernetes # Output is yaml k8s or default single files
configmap:
nova-control:
grains:
# Alternate grains as OS running in container may differ from
# salt minion OS. Needed only if grains matters for config
# generation.
os_family: Debian
pillar:
# Generic pillar for nova controller
nova:
controller:
enabled: true
versionn: liberty
...
To tell which services supports config generation, you need to ensure pillar structure like this to determine support:
nova:
_support:
config:
enabled: true
initContainers¶
Example pillar:
kubernetes:
control:
service:
memcached:
init_containers:
- name: test-mysql
image: busybox
command:
- sleep
- 3600
volumes:
- name: config
mount: /test
- name: test-memcached
image: busybox
command:
- sleep
- 3600
volumes:
- name: config
mount: /test
Affinity¶
podAffinity¶
Example pillar:
kubernetes:
control:
service:
memcached:
affinity:
pod_affinity:
name: podAffinity
expression:
label_selector:
name: labelSelector
selectors:
- key: app
value: memcached
topology_key: kubernetes.io/hostname
podAntiAffinity¶
Example pillar:
kubernetes:
control:
service:
memcached:
affinity:
anti_affinity:
name: podAntiAffinity
expression:
label_selector:
name: labelSelector
selectors:
- key: app
value: opencontrail-control
topology_key: kubernetes.io/hostname
nodeAffinity¶
Example pillar:
kubernetes:
control:
service:
memcached:
affinity:
node_affinity:
name: nodeAffinity
expression:
match_expressions:
name: matchExpressions
selectors:
- key: key
operator: In
values:
- value1
- value2
Volumes¶
hostPath¶
service:
memcached:
container:
memcached:
volumes:
- name: volume1
mountPath: /volume
readOnly: True
...
volume:
volume1:
name: /etc/certs
type: hostPath
path: /etc/certs
emptyDir¶
service:
memcached:
container:
memcached:
volumes:
- name: volume1
mountPath: /volume
readOnly: True
...
volume:
volume1:
name: /etc/certs
type: emptyDir
configMap¶
service:
memcached:
container:
memcached:
volumes:
- name: volume1
mountPath: /volume
readOnly: True
...
volume:
volume1:
type: config_map
item:
configMap1:
key: config.conf
path: config.conf
configMap2:
key: policy.json
path: policy.json
To mount single configuration file instead of whole directory:
service:
memcached:
container:
memcached:
volumes:
- name: volume1
mountPath: /volume/config.conf
sub_path: config.conf
Generating Jobs¶
Example pillar:
kubernetes:
control:
job:
sleep:
job: sleep
restart_policy: Never
container:
sleep:
image: busybox
tag: latest
command:
- sleep
- "3600"
Volumes and Variables can be used as the same way as during Deployment generation.
Custom params:
kubernetes:
control:
job:
host_network: True
host_pid: True
container:
sleep:
privileged: True
node_selector:
key: node
value: one
image_pull_secretes: password
Role-based access control¶
To enable RBAC, you need to set following option on your apiserver:
kubernetes:
master:
auth:
mode: Node,RBAC
Then you can use kubernetes.control.role state to orchestrate role and
rolebindings. Following example shows how to create brand new role and binding
for service account:
control:
role:
etcd-operator:
kind: ClusterRole
rules:
- apiGroups:
- etcd.coreos.com
resources:
- clusters
verbs:
- "*"
- apiGroups:
- extensions
resources:
- thirdpartyresources
verbs:
- create
- apiGroups:
- storage.k8s.io
resources:
- storageclasses
verbs:
- create
- apiGroups:
- ""
resources:
- replicasets
verbs:
- "*"
binding:
etcd-operator:
kind: ClusterRoleBinding
namespace: test # <-- if no namespace, then it's clusterrolebinding
subject:
etcd-operator:
kind: ServiceAccount
Simplest possible use-case, add user test edit permissions on it’s test namespace:
kubernetes:
control:
role:
edit:
kind: ClusterRole
# No rules defined, so only binding will be created assuming role
# already exists
binding:
test:
namespace: test
subject:
test:
kind: User
More Information¶
/opencontrail-integration/docs /getting-started-guides/opencontrail.md * https://github.com/kubernetes/kubernetes/tree/master/cluster/saltbase
Documentation and Bugs¶
To learn how to install and update salt-formulas, consult the documentation available online at:
In the unfortunate event that bugs are discovered, they should be reported to the appropriate issue tracker. Use Github issue tracker for specific salt formula:
For feature requests, bug reports or blueprints affecting entire ecosystem, use Launchpad salt-formulas project:
You can also join salt-formulas-users team and subscribe to mailing list:
Developers wishing to work on the salt-formulas projects should always base their work on master branch and submit pull request against specific formula.
Any questions or feedback is always welcome so feel free to join our IRC channel:
#salt-formulas @ irc.freenode.net