Skip to content

Ansible Kubernetes Cluster

This page will demonstrate how to setup a Kubernetes Cluster using Ansible.

Note

I am using TechnoTim's k3s-ansible ansible role.

Ansible Project Setup

Fetch the repository:

pushd /tmp
git clone https://github.com/techno-tim/k3s-ansible && cd k3s-ansible
git checkout v1.25.9+k3s1 
cd ..

Create the directory structure for the cluster:

mkdir -p k3s-cluster/inventory/sektorlab

Copy content across:

cp -R k3s-ansible/inventory/sample k3s-cluster/inventory/sektorlab
cp -R k3s-ansible/collections k3s-cluster/collections
cp -R k3s-ansible/example k3s-cluster/example
cp -R k3s-ansible/roles k3s-cluster/roles
cp -R k3s-ansible/ansible.example.cfg k3s-cluster/ansible.cfg
cp -R k3s-ansible/*.yml k3s-cluster/

Configure ansible.cfg:

cat > k3s-cluster/ansible.cfg << EOF
[defaults]
inventory = inventory/sektorlab/hosts.ini
host_key_checking = False
timeout = 60
EOF

Configure inventory with the nodes that we provisioned in Proxmox:

cat > k3s-cluster/inventory/sektorlab/hosts.ini << EOF
[master]
10.8.0.101
10.8.0.102
10.8.0.103

[node]
10.8.0.104
10.8.0.105
10.8.0.106

[k3s_cluster:children]
master
node
EOF

In inventory/sektorlab/group_vars/all.yml we need to configure a couple of configuration options, which we enable metallb, we define the address of our api server, and the range of addresses that metallb will use, etc:

---
k3s_version: v1.25.12+k3s1
ansible_user: ruan
systemd_dir: /etc/systemd/system
system_timezone: "Africa/Johannesburg"

# interface which will be used for flannel
flannel_iface: "ens18"

# apiserver_endpoint is virtual ip-address which will be configured on each master
apiserver_endpoint: "10.8.0.100"

# k3s_token is required  masters can talk together securely
# this token should be alpha numeric only
k3s_token: "x0x0x0x0x" # make sure to change this

# The IP on which the node is reachable in the cluster.
k3s_node_ip: '{{ ansible_facts[flannel_iface]["ipv4"]["address"] }}'

# Disable the taint manually by setting: k3s_master_taint = false
k3s_master_taint: "{{ true if groups['node'] | default([]) | length >= 1 else false }}"

# these arguments are recommended for servers as well as agents:
extra_args: >-
  --flannel-iface={{ flannel_iface }}
  --node-ip={{ k3s_node_ip }}

# change these to your liking, the only required are: --disable servicelb, --tls-san {{ apiserver_endpoint }}
extra_server_args: >-
  {{ extra_args }}
  {{ '--node-taint node-role.kubernetes.io/master=true:NoSchedule' if k3s_master_taint else '' }}
  --tls-san {{ apiserver_endpoint }}
  --disable servicelb
  --disable traefik
  --write-kubeconfig-mode 644 
  --kube-apiserver-arg default-not-ready-toleration-seconds=30 
  --kube-apiserver-arg default-unreachable-toleration-seconds=30 
  --kube-controller-arg node-monitor-period=20s 
  --kube-controller-arg node-monitor-grace-period=20s 
  --kubelet-arg node-status-update-frequency=5s
extra_agent_args: >-
  {{ extra_args }}
  --kubelet-arg node-status-update-frequency=5s

# image tag for kube-vip
kube_vip_tag_version: "v0.5.12"

# metallb type frr or native
metal_lb_type: "native"

# metallb mode layer2 or bgp
metal_lb_mode: "layer2"

# image tag for metal lb
metal_lb_speaker_tag_version: "v0.13.9"
metal_lb_controller_tag_version: "v0.13.9"

# metallb ip range for load balancer
metal_lb_ip_range: "10.8.0.110-10.8.0.115"

in .gitlab-ci.yml set:

default:
  tags:
    - sektorlab

stages:
  - deploy

kubernetes:deploy:
  stage: deploy
  image: ruanbekker/ansible:2.12.5-kubernetes
  variables:
    ANSIBLE_CONFIG: $CI_PROJECT_DIR/ansible.cfg
    ROLES_REQUIREMENTS_FILE: $CI_PROJECT_DIR/collections/requirements.yml
    EXTERNAL_ROLE_DIR: $CI_PROJECT_DIR/roles
    ANSIBLE_COLLECTIONS_PATH: $CI_PROJECT_DIR/roles
  script:
    - ansible-galaxy install -r "$ROLES_REQUIREMENTS_FILE" -p "$EXTERNAL_ROLE_DIR"
    - ansible-playbook -e "ansible_ssh_user=ruan ansible_ssh_pass=$ANSIBLE_PASSWORD" site.yml
  only:
    - /^kubernetes-.*$/

And make sure ANSIBLE_PASSWORD is defined in the CICD variables.

Then tag the commit:

cd k3s-cluster
git add .
git commit -m "initial"
git tag -f kubernetes-v1.25.12
git push origin -f kubernetes-v1.25.12

Kubeconfig Setup

To access the kubeconfig, ssh to one of the master nodes and view the file:

cat /etc/rancher/k3s/k3s.yaml

You will need to replace server: https://127.0.0.1:6443 with the address of your apiserver endpoint, in my case server: https://10.8.0.100:6443 and the store it on your local workstation at ~/.kube/config and set it to:

export KUBECONFIG=~/.kube/config

Then view your nodes:

kubectl get nodes

NAME    STATUS   ROLES                       AGE    VERSION
node1   Ready    control-plane,etcd,master   2d     v1.25.12+k3s1
node2   Ready    control-plane,etcd,master   2d     v1.25.12+k3s1
node3   Ready    control-plane,etcd,master   2d     v1.25.12+k3s1
node4   Ready    <none>                      2d     v1.25.12+k3s1
node5   Ready    <none>                      2d     v1.25.12+k3s1
node6   Ready    <none>                      2d     v1.25.12+k3s1