Kubernetes Cluster Deployment with Ansible for Openstack

Building a production-grade Kubernetes cluster as the foundation for an OpenStack-Helm cloud involves far more than running kubeadm init. Multi-NIC node topologies, CNI interface selection around OVS/OVN bridges, bare-metal load balancing, distributed block storage, Gateway API TLS routing, and out-of-the-box observability all need to be wired together reproducibly.

This guide walks through my openstack-k8s-cluster ↗ Ansible automation, which provisions a full Kubernetes 1.34.9 cluster on Ubuntu nodes from bare OS to ready-to-use dashboards and OpenStack prerequisites.

Component Versions#

Architecture Overview#

Ingress Path#

• MetalLB (v0.16.1) assigns a virtual IP (gateway_ip, default 192.168.64.100) from the configured address pool (192.168.64.100-192.168.64.110) using L2 (ARP) or BGP mode.
• Envoy Gateway (v1.8.1) binds to that VIP through an EnvoyProxy resource (gateway-proxy-default in envoy-gateway-system) and exposes a shared Gateway (gateway-default) listening on :80 (HTTP) and :443 (HTTPS, TLS termination).
• HTTPRoute resources route traffic by hostname (*.<gateway_ip>.nip.io) to the backends:
  • grafana.<gw-ip>.nip.io → kube-prometheus-stack-grafana:80 (monitoring)
  • dashboard.<gw-ip>.nip.io → kubernetes-dashboard-auth:8000 (/auth), kubernetes-dashboard-api:8000 (/api), and kubernetes-dashboard-web:8000 (/) (kubernetes-dashboard)
  • ceph.<gw-ip>.nip.io → rook-ceph-mgr-dashboard:7000 (rook-ceph)

TLS#

• cert-manager (v1.13.0) bootstraps the openstack-ca PKI hierarchy (identical to clusters/prod/region1/openstack/pki/ in openstack-helm ↗): selfsigned-bootstrap (ClusterIssuer) → openstack-ca (Certificate in cert-manager, RSA 4096, 10y, secretName: openstack-ca-secret) → openstack-ca (ClusterIssuer) → gateway-openstack-ca-cert (Certificate in envoy-gateway-system, *.<public_domain>, secretName: gateway-openstack-ca-tls).
• The shared Gateway (gateway-default) serves gateway-openstack-ca-tls from the very first run so Grafana, Kubernetes Dashboard, and Rook-Ceph Dashboard are immediately reachable over HTTPS, and FluxCD later adopts the exact same PKI resources in-place without regenerating the CA or switching the Gateway certificate.
• Supports multi-region CA sharing via openstack_ca_bootstrap: false (verifies an imported openstack-ca-secret in cert-manager instead of generating a new root CA) and automatically cleans up legacy self-signed certificates (legacy_selfsigned_certs).

Monitoring#

• kube-prometheus-stack (87.2.1) deploys Prometheus (30d retention) and Grafana (admin / grafana_admin_password).
• Rook-Ceph ServiceMonitor resources are applied and labeled (release: kube-prometheus-stack) so Prometheus automatically scrapes Ceph metrics.
• Three Ceph Grafana dashboards (Cluster, OSD, Pools) are downloaded on the control plane directly from the official Rook repository (v1.19.7) and injected as labeled ConfigMaps (grafana_dashboard: "1"), which Grafana’s sidecar (searchNamespace: ALL) auto-discovers.

Storage#

• Rook-Ceph (1.19.7) deploys the operator and a CephCluster across worker nodes, exposing a general RBD StorageClass backed by ceph-blockpool (allowVolumeExpansion: true) for persistent volumes.

Platform#

• Calico (3.32.0) provides pod networking (10.244.0.0/16) using VXLAN encapsulation pinned to the management interface via can-reach=<control_plane_ip>.
• FluxCD (2.18.4) runs in flux-system for GitOps continuous delivery.
• Gateway API standard CRDs and Envoy Gateway CRDs are installed cluster-wide.

Network Configuration#

Every node attaches to the same set of L2 networks. Netplan renames the hypervisor-assigned NICs (enp0s1, enp0s2, …) by matching their MAC addresses so interface names are identical across all nodes (ens160, ens192, ens224, ens256).

Interfaces#

Why ens224 and ens256 have no IP: Both interfaces are raw L2 ports bridged into OpenStack virtual switches (br-ex and the Octavia bridge) by the companion OpenStack deployment. Calico is explicitly pinned to ens160 (IP_AUTODETECTION_METHOD=can-reach={{ control_plane_ip }}) and MetalLB L2 advertisements are restricted to ens160 so neither component accidentally binds to these unnumbered interfaces.

Node IP Assignment (inventory/dev)#

Netplan Configuration (/etc/netplan/50-cloud-init.yaml)#

Control Plane (openstack-master1 — 3 NICs)#

network:
  version: 2
  ethernets:
    ens160:
      match:
        macaddress: '5a:96:25:a0:20:fb' # replace with node's NIC 1 MAC
      set-name: 'ens160'
      addresses:
        - 192.168.64.150/24
      nameservers:
        addresses:
          - 8.8.8.8
          - 8.8.4.4
      routes:
        - to: default
          via: 192.168.64.1
    ens192:
      match:
        macaddress: '26:e4:40:b9:1b:0d' # replace with node's NIC 2 MAC
      set-name: 'ens192'
      addresses:
        - 10.0.0.1/24
    ens224:
      match:
        macaddress: 'da:cb:03:ae:57:d7' # replace with node's NIC 3 MAC
      set-name: 'ens224'
      dhcp4: false

Worker Nodes (openstack-worker1..3 — 4 NICs, showing openstack-worker1)#

network:
  version: 2
  ethernets:
    ens160:
      match:
        macaddress: 'fa:f0:a2:bf:73:a9' # replace with node's NIC 1 MAC
      set-name: 'ens160'
      addresses:
        - 192.168.64.151/24 # .152 on worker2, .153 on worker3
      nameservers:
        addresses:
          - 8.8.8.8
          - 8.8.4.4
      routes:
        - to: default
          via: 192.168.64.1
    ens192:
      match:
        macaddress: 'be:fc:0e:7b:28:18' # replace with node's NIC 2 MAC
      set-name: 'ens192'
      addresses:
        - 10.0.0.2/24 # .3 on worker2, .4 on worker3
    ens224:
      match:
        macaddress: 'ce:fd:c0:be:84:88' # replace with node's NIC 3 MAC
      set-name: 'ens224'
      dhcp4: false
    ens256:
      match:
        macaddress: 'ae:77:f5:16:65:6f' # replace with node's NIC 4 MAC
      set-name: 'ens256'
      dhcp4: false

Apply the configuration on each node with:

sudo netplan apply

Ansible Roles#

The repository is structured into five Ansible roles under roles/, each mapped to a dedicated playbook under playbooks/:

1. common (playbooks/common.yaml)#

Runs on all nodes (control planes and workers) to prepare the OS for Kubernetes:

• Disables swap at runtime (swapoff -a) and removes swap entries from /etc/fstab.
• Loads required kernel modules (overlay, br_netfilter) and persists them in /etc/modules-load.d/k8s.conf.
• Configures sysctl networking parameters in /etc/sysctl.d/99-kubernetes-cri.conf (net.bridge.bridge-nf-call-iptables = 1, net.ipv4.ip_forward = 1, net.bridge.bridge-nf-call-ip6tables = 1).
• Installs containerd (containerd.io > 1.6) from the official Docker Ubuntu APT repository, generates /etc/containerd/config.toml, and enables the SystemdCgroup = true driver required by kubelet.
• Adds the official Kubernetes v1.34 APT repository (pkgs.k8s.io), installs kubelet, kubeadm, and kubectl pinned to 1.34.9-1.1, and places kubelet, kubeadm, kubectl, and containerd.io on hold via apt-mark hold.

2. control_planes (playbooks/control_planes.yaml)#

Runs on the control_planes group to bootstrap the Kubernetes control plane:

• Initializes the cluster with kubeadm init using --apiserver-advertise-address={{ control_plane_ip }}, --pod-network-cidr={{ pod_network_cidr }} (10.244.0.0/16), --kubernetes-version={{ kube_version }}, and --cri-socket /var/run/containerd/containerd.sock (skipped idempotently if /etc/kubernetes/admin.conf already exists).
• Sets up /root/.kube/config for the root user.
• Installs the Calico (3.32.0) CNI plugin, pins IP_AUTODETECTION_METHOD=can-reach={{ control_plane_ip }} on daemonset/calico-node, patches the default IPPool to use VXLAN encapsulation, and waits for calico-node to roll out.
• Patches the CoreDNS ConfigMap (kube-system/coredns) to add endpoint_pod_names under pods insecure and restarts CoreDNS (required for OpenStack-Helm OVN RAFT clustering).

3. nodes (playbooks/nodes.yaml)#

Runs on the nodes group to join worker nodes to the cluster:

• Checks whether /etc/kubernetes/kubelet.conf already exists on each worker.
• Generates a fresh join token on control_planes[0] (kubeadm token create --print-join-command) and joins unjoined workers with --cri-socket /var/run/containerd/containerd.sock.

4. cluster_addons (playbooks/cluster_addons.yaml)#

Runs once against control_planes[0] to deploy the full platform stack in 11 ordered stages:

1. Python Kubernetes client: Installs python3-pip and python3-kubernetes via APT on the control plane node so kubernetes.core.k8s tasks execute natively on the remote host.
2. MetalLB (v0.16.1): Enables strictARP: true in the kube-proxy ConfigMap (when metallb_mode == 'l2'), deploys MetalLB (metallb-frr.yaml), waits for the controller pods, and creates the IPAddressPool (default-pool) plus an L2Advertisement (pinned to metallb_l2_interface: ens160) or BGPPeer/BGPAdvertisement (when metallb_mode == 'bgp').
3. Helm (3.21.2) & CNI binaries: Copies Calico CNI binaries from /opt/cni/bin/* to /usr/lib/cni and installs the Helm CLI binary (automatically detecting amd64 vs arm64).
4. cert-manager (v1.13.0): Installs cert-manager with CRDs via Helm and waits for the deployment to become ready.
5. Rook-Ceph (1.19.7): Deploys the rook-ceph operator and rook-ceph-cluster (with toolbox.enabled: true) via Helm in the rook-ceph namespace, then creates the general RBD StorageClass backed by ceph-blockpool.
6. FluxCD (2.18.4): Installs flux2 in the flux-system namespace via Helm.
7. Gateway API & Envoy Gateway (v1.8.1): Applies standard Gateway API CRDs, installs Envoy Gateway CRDs from oci://docker.io/envoyproxy/gateway-crds-helm via server-side apply, deploys Envoy Gateway (v1.8.1), and creates the default GatewayClass and gateway-proxy-default EnvoyProxy pinned to gateway_ip (192.168.64.100).
8. kube-prometheus-stack (87.2.1): Installs Prometheus and Grafana in monitoring (30d retention, Grafana dashboard sidecar scanning all namespaces), applies Rook-Ceph Prometheus RBAC and ServiceMonitor manifests, labels the rook-ceph-mgr ServiceMonitor with release: kube-prometheus-stack, and imports three Ceph Grafana dashboards from the official Rook repository (v1.19.7) as labeled ConfigMaps.
9. Kubernetes Dashboard (7.14.0): Installs Kubernetes Dashboard in kubernetes-dashboard with HTTP enabled on kong-proxy, and creates an admin-user ServiceAccount bound to cluster-admin.
10. PKI (openstack-ca) & Shared Gateway: Bootstraps ClusterIssuer/selfsigned-bootstrap → Certificate/openstack-ca (cert-manager/openstack-ca-secret, RSA 4096, 10y) → ClusterIssuer/openstack-ca → Certificate/gateway-openstack-ca-cert (envoy-gateway-system/gateway-openstack-ca-tls, *.<public_domain>), waits for each resource to report Ready, creates the shared gateway-default Gateway in envoy-gateway-system serving gateway-openstack-ca-tls, and removes any legacy self-signed certificates (legacy_selfsigned_certs).
11. HTTPRoutes & Access Summary: Creates HTTPS HTTPRoutes for Grafana (grafana.<gw-ip>.nip.io), Kubernetes Dashboard (dashboard.<gw-ip>.nip.io), and Rook-Ceph Dashboard (ceph.<gw-ip>.nip.io, after setting dashboard.ssl: false on the CephCluster so rook-ceph-mgr-dashboard serves HTTP on port 7000), then prints the complete cluster access summary with URLs, credentials, and tokens.

5. update-and-reboot (playbooks/update-and-reboot.yaml)#

• Runs apt update && apt dist-upgrade across all nodes and reboots any node where /var/run/reboot-required is present.

Prerequisites#

System Requirements#

Control Machine Setup#

Install Ansible and the kubernetes.core collection on your workstation or bastion host:

# macOS
brew install ansible

# Ubuntu / Debian
sudo apt update && sudo apt install -y ansible

# Install the required Ansible Kubernetes collection
ansible-galaxy collection install kubernetes.core

Note: You do not need to install the Python kubernetes library on your local machine—the cluster_addons role automatically installs python3-kubernetes via APT on the remote control plane node where all kubernetes.core.k8s tasks are delegated.

SSH Key Configuration#

Copy your SSH public key to the root user on every node and verify connectivity:

# Generate an SSH key pair if you do not have one
ssh-keygen -t ed25519

# Copy the key to the control plane and worker nodes
ssh-copy-id root@192.168.64.150
ssh-copy-id root@192.168.64.151
ssh-copy-id root@192.168.64.152
ssh-copy-id root@192.168.64.153

# Clone the repository and verify Ansible connectivity
git clone https://gitlab.com/openstack7461002/openstack-k8s-cluster.git
cd openstack-k8s-cluster
ansible -i inventory/dev all -m ping

Required Ports#

Ensure the following ports are open between nodes on the management network (ens160):

Installation#

Step 1: Configure the Inventory (inventory/dev)#

Update inventory/dev with your node hostnames and management IPs (ens160):

[control_planes]
openstack-master1 ansible_host=192.168.64.150

[nodes]
openstack-worker1 ansible_host=192.168.64.151
openstack-worker2 ansible_host=192.168.64.152
openstack-worker3 ansible_host=192.168.64.153

Step 2: Review Cluster Variables (inventory/group_vars/all.yaml)#

All versions, network CIDRs, MetalLB settings, PKI settings, and add-on chart versions are centralized in inventory/group_vars/all.yaml:

# --- Kubernetes ---
kube_version: '1.34.9'
k8s_apt_version: 'v1.34'
service_cidr: '10.96.0.0/12'
pod_network_cidr: '10.244.0.0/16'
control_plane_ip: "{{ hostvars[groups['control_planes'][0]]['ansible_default_ipv4'].address | default(groups['control_planes'][0]) }}"

# --- Component versions ---
calico_version: '3.32.0'
calico_encapsulation: 'VXLAN'
metallb_version: '0.16.1'
cert_manager_version: 'v1.13.0'
rook_ceph_version: '1.19.7'
fluxcd_version: '2.18.4'
envoy_gateway_version: '1.8.1'
prometheus_stack_version: '87.2.1'
k8s_dashboard_version: '7.14.0'
helm_version: '3.21.2'

# --- Gateway API & PKI (openstack-ca) ---
gateway_ip: '192.168.64.100'
public_domain: '{{ gateway_ip }}.nip.io'
openstack_ca_bootstrap: true

# --- MetalLB ---
metallb_mode: 'l2'
metallb_ip_range: '192.168.64.100-192.168.64.110'
metallb_l2_interface: 'ens160'

# --- Monitoring ---
grafana_admin_password: 'ChangeMe123!'

Step 3: Run the Playbooks#

You can run the entire deployment in one command using playbooks/k8s_all.yaml (note that k8s_all.yaml also runs update-and-reboot at the end, which upgrades APT packages and reboots nodes if required):

ansible-playbook -i inventory/dev playbooks/k8s_all.yaml

Or execute each stage step by step:

# 1. Prepare all nodes (swap, kernel modules, sysctl, containerd, kubeadm/kubelet/kubectl)
ansible-playbook -i inventory/dev playbooks/common.yaml

# 2. Initialize the control plane, Calico (VXLAN), and CoreDNS
ansible-playbook -i inventory/dev playbooks/control_planes.yaml

# 3. Join worker nodes to the cluster
ansible-playbook -i inventory/dev playbooks/nodes.yaml

# 4. Deploy MetalLB, cert-manager, Rook-Ceph, FluxCD, Envoy Gateway, PKI, Prometheus/Grafana, and K8s Dashboard
ansible-playbook -i inventory/dev playbooks/cluster_addons.yaml

Step 4: Fetch kubeconfig and Verify the Cluster#

Copy /root/.kube/config from the control plane to your local workstation:

mkdir -p ~/.kube
scp root@192.168.64.150:~/.kube/config ~/.kube/config
kubectl get nodes -o wide

Expected output:

NAME                STATUS   ROLES           AGE   VERSION   INTERNAL-IP      OS-IMAGE             KERNEL-VERSION       CONTAINER-RUNTIME
openstack-master1   Ready    control-plane   12m   v1.34.9   192.168.64.150   Ubuntu 24.04.2 LTS   6.8.0-57-generic     containerd://1.7.27
openstack-worker1   Ready    <none>          10m   v1.34.9   192.168.64.151   Ubuntu 24.04.2 LTS   6.8.0-57-generic     containerd://1.7.27
openstack-worker2   Ready    <none>          10m   v1.34.9   192.168.64.152   Ubuntu 24.04.2 LTS   6.8.0-57-generic     containerd://1.7.27
openstack-worker3   Ready    <none>          10m   v1.34.9   192.168.64.153   Ubuntu 24.04.2 LTS   6.8.0-57-generic     containerd://1.7.27

Step 5: Verify the Rook-Ceph Cluster#

Verify that Rook-Ceph has discovered the 100 GB raw disk on each of the three worker nodes (3 osds: 3 up, 3 in, 300 GiB total capacity) and that the cluster status is HEALTH_OK:

kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph -s

Expected output:

  cluster:
    id:     85cbf791-3802-4b59-9abc-9c1dda79759b
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum a,b,c (age 19s)
    mgr: b(active, since 5m), standbys: a
    mds: 1/1 daemons up, 1 hot standby
    osd: 3 osds: 3 up (since 11s), 3 in (since 9m)
    rgw: 1 daemon active (1 hosts, 1 zones)

  data:
    volumes: 1/1 healthy
    pools:   12 pools, 241 pgs
    objects: 297 objects, 619 KiB
    usage:   148 MiB used, 300 GiB / 300 GiB avail
    pgs:     240 active+clean
             1   active+clean+scrubbing

  io:
    recovery: 6.3 KiB/s, 1 keys/s, 38 objects/s

Deep Dive: Networking, CoreDNS, and Gateway API Logic#

Several subtle networking problems surface when preparing a multi-NIC Kubernetes cluster for OpenStack-Helm. Below is how the playbooks solve each one automatically.

1. Pinning Calico to ens160 and Enforcing VXLAN#

By default, Calico’s first-found IP autodetection scans node interfaces and can pick up internal networks (ens192) or OpenStack OVS/OVN bridges (ens224, ens256, br-ex, br-int), breaking cross-node pod routing. Furthermore, Calico’s default manifest enables IP-in-IP (IPIP) encapsulation (IP protocol 4), which is silently dropped by several hypervisor virtual networks (including Apple vmnet on macOS).

In roles/control_planes/tasks/main.yaml, the playbook:

1. Pins daemonset/calico-node to the interface that routes to control_plane_ip (ens160):

   kubectl set env daemonset/calico-node -n kube-system \
     IP_AUTODETECTION_METHOD=can-reach={{ control_plane_ip }}

   bash
2. Directly patches the default-ipv4-ippool CRD (ippools.crd.projectcalico.org) to enforce VXLAN (UDP 4789) instead of relying on CALICO_IPV4POOL_* environment variables, which calico-node ignores once the initial pool already exists:

   kubectl patch ippools.crd.projectcalico.org default-ipv4-ippool --type=merge -p \
     '{"spec":{"ipipMode":"Never","vxlanMode":"Always"}}'

   bash

2. CoreDNS endpoint_pod_names for OpenStack OVN RAFT#

OpenStack-Helm’s ovn chart deploys Northbound and Southbound ovsdb-server StatefulSets that form RAFT clusters using headless Service DNS records (<pod-name>.<service>.openstack.svc.cluster.local). By default, CoreDNS resolves endpoint records using IP dashes (192-168-x-x) rather than pod names, causing ovsdb-server RAFT initialization to fail.

The control_planes role patches the kube-system/coredns ConfigMap to insert endpoint_pod_names inside the kubernetes cluster.local block right after pods insecure and triggers a rolling restart of deployment/coredns.

3. MetalLB strictARP and Interface Pinning#

When running MetalLB in Layer 2 mode (metallb_mode: "l2"), kube-proxy in IPVS mode requires strictARP: true so that only the elected MetalLB speaker node answers ARP requests for LoadBalancer VIPs. The cluster_addons role patches configmap/kube-proxy in kube-system to set strictARP: true and pins the L2Advertisement to ens160 (interfaces: [ens160]) so ARP announcements never leak onto ens224 (br-ex).

4. Gateway API Routing & Unified openstack-ca PKI#

Instead of using the legacy NGINX Ingress Controller, the cluster uses Envoy Gateway (v1.8.1) implementing the Kubernetes Gateway API. Three key details are handled in roles/cluster_addons/tasks/main.yaml:

• Unified openstack-ca PKI hierarchy: cluster_addons bootstraps the exact same CA hierarchy as clusters/prod/region1/openstack/pki/ in openstack-helm:
  • ClusterIssuer/selfsigned-bootstrap (self-signed, used only to sign the root CA when openstack_ca_bootstrap: true)
  • Certificate/openstack-ca in cert-manager (isCA: true, RSA 4096, 10 years, secret openstack-ca-secret)
  • ClusterIssuer/openstack-ca (CA issuer backed by openstack-ca-secret)
  • Certificate/gateway-openstack-ca-cert in envoy-gateway-system (*.{{ public_domain }} and {{ public_domain }}, 1 year, secret gateway-openstack-ca-tls) Every task waits for Ready=True before creating Gateway/gateway-default, so HTTPS works immediately after the playbook finishes. When FluxCD later reconciles the openstack-pki Kustomization, it adopts these resources as-is without regenerating the CA or re-issuing the Gateway certificate. For secondary regions (region2) that share an imported CA (imported-ca-secret.yaml), setting openstack_ca_bootstrap: false skips creating a new root CA and verifies that cert-manager/openstack-ca-secret is already present. Any legacy self-signed resources (legacy_selfsigned_certs in roles/cluster_addons/vars/main.yaml) are also automatically removed.
• Kubernetes Dashboard v7 split architecture: Kubernetes Dashboard 7.14.0 separates authentication (kubernetes-dashboard-auth:8000), API (kubernetes-dashboard-api:8000), and the Angular SPA frontend (kubernetes-dashboard-web:8000). While kong-proxy is enabled on HTTP port 80, the HTTPRoute explicitly matches /auth, /api, and / so login requests (POST /api/v1/login) and static assets route cleanly without nested reverse-proxy redirects.
• Rook-Ceph Dashboard HTTP backend: By default, rook-ceph-mgr-dashboard uses self-signed HTTPS on port 8443, which causes Envoy Gateway to return 503 Service Unavailable unless custom BackendTLSPolicies are configured. The playbook patches the CephCluster spec with dashboard.ssl: false, waits for rook-ceph-mgr-dashboard to re-bind on HTTP port 7000, and attaches the HTTPRoute to port 7000 behind Envoy Gateway’s HTTPS listener.

Deep Dive: Monitoring Stack & Ceph Dashboards#

The cluster_addons role deploys kube-prometheus-stack (87.2.1) in the monitoring namespace with a 30-day Prometheus retention policy and Grafana’s dashboard sidecar (kiwigrid/k8s-sidecar) configured with searchNamespace: ALL.

Why Standard Grafana Dashboard Provisioning Failed#

Getting official Ceph dashboards into Grafana reliably inside an Ansible playbook required working around three common pitfalls:

1. Helm gnetId provider: Downloads run inside the pod at Helm install time into an emptyDir volume and are lost on every Grafana pod restart.
2. Ansible lookup('url', ...): Runs on the Ansible controller rather than the target control plane node and embeds large JSON payloads inline inside Ansible variables.
3. kubectl port-forward + curl: Background port-forwards inside non-interactive Ansible tasks suffer from race conditions and drop connections before curl finishes.

The Idempotent ConfigMap Sidecar Solution#

The playbook downloads each dashboard JSON directly on the control plane node from the official Rook GitHub repository (https://raw.githubusercontent.com/rook/rook/v{{ rook_ceph_version }}/deploy/examples/monitoring/grafana/) into a temporary file (mktemp), and pipes kubectl create configmap --dry-run=client -o yaml | kubectl apply -f - before labeling it with grafana_dashboard=1:

Additionally, Rook-Ceph’s ServiceMonitor (rook-ceph-mgr in rook-ceph) is patched with the label release: kube-prometheus-stack so the Prometheus Operator includes Ceph metrics automatically.

Accessing Dashboards#

At the end of playbooks/cluster_addons.yaml, Ansible prints a complete summary with URLs and credentials. With the default gateway_ip (192.168.64.100), the services are available over HTTPS (signed by openstack-ca) at:

To retrieve the credentials or export the openstack-ca certificate for browser/CLI trust at any time:

# Export the openstack-ca root certificate for browser/CLI trust
kubectl get secret openstack-ca-secret -n cert-manager \
  -o jsonpath='{.data.ca\.crt}' | base64 --decode > /tmp/openstack-ca.crt

# Kubernetes Dashboard token
kubectl -n kubernetes-dashboard create token admin-user

# Rook-Ceph Dashboard password
kubectl -n rook-ceph get secret rook-ceph-dashboard-password \
  -o jsonpath='{.data.password}' | base64 --decode && echo

Maintenance and Troubleshooting#

Inspecting Rook-Ceph Health with the Toolbox#

The rook-ceph-cluster Helm release enables toolbox.enabled: true. You can run ceph CLI commands directly inside the toolbox pod:

kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph status
kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd status

Checking Gateway API & Envoy Gateway Status#

Verify that the Gateway has been assigned the MetalLB VIP (192.168.64.100) and that all HTTPRoutes are accepted:

kubectl get gateway -n envoy-gateway-system
kubectl get httproute -A

Upgrading System Packages and Rebooting Nodes#

To apply Ubuntu security updates and automatically reboot nodes that require a kernel restart:

ansible-playbook -i inventory/dev playbooks/update-and-reboot.yaml