Kubeadm to Talos in place: nine production clusters, zero downtime

At Inherent, we have nine clusters on our OpenStack infrastructure, all originally deployed with Ansible and Kubeadm.

This is not a blog post introducing Talos. I think there are plenty of articles on the internet that do that better than I could. What pushed us towards Talos was the instability of Ansible and the deployment processes built around it. It was not really Ansible itself, but rather the way deployments had been implemented at Inherent.

What was meant to be a collection of simple playbooks grew heavier over the years: one configuration added here, a Jinja template there, then a specific argument somewhere else to make things work. Over time, some elements stopped being maintained, changes were made by hand, and so on.

Deploying or upgrading a cluster had become a heavy task, and it was rare for the playbook to run from start to finish without a problem.

Talos looked like a good way out. It let us manage the whole infrastructure as code, using the OpenStack and Talos providers. However, critical services were running on our clusters, so the change had to be transparent. Then we found this documentation.

A live, zero-downtime migration was possible.

The documentation is already enough on its own. I am writing this article to share the problems we encountered and how we solved them, in the hope that it will be useful to someone else.

Here are the field notes, in order, with the traps where they actually bit us.

The one idea the whole thing rests on

A Talos cluster is not defined by its nodes. It is defined by its secrets bundle — the cluster CA, the front-proxy CA, the service-account keypair, the etcd CA and the bootstrap token. If you generate that bundle from the PKI of your existing kubeadm cluster, then a Talos node booting with it is not forming a new cluster. It is joining the one you already have: same CA, so existing kubeconfigs and service-account tokens stay valid; same etcd CA, so a Talos control plane becomes a voting member of the etcd cluster already running.

talosctl gen secrets \
  --kubernetes-bootstrap-token <token> \
  --from-kubernetes-pki ./pki

That single flag is what turns a rebuild into a rolling replacement. Everything else in this note is making the cluster survive the moment those two worlds overlap — because for a few hours, you have kubeadm nodes and Talos nodes in the same cluster, and every add-on has to work on both.

That is the actual design constraint, and it is worth stating as a rule before the steps:

Every change you make in preparation must be a no-op on the kubeadm nodes and correct on the Talos nodes. Anything that is only correct for Talos has to be gated so it does not restart something that is currently serving traffic.

Three of the six steps below violate that rule if you apply them naively. Those are the ones with warnings.

Before you touch anything

Working access to the kubeadm nodes. You will be on those machines a lot, and it is the last time you ever will be — Talos has no SSH and no shell. Our nodes sit behind Boundary, so the SSH config needs the proxy in it:

Host <region>-<env>-*
    HashKnownHosts no
    User debian
    IdentityFile ~/.ssh/<key>
    ProxyCommand sh -c "boundary connect %n -exec nc -- {{boundary.ip}} {{boundary.port}}"

Take the Kubernetes and Helm providers out of the loop. In the OpenTofu code, comment out the kubernetes and helm providers and the whole kubernetes-resources.tf. During the migration the API endpoint is being served by a control plane that is halfway through changing identity, and you do not want a plan trying to reconcile in-cluster objects against it. Those resources get imported back afterwards. This is also why we ran the migration from a dedicated branch — one branch holding the Talos IaC, one holding the scripts and the preparation, and the second one deleted once the last cluster was done.

Pin the Talos version deliberately, because it pins etcd. We chose 1.10.6 for the etcd version its image ships (3.5.21), not for anything in Talos itself. Talos runs etcd as its own static pod, and a Talos control plane joining your existing etcd cluster means the etcd binary that image carries is one your quorum has to live with. Look up what your kubeadm control planes are running today, then choose the Talos release whose etcd is that version or a known-good step forward. Do not let the default decide it for you — a mixed-member etcd cluster is exactly where version-specific bugs stop being theoretical.

Read your kubeadm ClusterConfiguration as a spec, not as history. Everything in it has to be transcribed faithfully into the Talos machine config, and the things that hurt when you forget them are the ones nothing tests until the last kubeadm master is gone:

• controlPlaneEndpoint — identical, or every certificate and kubeconfig in the estate breaks.
• podSubnet and serviceSubnet — identical, non-negotiable.
• Every apiServer.extraArgs entry. Our clusters authenticate through Keycloak; the five OIDC flags below live in the kubeadm config, and if they do not make it into the Talos config, SSO keeps working right up until the last kubeadm apiserver is drained, then everyone loses access at once.
• Whether control planes are schedulable, and any taints you removed by hand years ago.

Step 1 — Add localhost to the kubeadm certSANs

Talos routes in-cluster API traffic through KubePrism, a local proxy listening on localhost:7445 on every node. So Talos components talk to the apiserver over loopback — and your existing apiserver certificate, generated by kubeadm, almost certainly does not have 127.0.0.1 or localhost in its SANs. First Talos node joins, TLS fails, and you are debugging a cluster you can no longer SSH into.

Do this on all three kubeadm control planes, before anything else. Start with the master that definitely still has the kubeadm config on disk — if the others do not have it, copy it from the first one rather than reconstructing it.

# Keep the current certificate. This is your rollback.
mv /etc/kubernetes/pki/apiserver.{crt,key} ~

# Add 127.0.0.1 and localhost to apiServer.certSANs.
# Edit it once on the first master, then copy that file to the other two.
vim /etc/kubernetes/unyc-manifests/config-kubeadm.yaml

# Regenerate the serving certificate from the updated config.
kubeadm init phase certs apiserver --config /etc/kubernetes/unyc-manifests/config-kubeadm.yaml

# Persist the change in the cluster's kubeadm-config ConfigMap.
kubeadm init phase upload-config kubeadm --config /etc/kubernetes/unyc-manifests/config-kubeadm.yaml

The config, with the SANs in place:

apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
kubernetesVersion: stable
clusterName: "<cluster>-k8s"
imageRepository: "registry.k8s.io"
controlPlaneEndpoint: "k8s.<env>.example.internal:6443"
networking:
  podSubnet: 192.168.0.0/19
  serviceSubnet: 192.168.32.0/19
apiServer:
  certSANs: ['k8s.<env>.example.internal', '127.0.0.1', 'localhost']
  extraArgs:
    oidc-issuer-url: "https://sso.example.com/realms/<realm>"
    oidc-client-id: "kubernetes-sso-<cluster>"
    oidc-username-claim: "preferred_username"
    oidc-username-prefix: "oidc:"
    oidc-groups-claim: client_roles
    oidc-groups-prefix: "oidc:"

The kubelet notices the new certificate file and restarts the apiserver static pod on its own. Verify it actually took before moving on — openssl s_client -connect 127.0.0.1:6443 </dev/null | openssl x509 -noout -text | grep -A1 'Subject Alternative Name' — and do the three masters one at a time, waiting for each apiserver to come back healthy. This step is additive and safe on a live cluster; it is also the step people skip because it looks cosmetic.

Step 2 — Extract the PKI and mint the secrets bundle

Two things have to come off a live control plane: the PKI directory, and a bootstrap token that will not expire mid-migration.

# On a control-plane node.
cp -r /etc/kubernetes/pki /home/debian/
chmod -R 777 /home/debian/pki/

# A non-expiring join token. Create it before you disconnect.
kubeadm token create --ttl 0

# From your workstation.
scp -r <region>-<env>-k8s-master-1:~/pki ./
talosctl gen secrets --kubernetes-bootstrap-token <token> --from-kubernetes-pki ./pki

Two words about the chmod 777. It is there because you are copying as an unprivileged user, and it works. It also means the entire trust root of a production cluster — ca.key, sa.key, etcd/ca.key — is world-readable on a shared machine for as long as you leave it there. sudo tar -C /etc/kubernetes -cf - pki | ... and a chown is the version to use if anyone else has an account on that host. Either way: delete the copy from the node and from your workstation the moment secrets.yaml exists, and treat that file with the same care. It is the cluster.

Check the extracted directory has ca.{crt,key}, front-proxy-ca.{crt,key}, sa.{key,pub} and etcd/ca.{crt,key} before generating. A bundle missing the etcd CA generates fine and fails much later, when a Talos control plane cannot join etcd.

Step 3 — Import the bundle into OpenTofu

talosctl gen secrets wrote a secrets.yaml. It must become state, not a file someone has locally:

tofu import talos_machine_secrets.this ~/path/to/secrets.yaml

Import rather than generate — if a tofu apply ever generates that resource itself, it produces a fresh CA and you have quietly built a second, empty cluster wearing your cluster’s name.

Step 4 — Update the OpenStack Cinder CSI driver

The version we were running mounted /etc/cacert from the host. We never needed it, and on Talos it cannot work: the root filesystem is read-only and that path does not exist. The pods stay ContainerCreating, and the first Talos worker you add cannot attach a single volume.

Apply the current driver from the platform repo, without the SOPS secrets, and recreate the workloads — the pod spec changes are not something a rolling update will take:

# Controller plugin.
kubectl delete deployment openstack-cinder-csi-controllerplugin
kubectl apply -f bases/csi-driver/csidriver.yaml

# One node plugin per availability zone.
kubectl delete daemonset <az1>-openstack-cinder-csi-nodeplugin
kubectl apply -f bases/csi-driver/nodeplugin-<az1>.yaml
# repeat per AZ

Do this while the cluster is still entirely kubeadm, verify a PVC binds and a pod mounts it, then move on. The generalisation is worth writing down, because Cinder was not the only offender we found: every hostPath your add-ons rely on is a migration blocker. Cloud credentials read from /etc/kubernetes/cloud.conf, CA bundles from /etc/ssl, anything dropped on a node by a Kubespray role — all of it has to become a Secret or a ConfigMap first. Grep your manifests for hostPath before you plan the dates.

Step 5 — Give the external health probes an identity

Talos runs the apiserver with anonymous authentication disabled. Our kubeadm clusters had it on, and something was quietly relying on it: the external traffic manager doing GET /healthz unauthenticated to decide whether a region is alive. On the first Talos control plane, those probes turn into 401, the region is marked down, and traffic moves away from a cluster that is perfectly healthy.

Bind the non-resource health URLs to a real identity, and point the probe at a token:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: <tenant>-read-kube-health
  labels:
    app: traffic-manager-health
    managed-by: terraform
rules:
  - nonResourceURLs: ["/readyz", "/readyz/*", "/livez", "/livez/*", "/healthz"]
    verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: <tenant>-read-kube-health-static-token
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: <tenant>-read-kube-health
subjects:
  - kind: ServiceAccount
    name: <health-probe-sa>
    namespace: kube-system

Then go looking for the others. Anything unauthenticated against the apiserver — an old blackbox exporter, a load-balancer probe, a status page — works today because of a kubelet and apiserver configuration Talos will not reproduce. Find them on kubeadm, while you still can.

Step 6 — Rewrite the Cilium values

Cilium needs four Talos-specific things: KubePrism as its apiserver endpoint, no cgroup automount because Talos owns /sys/fs/cgroup and the root filesystem is read-only, an explicit capability set because nothing is privileged by default, and Envoy left out if you are not running it.

ipam:
  mode: kubernetes
kubeProxyReplacement: true
k8sServiceHost: localhost
k8sServicePort: 7445
cgroup:
  autoMount:
    enabled: false
  hostRoot: /sys/fs/cgroup
envoy:
  create: false
securityContext:
  capabilities:
    ciliumAgent:
      - CHOWN
      - KILL
      - NET_ADMIN
      - NET_RAW
      - IPC_LOCK
      - SYS_ADMIN
      - SYS_RESOURCE
      - DAC_OVERRIDE
      - FOWNER
      - SETGID
      - SETUID
    cleanCiliumState:
      - NET_ADMIN
      - SYS_ADMIN
      - SYS_RESOURCE

This is the step that breaks the rule about no-ops, and it is the one to think hardest about. k8sServiceHost: localhost and port 7445 describe a proxy that exists on Talos nodes and does not exist on a kubeadm node. Push these values with a default rolling update and Helm restarts the agent DaemonSet across a cluster that is still mostly kubeadm; each restarted agent comes up looking for an apiserver on a loopback port where nothing is listening, and you lose networking on nodes that were fine a minute ago.

Set the agent DaemonSet’s update strategy so the release does not restart running pods — OnDelete, or a rollout paused before it touches anything — and let the new pod spec arrive on each node as part of that node being replaced. New Talos node, fresh agent pod, correct values. The kubeadm agents keep the spec they booted with until the node they are on ceases to exist. Verify that with kubectl rollout status refusing to move before you believe it, not after.

Step 7 — Roll the nodes

Only now does a single node change.

Control planes, strictly one at a time. Three etcd members means you can lose exactly one. Drain the kubeadm master, remove its etcd member cleanly, let OpenTofu provision the Talos node in its place, and then wait — talosctl -n <ip> etcd members and talosctl -n <ip> health — until the replacement is a voting member and the apiserver behind it answers. Only then touch the second one. The temptation to parallelise here is the single most expensive mistake available in this whole procedure; two members down in a three-member cluster is a read-only cluster and a restore from backup.

Workers after, in batches your PodDisruptionBudgets actually gate. If a PDB is missing or wrong, the drain will happily take the last replica of something. This is where the Talos nodes bring up their own Cilium agent with the step-6 values and mount Cinder volumes with the step-4 driver, so if either of those is wrong you find out on worker one, with everything else still running on kubeadm. That is the good failure mode, and it is why the order in this note is the order it is.

Keep the last kubeadm master until the end. It is the only place left where you can read the old configuration, compare a rendered manifest, or check what an argument used to be. Once it is gone, there is no shell anywhere in the cluster.

Afterwards

Uncomment the kubernetes and helm providers, import the in-cluster resources back into state, and confirm a plan comes back clean. Delete the --ttl 0 bootstrap token. Delete every copy of the PKI directory. Delete the preparation branch. Then use the thing you just built: replace a node for no reason at all and watch it come back in five minutes, because that is the capability you actually bought.

What I would do differently

Two things.

I would grep for hostPath and unauthenticated apiserver access on day one, before proposing any dates. Our two genuine surprises — the CSI driver and the health probes — were in that set, and both could have been found by searching rather than deploying. A morning of grep would have moved them from “incident during the window” to “line item in the plan”.

I would also check the etcd version jumps carefully. Having never had to deal with breaking changes in etcd, I naively did not check the incompatible changes. When moving from Kubeadm to Talos, I had not noticed that etcd had to jump from the v2 storage backend to v3. The version jump we made during our first attempts broke the etcd cluster.

Look carefully into the jumps you need to make.

And good luck!