Skip to main content

Clusters and nodes

This is the operator view: how to target a cluster, shape it, add and remove nodes, and what runs underneath. Verify any command with --help:

runos clusters --help
runos nodes --help
runos integrations --help

Targeting a cluster (cid)​

A cluster is identified by a cid: 3 to 16 lowercase alphanumeric characters. Live examples: ky3, xym. Every cluster belongs to an account aid (example: rjwrn).

runos clusters add allocates the cid. The cluster becomes a real Kubernetes cluster when the first node joins and bootstraps the control plane, then runos clusters configure installs base infrastructure.

List your clusters:

runos clusters list
# CID  NAME     ISCONFIGURED
# ky3  tc3-ky3  true
# xym  tc2-xym  false

isConfigured is true once the base stack is installed. Most commands take --cid <cid>. Set a default so you can drop the flag:

runos config set cid ky3

Cluster shape (roles, node types, HA)​

A node carries one or both roles:

  • control-plane: runs the Kubernetes API server and etcd.
  • worker: runs your workloads.

A node can be both: a single-node cluster has one node that is isCp: true and isWorker: true.

Nodes come in two types:

  • manual: your own server, joined with a generated command.
  • cloud-provisioned: created by RunOS through a compute integration. The CLI provisions on hetzner, digitalocean, and scaleway. (hyperstack and lambdacloud are compute-capable integrations with no provisioning command, so bring those as manual nodes.)

One node is not high availability. HA needs an odd number of control-plane nodes so etcd keeps quorum; 3 is the standard count.

Inspect the shape:

runos nodes list --cid ky3
runos nodes show <nid> --cid ky3

Add a node​

Cloud-provision a node (the provider needs an integration first, see runos integrations list):

runos integrations hetzner add-server --cid ky3 \
  --integration-id <id> --request-as-worker

Or join a server you already own:

runos nodes join-command ssh-remote --cid ky3 --request-as-worker

The join token is single-use and short-lived (the response carries an expiresAt). Run the command once, on the target host.

Each node must clear preflight before it installs. The floor: at least 2 CPU, 3.5 GB RAM, cgroup v2 (with cpu, memory, pids controllers), and Ubuntu 22.04, 24.04, or 26.04 (interim releases like 25.04 are rejected). Disk: at least 15 GiB free on the volume backing /var/lib/containerd (RunOS recommends 20 GB+ free on /var; containerd images and etcd fill it fast). The node must also reach the Kubernetes registries pkgs.k8s.io and registry.k8s.io on egress.

A node's status is ready while its agent heartbeats. Miss heartbeats for 30 seconds and it flips to no_heartbeat.

Remove a node​

Removal is driven by the control plane: it drains the node (cordon, then evict its pods) and, for a control-plane node, removes its etcd member.

runos nodes delete <nid> --cid ky3

Two things to know:

  • The cloud server is not deleted by default. You keep getting billed until you pass --delete-cloud-instance, or delete it at the provider.
  • You cannot remove the last control-plane node normally. The delete is refused unless you also reset the cluster (--reset-cluster), which is irreversible.

runos nodes delete-preflight warns if a removal would leave etcd on an even voting-member count. It is advisory and never blocks the delete.

Kubernetes version​

The Kubernetes version is set per cluster and is currently 1.35. Upgrades run node by node: each node is cordoned, drained, upgraded, and uncordoned, so the cluster stays up.

Networking and ingress​

The pod network is Cilium. Ingress is Traefik, run as a host-network DaemonSet on ports 80 and 443.

There is no cloud load balancer. The Traefik Service is type: ClusterIP, not type=LoadBalancer. Traffic reaches Traefik directly on the node's ports.

Control-plane high availability​

The Kubernetes API is fronted by a control-plane VIP among the healthy control-plane nodes. Each node runs an on-node load balancer in front of the Kubernetes API, so the API stays reachable if a control-plane node goes down.

Connectivity (internal overlay vs user VPN)​

Two separate network planes:

  1. Internal overlay: a private WireGuard mesh carries node-to-node cluster traffic. It is not user-reachable.
  2. User-access VPN: an opt-in tunnel (a separate wg1 interface) for operators to reach cluster-internal services. You pick one of WireGuard or NetBird; they are mutually exclusive.

The internal overlay always exists. The user VPN is something you turn on.

Turning the user VPN on​

The VPN server is a wireguard service running on one node of the cluster. Install it once:

runos services wireguard add --cid <cid> --target-node-id <nid> --name <name> --follow

Pick the node with care, because your devices will dial its ingress address (the INGRESSIP column of runos nodes list --cid <cid>), on port 32768 by default:

  • For a cluster reached only from your own network (a homelab), give the target node a private ingress address first: runos nodes ingress <nid> --cid <cid> --ingress-ip <lan-ip> --ingress-not-published. Most routers will not hairpin a public address back inside, so a device on the same LAN cannot reach the server through a public ingress address.
  • Any node can host the server, a control plane included. Withholding the last DNS-published control plane is allowed only together with a dialable private --ingress-ip; without one the call is refused, because the cluster would then keep no address anyone could dial.

A node that hosts the VPN server refuses deletion; move the server off it first.

Connecting from your machine​

Each machine is a device with its own key, generated locally and never uploaded. A sign-in lasts 24 hours.

sudo runos vpn install     # once per machine, installs the background service
runos vpn up               # signs you in (browser) and connects to your default cluster
runos vpn connect <cid>    # reach another cluster too
runos vpn status           # the tunnel, the session, and each cluster
runos vpn down             # disconnect and end the session

Only the connected clusters' names resolve over the tunnel (split DNS); everything else resolves as before. When the session lapses, runos vpn up signs you in again.

On macOS, runos desktop install adds RunOS Desktop, a menu-bar app over the same service: the toggle runs vpn up/vpn down and each cluster row connects or disconnects that cluster. At login it reconnects silently and never opens a browser; when a fresh sign-in is needed it stays disconnected until you sign in from the toggle or the CLI.

Moving the VPN server to another node​

A cluster runs one user-VPN server. Move it with:

runos services wireguard move <id> --cid <cid> --target-node-id <nid> --follow

Your devices reconfigure themselves. Nothing is reissued and nobody runs anything on a laptop: each device reads the server's node, key and address on its next poll and re-handshakes. In practice the switch completes in about ten seconds.

Check the address the new node will hand out first. Clients are given that node's declared ingress address, falling back to its detected public address if none is declared:

runos nodes list --cid <cid>    # compare the INGRESSIP column on both nodes

If your devices connect from your own network, the target node needs a private ingress address, set with runos nodes ingress <nid> --ingress-ip <private-ip> --ingress-not-published. Move to a node advertising a public address and machines on your own network will usually lose the tunnel, because most routers will not route a public address back inside. The move tells you which address devices will be given and warns when it changes between private and public.

One rule worth knowing before you plan around it: withholding the last DNS-published control plane is refused unless the node keeps a dialable private ingress address. RunOS does not need the public record to manage the cluster (management runs over the agent's outbound tunnel); the k8s.<cid> record serves only the customer's external, non-VPN kubectl, and *.<cid> their public app ingress. An intranet-only cluster with a private ingress is therefore fine; a cluster with no address anyone can dial is not, and that is the case the refusal blocks. Withholding is reversible: re-publishing the node rebuilds the record.

Afterwards, confirm the tunnel is really carrying traffic rather than merely configured:

runos vpn status        # peerUp reflects the live WireGuard handshake

Opening cluster pages in a browser over the VPN​

On the VPN, a cluster's own hostnames resolve to private addresses. The CLI, curl and kubectl reach them with no extra step. A browser needs one permission first.

Chrome (and other Chromium browsers) ask before letting a page served from a public address reach a private one. The RunOS console is public; your cluster is private on the VPN. The first time the console opens a terminal, or you visit any cluster page, Chrome shows a prompt asking whether the site may find devices on your local network. Choose Allow.

Until you do, cluster pages hang and then time out, and the console's terminal sits on "Connecting..." before reporting that a firewall is probably blocking port 443. Nothing is actually wrong with the cluster: it is the browser waiting on that prompt. Granting it once fixes the terminal, the Traefik dashboard, Harbor, the MinIO console and any app you reach on a cluster address.

If you dismissed the prompt, reopen it from the padlock in Chrome's address bar, under the site's permissions, then reload the page.

This applies only to clusters you reach over the VPN. A cluster published on a public address needs nothing, and the runos CLI is never affected.

Cluster domains and certificates​

Every cluster gets a built-in domain in the form cid.aid.<root>. Example: ky3.rjwrn.example.com.

Its wildcard TLS certificate is issued by cert-manager using a DNS-01 challenge. List and add domains:

runos cluster-domains list-by-cluster --cid ky3
runos cluster-domains add ...   # custom wildcard via Cloudflare DNS-01

A custom domain provisions its own ClusterIssuer, wildcard cert, and wildcard DNS record.

Storage and durability​

Storage is node-local by default, backed by OpenEBS (the openebs-local StorageClass is the default). Node-local means data lives on one node, so a single-node cluster has no cross-node redundancy. Add nodes and opt into LINSTOR for volumes that survive a node loss.

For durability across nodes, opt into LINSTOR distributed block storage (DRBD-backed, install-gated). On cloud nodes you can carve a raw data partition for it at provision time with --provider-config.repartition.