Skip to main content

Before you upgrade

Nothing in RunOS upgrades on its own. A service, Kubernetes and a node each move only when you ask. This page lists what to do first, the safe order, and what RunOS refuses so you do not find out halfway through.

Verify any command with --help, for example runos clusters --help.

The short version​

  1. Back up first. Backups are opt-in. If you never turned them on, there is nothing to go back to.
  2. Test on a copy when the data matters. Restore the backup into a new instance and upgrade that one first.
  3. There is no way back. RunOS refuses most downgrades because the data on disk has already moved forward.
  4. Move one thing at a time. One service, one Kubernetes minor, one node.
  5. Read the warning lines. A service or cluster on a retired version says so on show and list.

Services​

A service keeps the version it was created with. Upgrades are an explicit update with a new --version, never automatic (see Managed services).

Back up first. A new database is not backed up until you turn backups on. For PostgreSQL the backup covers the whole instance, and a restore creates a new instance. See Backups and restore. Take a one-shot backup right before the upgrade and check that it finished.

RunOS refuses to go back. For these service types the data or schema moves forward when the new version starts, and an older version cannot read it: Harbor, Prometheus, Grafana, Langfuse, LiteLLM, Umami, PostgreSQL, MySQL, ClickHouse, Kafka, RabbitMQ, cert-manager, Traefik and NetBird. An update to an older version is refused with service.version_downgrade_refused. The only way back is to restore a backup into a new instance, so the backup you take before the upgrade is your rollback.

Other refusals. An update is also refused when the version is unknown (service.version_unsupported), when it is retired (service.version_retired), and for a type whose version cannot change after create (service.version_not_editable). The refusal comes before anything is written, so a refused update changes nothing.

Retired versions. RunOS retires a version when it has known security flaws, or when a newer patch of the same release line replaces it. A retired version:

  • is not offered when you create a new service, and a create or a move onto it is refused;
  • keeps working on a service that already runs it, with a version_retired warning on show and list that names the version to move to;
  • can always be left: an update to its replacement is accepted.

Treat the warning as a to-do. For PostgreSQL, MySQL and MinIO the warning also says what the next apply does, for example that any apply restarts an instance on the current patch.

Shared operators stay where they are. Several service types run under an operator that is installed once per cluster and shared by every service of that type: PostgreSQL, MySQL, ClickHouse, Kafka, RabbitMQ and MinIO. When you create a service and the operator is already on the cluster, RunOS leaves it as it is. Updating a service moves the service, not the operator, so a cluster can keep an operator from the day its first service of that type was created. The MinIO operator is never replaced by a service create or update, because replacing it upgrades the operator and its custom resource definitions for every tenant on the cluster. If you rely on a feature that needs a newer operator, check which operator release your cluster runs first.

Kubernetes​

The version moves one minor at a time and never back. To go from 1.33 to 1.35, upgrade to 1.34 first. RunOS refuses a jump of more than one minor, a move back, and a retired minor. runos clusters k8s-versions lists what is offered and what is retired.

Take an etcd snapshot first. etcd holds the whole state of the cluster, and a failed upgrade of a control plane is recovered from a snapshot. Create an etcd backup schedule once, and trigger it right before you start:

runos etcd-backups schedules
runos etcd-backups schedules trigger <scheduleId>
runos etcd-backups runs --cid <cid>

Check that the run finished before you go on.

Patch updates inside a minor. Look first, then roll:

runos clusters k8s-patch-status --cid <cid>
runos clusters k8s-patch-preflight --cid <cid>

The preflight is a dry run. It checks, among other things, that every node is Ready and connected, that no other upgrade or maintenance job is running, that etcd is healthy, and that draining each node will not take down a pod that has nowhere else to go. A failed check refuses the roll and says which. The roll then goes control planes first, then workers, one node at a time, then the network plugin (Cilium) to its pinned version. If the preflight reports that a drain would disrupt pods, the roll starts only when you explicitly accept the disruption.

One control plane means an API outage. A cluster with a single control-plane node has one API server. While that node upgrades, kubectl, the RunOS console and RunOS itself cannot reach the Kubernetes API of that cluster. Running pods keep running, but nothing can be scheduled, changed or healed until the control plane is back. A cluster with three control planes keeps the API up because the other two answer. See Clusters and nodes for how many control planes to run.

New nodes get the running patch. A node that joins installs the Kubernetes patch the API server runs, not the newest one.

Nodes and the operating system​

Updates inside the release. The node maintenance job node-apt-upgrade-reboot cordons and drains the node, installs the pending updates for the Ubuntu release it already runs, then reboots it and brings it back. It never does a distribution upgrade, never removes packages beyond cleaning up unused ones, and keeps held packages held. On a control-plane node it also renews the Kubernetes certificates and takes an etcd snapshot. Run it on one node at a time, and wait until the node is Ready and uncordoned before you start the next. On a control-plane node of a cluster with three, check that the other two are healthy first: two members is the least etcd needs to keep quorum.

No in-place Ubuntu release upgrade. RunOS does not move a node from one Ubuntu release to the next (for example 22.04 to 24.04), and you should not run do-release-upgrade on a node yourself, because RunOS and Kubernetes are installed for the release the node joined with. Plan a replacement instead: add a node installed with the new release, then remove the old one (see Remove a node). Before you do, check runos clusters k8s-versions: some Kubernetes versions run only on newer Ubuntu releases, and a new cluster needs Ubuntu 24.04 or 26.04 nodes.

Before you start​

  • A backup or snapshot exists, and you checked that it finished.
  • For a database, you tried the upgrade on a restored copy.
  • show or list has no unread warning on the thing you are about to change.
  • Only one upgrade or maintenance job is running on the cluster.
  • You know how you would go back, or you accepted that you cannot.