Operations

This page covers running RunOS day to day: tracking async work, reading logs and metrics, debugging a bad deploy, alerting, healing certificates, operating managed services, and managing accounts and credentials.

Jobs (the async model)

Every slow operation (create a service, deploy an app, add a node) returns a jobId and runs in the background. You track it; you do not wait on the request.

runos jobs list --cid ky3       # recent jobs
runos jobs show <jobId>         # status and details
runos follow <jobId>            # block until it finishes

A job moves through pending, then running, then one terminal state: completed, failed, cancelled, or timed_out.

timed_out is record-only. RunOS stops tracking a job that has been running with no progress past a timeout. It does not kill the underlying Kubernetes or Helm work, which keeps reconciling. So timed_out is not the same as failed: do not conflate them. Re-check the cluster and re-run the operation idempotently rather than assuming nothing happened.

runos jobs cancel <jobId> is advisory. The current work item finishes and later items are skipped, but side effects already applied to the cluster stay. The response sets partialState: true when a job was cancelled mid-run, so inspect runos jobs show and decide whether to roll back.

Per-step detail lives under runos jobs workitems <jobId> and runos jobs workitem-logs.

Logs

Live container logs are always available. They are read straight from the kubelet, for every service type and app, with no extra install.

runos apps logs <id> --tail 200 --since 600 --follow
runos services postgresql logs <id> --previous

Flags: --tail (lines from the end, default 100), --since (last N seconds), --previous (the previous container instance, useful after a crash), --follow (stream).

Historical log search is opt-in. It needs a system Vector DaemonSet (one per cluster); live logs work without it. Vector auto-provisions a ClickHouse (or reuses one you pass). Retention defaults to 7 days, set at create (1-90), so logs age out past the window. On a fresh cluster only live logs work until Vector is added:

runos services vector add

Once installed, runos services vector search-logs filters by namespace, pod, container, node, and substring (--search), with an optional time window (--since, or --start/--end).

Metrics

metrics-server is always on. It is installed when the cluster is configured and gives point-in-time CPU and memory, the same data as kubectl top.

Historical metrics are opt-in, one command away with the system Prometheus. Time-series resource metrics, PromQL query, and metric-name discovery (labels) all come from it. One system Prometheus per cluster:

runos services prometheus add
runos services postgresql resource-metrics <id>   # CPU/mem over time

Debugging a failed deploy

Work in order. Stop when you have the answer.

1. Status first. runos apps status <id> and runos jobs show <jobId>.
2. Runtime logs. runos apps logs <id> for a crash after start (missing env var, exception, OOMKilled).
3. Build logs. runos apps builds <id> for a CLI deploy, or runos apps github-builds / runos apps gitlab-builds for a VCS deploy that failed to build or push.
4. Historical logs. runos services vector search-logs (needs Vector) for a cross-service or time-bounded pattern.
5. Metrics. resource-metrics for memory pressure or OOM.

Note: cluster-reachability errors (agent not connected, in-cluster DNS, connection refused) surface as 503 "retry" and are not real failures. Retry them.

Alerting

Alertmanager is opt-in on the system Prometheus, one command away. Enable it, then choose a rule sensitivity profile:

runos services prometheus install-alertmanager
runos services prometheus set-alert-profile   # essential | standard | aggressive

The three profiles, essential, standard, and aggressive, set the baseline thresholds; you can pin individual rules on top. Configure receivers (webhook, email, slack), routing, and silences with set-receiver, set-routing, and create-silence. List what is firing with runos services prometheus alertmanager-alerts.

Certificates and ACME

TLS terminates at Traefik on port 443 (websecure), running as a host-network DaemonSet on each node. cert-manager issues the certificates (one instance per cluster). The cluster wildcard certificate uses a DNS-01 challenge.

When the cluster-domain issuer hits the rare cert-manager No Key ID in JWS header account mixup (ACME Orders fail to sign), heal it:

runos clusters acme-heal

It restarts cert-manager and deletes the failed Order so a fresh one is signed. It is idempotent and no-ops when the cert is already Ready, so it is safe to run anytime, including at renewal. Add --scope cluster-domain --cluster-domain-id <id> for a custom-domain wildcard.

Managed-service operations

The storage backend is immutable after create. It is chosen when you provision the service; changing it means recreating the service. openebs-local is the default (node-local, no replication). LINSTOR is the opt-in distributed backend (DRBD-replicated, survives node loss, online resize).

Every managed service exposes a common operator surface, logs, status, show, update, and resource-metrics, plus type-specific verbs (for example PostgreSQL adds replication-status, promote, and relocate). PostgreSQL, MySQL, Valkey, MinIO, Harbor, Kafka, RabbitMQ, ClickHouse, and the AI services are all fully managed.

Backups and restore

Backups are opt-in, not automatic. No backend backs itself up on a default schedule. Nothing is backed up until you name a destination and turn a job on. Valkey is always ephemeral (emptyDir, no PVC), so it holds no durable state to back up.

There is no runos command for backups. The CLI manifest carries no backup or restore verb for any service type, so runos services postgresql ... cannot do this. Use the console, or the API:

• POST /:aid/:cid/backups/destinations records an S3-style destination.
• POST /:aid/:cid/services/postgresql/:id/configure-backup turns on WAL archiving and a scheduled backup with a retention window.
• POST /:aid/:cid/services/postgresql/:id/trigger-backup takes a one-shot backup.
• GET /:aid/:cid/backups/candidates lists what is enabled and what is not.

The unit is the whole instance, not one database. RunOS lists backup candidates per service, and PostgreSQL backup is CloudNativePG continuous archiving of the whole cluster. Every database on that instance is in the same backup, and restoring means creating a new PostgreSQL instance from the backup, which brings back every database it held. There is no per-database restore. Plan for that before you put several tenants on one instance.

Umami analytics

Umami is a self-hosted web analytics service. It is MIT-licensed, and "Umami" is the upstream project's trademark; RunOS packages it and does not own it. Read this whole section before you point production traffic at it. Several of these limits are real and none of them are worked around by the platform.

Retention is manual

Umami v3.3.1 deletes nothing on its own. There is no automatic event purge and no scheduled job. Umami's "retention" report tells you how many users came back; it does not remove old analytics data. Left alone, the database grows forever.

Purge on a schedule you run yourself. Find the real table and column names first, because they change between Umami majors and you should not run SQL you copied against a schema you have not looked at:

runos services postgresql schema <postgres-id> --cid ky3

Then delete in bounded batches against the Umami database, not the whole table in one statement:

runos services postgresql exec-sql <postgres-id> --cid ky3 \
  --database <umami-db> --read-write \
  --query "DELETE FROM <event-table> WHERE ctid IN (SELECT ctid FROM <event-table> WHERE <timestamp-column> < now() - interval '90 days' ORDER BY <timestamp-column> LIMIT 50000)"

The ctid subquery is not decoration. PostgreSQL does not accept LIMIT on a DELETE, so the plain DELETE ... LIMIT 50000 you might reach for fails with ERROR: syntax error at or near "LIMIT". Both forms were run against PostgreSQL 17.6, the version RunOS ships: the plain one errors, the ctid one reports DELETE 50. This is the standard way to bound the batch.

Keep it to one statement per call. exec-sql sends each statement as a separate cluster-agent call on a fresh connection, so a multi-statement query does not run in one transaction.

Repeat until it deletes zero rows, then run your maintenance (VACUUM, and ANALYZE on the tables you touched). Two reasons for the batches: a large single DELETE holds a long transaction and bloats the table, and exec-sql is synchronous, so the API gateway cuts the HTTP call at roughly 300 seconds while the query keeps running inside PostgreSQL. Cancel a runaway with pg_cancel_backend, not by re-running the command.

Do not reach for Umami's own website reset or website delete as a retention tool. Both remove the website's analytics rows wholesale, which is a different operation from ageing data out.

Sizing and alerting

A measured run on a development cluster recorded 59,405 events and grew the database from 9,727,123 bytes to 58,248,339 bytes. That is about 817 bytes per event. Plan with 1.3 KiB per event, which carries roughly 63 percent headroom over the measurement. The 30 percent this used to claim came from reading 1.3 KiB as 1.3 times 817 bytes; it is 1,331 bytes, so the margin is larger than it looked.

That number came from a synthetic load with small payloads. Custom event properties, longer URLs, and index growth all push it up, so re-measure against your own traffic before you size a year ahead.

Alert on database growth, not on event count. Event count tells you nothing about the bytes on disk once properties and indexes are in play.

RunOS ships no per-database and no per-volume growth alert. The shipped catalog watches disk at two coarser levels: node.disk_space_low on node filesystems, and linstor.pool_space_low on a LINSTOR pool. Both fire long after one database has quietly eaten a volume, and neither names the Umami database.

Read your own catalog before you assume anything is watching. Alert rules never roll onto an existing cluster on their own, and a service you deployed after the last config touch is not in the rendered rules until you refresh:

runos services prometheus alert-rule-catalog <prometheus-id> --cid ky3
runos services prometheus refresh-alert-catalog <prometheus-id> --cid ky3

set-alert-profile changes the sensitivity of every rule at once (essential, standard, aggressive), so pick it deliberately rather than as a reflex.

Do not reach for runos services postgresql resource-metrics to fill the gap. It returns CPU and memory only, with no disk and no database-size series.

For a real growth alert, author a custom rule. Validate the expression first, because it has to match series your own cluster actually scrapes:

runos services prometheus labels <prometheus-id> --cid ky3 --search volume

runos services prometheus validate-custom-rule <prometheus-id> --cid ky3 \
  --expr 'kubelet_volume_stats_used_bytes{namespace="postgresql-bcv4l"} > 40e9'

Watch the PostgreSQL namespace, not the Umami one. The analytics rows live in a database on the PostgreSQL instance, so the volume that grows belongs to postgresql-<id>. A namespace equals the service OSID, so substitute the OSID of the instance that hosts the Umami database.

Read the verdict rather than assuming the metric name. ok: true with a warning means the expression parses but currently matches nothing, which is what you get when that metric is not scraped here: volume usage bytes come from the kubelet, a scrape target no shipped RunOS rule depends on. labels --search volume tells you what your instance really holds, and if it holds nothing usable, fall back to node_filesystem_avail_bytes, which node.disk_space_low already proves is scraped.

Once it validates, persist it:

runos services prometheus custom-rules-add <prometheus-id> --cid ky3 \
  --name umami-db-growth \
  --expr 'kubelet_volume_stats_used_bytes{namespace="postgresql-bcv4l"} > 40e9' \
  --severity warning \
  --for 15m

Three constraints before you try. Custom rules need the system Prometheus instance with Alertmanager installed and are refused with a 400 anywhere else. The expression is validated live again on the way in, so a bad one is rejected without rolling the pod. The cap is 50 rules per instance. List them with runos services prometheus custom-rules <prometheus-id> --cid ky3.

Rate limits: three of them, and they apply before the pod does

RunOS puts Traefik rate limits in front of every Umami instance. Over-limit is HTTP 429 and the event is lost, because the tracker does not retry.

What it covers	Limit	Counted per
/api/send and /script.js, on both 443 and 8890	20 requests/s, burst 40	source IP
/api/send and /script.js, on both 443 and 8890	60 requests/s, burst 120	ingress node
Everything else on 8890, which is the dashboard and the login page	100 requests/s, burst 200	source IP

The per-node limit is not a cluster limit. Traefik keeps its counters in each process and runs one process per ingress node, so the cluster-wide collection ceiling is 60 per second multiplied by the number of ingress nodes. Quote the per-node number when you reason about a single node, and the multiple when you size the instance.

The dashboard limit is deliberately looser than the collection one. It bounds request rate, not concurrency, so it does not cap how many people can have the dashboard open.

Test from more than one source address. A load generator on one machine is capped at 20 requests per second by the first row above, whatever the pod could serve. That cap is the first thing you hit, not the last.

The collection endpoint has a real ceiling

This is the ceiling behind the rate limits above, measured with them out of the way. One pod with one CPU cannot absorb 100 collection requests per second without loss. Measured: a ten-minute HTTP/2 run launched exactly 60,000 requests at 100 per second. The client got 53,526 successful responses, 3,250 non-2xx responses, and timed out on 3,228 after 30 seconds. The database still recorded 59,405 events, so a client timeout is not the same as a lost event; the server drained its queue afterwards.

Read that as the honest shape: a single-CPU instance serves a low-traffic site comfortably and starts shedding and queueing well below 100 requests per second. Size up, or accept the shedding, rather than assuming the published class covers a traffic spike.

The counts include some bots

Do not promise bot-free numbers. Umami rejects obvious crawler user agents, and in the live test a synthetic user agent was refused and created no event. But a request with an empty user agent was counted. Anything that can send an HTTP request to a public collection endpoint can put a number in your dashboard.

The collection endpoint is unauthenticated by design, because a browser has no credential to present. Umami validates the website id and returns HTTP 400 with Website not found for an unknown one, which is the only application-level boundary on collection.

Treat the data as personal data

Umami sets no cookie. That is not the same as collecting nothing about a person.

IP addresses, user agents, page URLs, referrers, and any custom event properties you send are personal data. Cookie-less operation does not remove a consent obligation, a privacy-notice obligation, or a data-subject request obligation. Those duties sit with you as the operator of the site.

RunOS makes no claim of legal compliance for any regime, and neither should any page you build on top of this. If you need an assessment, get one; the platform cannot give you one.

Practical controls, all of them yours to set on the tracker tag (see Deploy apps): data-do-not-track, data-domains, and data-before-send.

Content Security Policy

If your app sends a CSP, the tracker needs two entries or the browser silently blocks it:

• script-src must allow the host of the scriptUrl value (UMAMI_SCRIPT_URL in the wiring example).
• connect-src must allow the host of the collectUrl value (UMAMI_COLLECT_URL in the example).

Both hosts come from the env vars your requires block mapped, so read them rather than hardcoding a hostname that changes with the cluster domain.

Backups and upgrades

The Umami database is an ordinary RunOS-managed PostgreSQL database, so it backs up and restores the same way as any other, and by the same rule: backups are opt-in. Nothing is backed up until you configure a destination. Follow Backups and restore above; there is no runos command for it, so it is a console or API job.

Read the blast radius before you rely on this. The backup covers the whole PostgreSQL instance, and restoring it means creating a new instance carrying every database that instance held, not just the Umami one. If the Umami database shares an instance with your application database, you cannot roll analytics back on its own. Give Umami its own PostgreSQL instance when that matters to you.

Umami runs prisma migrate deploy on every container start, so an upgrade applies migrations as the new pod boots. Verify a restore before you rely on it, and test an upgrade on a copy first: a schema migration is not automatically reversible.

Outside the support boundary

Umami v3.3.1 still contains undocumented ClickHouse and Kafka code paths. They are not part of the RunOS v1 service and RunOS does not support them. The managed service is PostgreSQL only. Umami v3 removed MySQL support upstream, so there is no MySQL option either.

Accounts and credentials

Roles are admin and limited. New members default to limited (everything except key and user management). Manage members and invites:

runos account users list
runos account invite --email teammate@example.com --account-role limited
runos account invites list

Invites expire (24h), so revoke a wrong address before then. Account, role, and invite management requires a human login session; PATs are rejected on those routes.

Personal access tokens (PATs) are for automation against the API. Format runos_pat_<keyId>.<secret>, sent as Authorization: Bearer. A PAT is account-scoped (one account, all clusters), expiring (expiresAt is required, future, and at most 365 days), and shown once: the server stores only a hash, so a lost token is recreated, not recovered.

runos account api-keys add      # token printed once
runos account api-keys list
runos account api-keys revoke <id>

The CLI reads a PAT from the RUNOS_API_KEY env var. Use a limited key for CI.

Notify API keys authenticate email sends through the Notify service. They are passed as X-API-Key (not Bearer), are account-scoped, and cannot read or change any other resource.

runos account notify-keys add   # keyValue printed once
runos account notify-keys list
runos account notify-keys delete <id>

A send returns 202 when the email is queued, not delivered: there is no bounce or delivery signal. Notify keys have no expiry and are hard-deleted (not revoked), so rotate by delete-and-recreate.