Skip to main content

Node Installation

When you add a server to your RunOS cluster through the Console, an automated installation process transforms it from a basic Ubuntu server into a fully configured Kubernetes node. This document explains what happens during this process and what gets installed on your server.

Overview​

The installation process typically takes 10-20 minutes and is completely automated. The Node Agent handles all configuration, requiring no manual intervention.

What Gets Installed:

  • Essential system utilities
  • WireGuard VPN for secure connectivity
  • DNS infrastructure for service discovery
  • Container runtime (containerd)
  • Kubernetes components (kubelet, kubeadm, kubectl)
  • Network plugins (Cilium CNI)
  • Supporting tools (HAProxy, Helm)

What Gets Configured:

  • System resources and kernel parameters
  • Network settings and routing
  • DNS resolution
  • VPN mesh connectivity
  • Kubernetes cluster membership
  • SSL certificate management

Installation Phases​

Phase 1: System Preparation​

What Happens: The foundation is laid with essential packages and system-level configurations.

Key Actions:

  • Install essential utilities (jq, htop, net-tools)
  • Configure hostname
  • Increase resource limits (inotify watchers, file descriptors)
  • Disable swap (Kubernetes requirement)
  • Load kernel modules (br_netfilter, overlay, nf_conntrack)
  • Configure network parameters for Kubernetes

Why These Changes:

Kubernetes requires specific system configurations to function properly. Swap must be disabled so pod memory limits work correctly. Kernel modules enable container networking and bridge traffic filtering. Resource limits are increased to handle many file watches and connections that microservices generate.

Phase 2: VPN Setup​

What Happens: Secure VPN infrastructure connecting all your servers is established.

Key Actions:

  • Install WireGuard VPN software
  • Generate unique public/private key pairs for this node
  • Configure two VPN interfaces:
    • wg0 (172.24.0.0/16) - Kubernetes internal communication
    • wg1 (172.24.200.0/21) - User access network
  • Register public keys with RunOS platform
  • Retrieve peer information for other cluster nodes
  • Configure persistent keepalive to maintain connections

Why VPN:

The VPN creates an encrypted mesh network between all your servers, regardless of their location or whether they have public IPs. This makes networking simple and secure - servers can be in different data centers, cloud providers, or behind NAT, and they'll still communicate securely.

Network Architecture After VPN:

Server A (172.24.1.10) ←→ Encrypted Tunnel ←→ Server B (172.24.1.20)
    ↓                                                  ↓
Pod Network                                      Pod Network
(172.25.1.0/24)                                 (172.25.2.0/24)

Phase 3: DNS Configuration​

What Happens: DNS infrastructure is configured so services can find each other by name.

Key Actions:

  • Install dnsmasq (lightweight DNS forwarder)
  • Configure systemd-resolved (system DNS resolver)
  • Disable DHCP DNS to prevent conflicts
  • Route cluster domains to internal DNS
  • Configure external DNS fallback (Cloudflare, Google)

How DNS Works After Setup:

Application queries database.default.svc.cluster.local
    ↓
systemd-resolved (127.0.0.53)
    ↓
dnsmasq (on wg0 VPN IP)
    ↓
CoreDNS (Kubernetes DNS)
    ↓
Returns pod IP address

External domains (like google.com) resolve through dnsmasq to Cloudflare/Google DNS, while cluster services resolve through Kubernetes internal DNS.

Phase 4: Container Runtime​

What Happens: containerd is installed as the container runtime for Kubernetes.

Key Actions:

  • Install containerd with version pinning
  • Generate default configuration
  • Enable SystemdCgroup (required by Kubernetes)
  • Configure sandbox image to match Kubernetes version
  • Start and enable containerd service

Why containerd:

containerd is the industry-standard container runtime for Kubernetes, providing reliable and efficient container execution. It's maintained by the Cloud Native Computing Foundation and used by most production Kubernetes deployments.

Phase 5: Kubernetes Installation​

What Happens: Kubernetes software and supporting tools are installed.

Key Components:

  • kubelet - Node agent that manages containers
  • kubeadm - Tool for cluster bootstrapping
  • kubectl - Command-line tool for cluster management
  • HAProxy - Load balancer for Kubernetes API
  • Helm - Kubernetes package manager
  • Cilium CNI - Network plugin for pod connectivity

All components are version-pinned to prevent accidental upgrades that could break compatibility.

HAProxy Configuration:

HAProxy provides a stable endpoint for Kubernetes API access. If a control plane node fails, HAProxy automatically routes to healthy nodes, ensuring your cluster remains accessible.

Phase 6: Cluster Join​

What Happens: The node joins your Kubernetes cluster. This phase differs based on node type.

First Control Plane Node:

This node creates a new Kubernetes cluster:

  • Initialize etcd (cluster state database)
  • Generate Certificate Authority
  • Enable secrets encryption at rest (AES-CBC with random key)
  • Assign pod network CIDR (172.25.0.0/16)
  • Deploy Cilium CNI for networking
  • Start control plane services (API server, controller manager, scheduler)
  • Generate join tokens for future nodes

Additional Control Plane Nodes:

These nodes join the existing control plane for high availability:

  • Synchronize encryption keys from first control plane
  • Join cluster with control plane flags
  • Install admin credentials
  • Replicate etcd data

Worker Nodes:

These nodes join as workers to run application workloads:

  • Verify connectivity to control plane
  • Execute cluster join command
  • Configure kubelet with VPN IP
  • Ready to accept pod workloads

Network Configuration After Installation​

IP Addressing​

Each server receives multiple IP addresses:

  1. Physical Network IP - Your server's actual interface IP
  2. wg0 VPN IP - Internal cluster network (172.24.X.X)
  3. wg1 VPN IP - User access network (172.24.200.X)
  4. Pod CIDR - Range of IPs for pods on this node (172.25.X.X/24)

Traffic Routing​

Pod-to-Pod Communication (Same Node):

Pod A → Container bridge → Pod B

Pod-to-Pod Communication (Different Nodes):

Pod A → Cilium CNI → Node A wg0 → WireGuard tunnel → Node B wg0 → Cilium CNI → Pod B

All inter-node traffic automatically flows through encrypted VPN tunnels.

File Locations​

Configuration Files​

Key configuration files created during installation:

  • /etc/wireguard/wg0.conf - Kubernetes internal VPN
  • /etc/wireguard/wg1.conf - User access VPN
  • /etc/dnsmasq.d/runos.conf - DNS forwarding rules
  • /etc/systemd/resolved.conf - System DNS resolver
  • /etc/containerd/config.toml - Container runtime
  • /etc/haproxy/haproxy.cfg - API load balancer

RunOS Files​

  • /etc/runos/config.yaml - Node Agent configuration
  • /etc/runos/mtls.crt - Node Agent certificate
  • /etc/runos/mtls.key - Node Agent private key
  • /var/log/runos/nodeagent.log - Node Agent logs

Kubernetes Files​

  • /etc/kubernetes/kubelet.conf - Node credentials
  • /etc/kubernetes/admin.conf - Admin credentials (control plane only)
  • /var/lib/kubelet/ - Kubelet working directory
  • /var/lib/etcd/ - etcd database (control plane only)

Verification After Installation​

Once installation completes, verify everything is working:

# Check node is ready
kubectl get nodes

# Verify all system pods running
kubectl get pods -A

# Test DNS resolution
nslookup kubernetes.default.svc.cluster.local

# Check VPN interfaces
sudo wg show

# Verify container runtime
systemctl status containerd

All commands should show healthy status.

Security Considerations​

Credentials Created​

During installation, sensitive credentials are generated and stored:

  • Kubernetes certificates in /etc/kubernetes/pki/ (control plane only)
  • WireGuard private keys in /etc/wireguard/ (600 permissions)
  • Node Agent certificates in /etc/runos/
  • Admin kubeconfig in /root/.kube/config

All files are protected with appropriate permissions and should never be shared or committed to version control.

Network Ports Used​

Control Plane Nodes:

  • 6443: Kubernetes API
  • 6446: HAProxy (API load balancer)
  • 2379-2380: etcd
  • 10250: Kubelet API
  • 51820/51821: WireGuard VPN (UDP)

Worker Nodes:

  • 10250: Kubelet API
  • 51820/51821: WireGuard VPN (UDP)

Encryption:

  • VPN: WireGuard encryption
  • Kubernetes API: TLS
  • Secrets at rest: AES-CBC

Common Installation Issues​

VPN Not Connecting​

Symptoms:

  • Can't ping other nodes via wg0
  • wg show displays no peers

Checks:

# Check WireGuard service
systemctl status wg-quick@wg0
sudo wg show wg0

# Resync VPN peers
sudo ./nodeagent sync vpn

# Check for firewall blocking UDP
sudo iptables -L OUTPUT -v -n | grep udp

DNS Not Resolving​

Symptoms:

  • Services can't find each other
  • nslookup fails for cluster domains

Checks:

# Check DNS services
systemctl status dnsmasq
systemctl status systemd-resolved

# Verify DNS configuration
cat /etc/resolv.conf

# Test resolution
nslookup your-cluster.runos.com

Node Won't Join Cluster​

Symptoms:

  • Node doesn't appear in kubectl get nodes
  • Shows offline in Console

Checks:

# Test VPN connectivity to control plane
ping <control-plane-wg0-ip>

# Check kubelet logs
journalctl -u kubelet -f

# Verify system time is accurate
timedatectl

# Check HAProxy is routing
systemctl status haproxy

Pods Not Starting​

Symptoms:

  • Pods stuck in "Pending" or "ContainerCreating"

Checks:

# Check containerd
systemctl status containerd
crictl ps -a

# Verify Cilium network
kubectl get pods -n kube-system | grep cilium

# View pod events
kubectl describe pod <pod-name>

What You Can Customize​

While RunOS handles most configuration automatically, you can customize:

Network Ranges (before cluster creation):

  • VPN IP ranges (if they conflict with existing networks)
  • Pod network CIDR
  • Service network CIDR

Resource Allocations:

  • Node labels for workload placement
  • Resource quotas per namespace
  • Storage class preferences

Security Policies:

  • Network policies to restrict traffic
  • Pod security standards
  • RBAC policies (coming soon)

Contact RunOS support if you need to customize network ranges or other fundamental settings before cluster creation.

Best Practices​

  1. Start with recommended specs - Meet minimum system requirements
  2. Use clean Ubuntu installations - Avoid pre-configured systems
  3. Ensure stable networking - Reliable internet connectivity required
  4. Monitor installation progress - Watch Console for phase completion
  5. Verify after installation - Run verification commands
  6. Document customizations - Keep track of any custom configurations
  7. Plan IP addressing - Ensure VPN ranges don't conflict with existing networks

Understanding What RunOS Changes​

RunOS modifies your server to prepare it for Kubernetes:

System Changes:

  • Kernel parameters (networking, security)
  • Network configuration (VPN, DNS, iptables)
  • Package installations (Kubernetes, containerd)
  • Service configurations (systemd services)

What Remains Unchanged:

  • Your existing data and applications
  • SSH configuration
  • User accounts
  • Other installed software (unless conflicts with Kubernetes)

You retain full root access to your servers and can access Kubernetes directly using kubectl when needed. RunOS manages the Kubernetes workloads, but you control the underlying infrastructure.