K3s HA + Cllium homelab set up - Updated

 

 

    Today I would like to document my base set up for my homelab. I try serveral Linux distros RHEL, Ubuntu, Fedora combined with K8s, RKE2, K0s, minikube and K3s. To focus more on learning Kubernetes, I select in my opinion simple but functional solution. When I tryed to install K8s, I focused too much time of setup that learning Kubernetes. This help experience, was not wasted time. It helps me to understand that YAGNI and KISS. 

 

Here I wanted to document my basic homelab set up.

 

I first step, I installed Ubuntu server in version 22.04.5 LTS  with installed OpenSSH server. Instalation OpenSSH during Ubuntu instalation was very productive.  

I established connection from my local laptop to my Ubuntu server's. 

 

 

After connection via SSH was etablished, I begin instalation. 

I present code used for instalation. Link for latest used code in doku(1). 

Here I check for updates and begin instalation of Docker. 

 

Step 1.

Step executed on master, worker 1 and worker 2 node.  

It is good practice to verify if Docker is installed(2).

sudo systemctl status docker

Step 0. MASTER NODE ONLY

Instalation K3s. This code execute only on master node. By default K3S has Flannel for CNI. For CNI, I choose Cilium(3).

 # Install K3s without Flannel:

curl -sfL https://get.k3s.io | sh -s - \
  --flannel-backend=none \
  --disable-kube-proxy \
  --disable servicelb \
  --disable-network-policy \
  --disable traefik \
  --cluster-init

mkdir -p $HOME/.kube
sudo cp -i /etc/rancher/k3s/k3s.yaml $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
echo "export KUBECONFIG=$HOME/.kube/config" >> $HOME/.bashrc
source $HOME/.bashrc

HINT:

Install Cilium before nodes will be added to cluster!!! 

After instalation K3s on master node, execute code below.

mkdir -p $HOME/.kube
sudo cp -i /etc/rancher/k3s/k3s.yaml $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
echo "export KUBECONFIG=$HOME/.kube/config" >> $HOME/.bashrc
source $HOME/.bashrc

To add new nodes to cluster, node-token is needed. To get token, execute code below:

 

sudo cat /var/lib/rancher/k3s/server/node-token 

 

STEP CILIUM

 

STEP APP;s

 

# Add Docker's official GPG key:

sudo apt-get update

sudo apt-get install ca-certificates curl

sudo install -m 0755 -d /etc/apt/keyrings

sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc

sudo chmod a+r /etc/apt/keyrings/docker.asc

# Add the repository to Apt sources:

echo \

  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \

  $(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}") stable" | \

  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

sudo apt-get update

 

 

# Software instalation CLI:

sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

 

 

Step.3. WORKER NODE ONLY

 

In this step, I will modify command to join worker node to cluster (3).

Join command from (3)

 

curl -sfL https://get.k3s.io | K3S_URL='https://${MASTER_IP}:6443' K3S_TOKEN=${NODE_TOKEN} sh -

I substituted:

${MASTER_IP} - 192.169.0.100

${NODE_TOKEN} - output from CLI sudo cat /var/lib/rancher/k3s/server/node-token 

 

sudo curl -sfL https://get.k3s.io | K3S_URL='https://192.168.0.100:6443' K3S_TOKEN=K10dcb607924c66ec6efde783a293266f687d7c8d684a3181e1ab63633b06f0d8a8::server:cd2e0147f813b077cf85c0c2c0b93959 sh -

 

 

To confirm presence worker nodes in cluster execute command.

 

 sudo kubectl get nodes -o wide

To update worker1 and worker2 roles execute command.

kubectl label nodes worker1 kubernetes.io/role=worker1

kubectl label nodes worker2 kubernetes.io/role=worker2

 

 

 
In last step, Install Cillium. MASTER NODE ONLY !

Latest code to add repo and instalation, You can find on website(3).

 # Add Cilium repo on K3s:

CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)  CLI_ARCH=amd64
curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-${CLI_ARCH}.tar.gz
sudo tar xzvfC cilium-linux-${CLI_ARCH}.tar.gz /usr/local/bin
sudo rm cilium-linux-${CLI_ARCH}.tar.gz

*Add rights to current user

sudo chown $USER:$USER /etc/rancher/k3s/k3s.yaml

 

 

export KUBECONFIG=/etc/rancher/k3s/k3s.yaml

 

 # Install Cilium repo on K3s:

API_SERVER_IP=192.168.0.110
API_SERVER_PORT=6443
cilium install \
  --set k8sServiceHost=${API_SERVER_IP} \
  --set k8sServicePort=${API_SERVER_PORT} \
  --set kubeProxyReplacement=true

 

 cilium status --wait

# Install Cilium confirmed:

 

In next entry I will Install Argo CD with first app. See You soon....

 

 

Install HELM:

curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | sudo bash 

 

Useful commands:

export KUBECONFIG=~/.kube/config

---

mkdir ~/.kube 2> /dev/null
sudo k3s kubectl config view --raw > "$KUBECONFIG"
chmod 600 "$KUBECONFIG"

 or

sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config && sudo chown $USER ~/.kube/config

sudo chmod 600 ~/.kube/config && export KUBECONFIG=~/.kube/config

~~~

Uninstall K3s (6). 

Uninstalling Servers:

 

To uninstall K3s from a server node, run:

/usr/local/bin/k3s-uninstall.sh

Uninstalling Agents

To uninstall K3s from an agent node, run:
/usr/local/bin/k3s-agent-uninstall.sh

Doku

 

 

1. https://docs.docker.com/engine/install/ubuntu/

2. https://computingforgeeks.com/install-kubernetes-on-ubuntu-using-k3s/

3. https://docs.cilium.io/en/stable/installation/k3s/#install-a-master-node

4. https://www.linuxbuzz.com/install-k3s-kubernetes-cluster-on-ubuntu/

5. https://computingforgeeks.com/install-kubernetes-on-ubuntu-using-k3s/

6. https://docs.k3s.io/installation/uninstall