Preparation
In part 1 and part 2 of this series, I stood up a few ssh-reachable Ubuntu machines. There are a few steps that I completed on each machine to ensure that they were ready to be in a cluster. If you’re managing a large swathe of machines (really, more than 2) then it’s worth the time to do this in Ansible. I’ll be posting raw commands here that can be manually applied to each server via SSH.
First, let’s extend our volumes to use the entire disk. For some reason, my machines were formatted with much of the disk unavailable. If your drive names are the same as mine then this will extend your disks to fill the space available.
sudo lvextend -l +100%FREE /dev/ubuntu-vg/ubuntu-lv
sudo resize2fs /dev/mapper/ubuntu--vg-ubuntu--lvNext, we’ll change the host names of our machines. I went with node1, node2, etc for mine.
hostnamectl your_new_hostname_hereI also rewrote my hosts files to match.
127.0.0.1 localhost
127.0.1.1 new_hostname_here
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allroutersNext, I installed docker. I’m fairly sure that this is the hard way to install docker, but this avoids using apt-key which is deprecated (see why here).
# Download and unwrap the signature
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor --yes -o /etc/apt/keyrings/docker.gpg
# Grab repo info and add
curl -fsL https://download.docker.com/linux/ubuntu/dists/jammy/Release |
sed -rn -e "s/(^Components:[ ]+.*$)/\1/p" \
-e "s/^Suite:[ ]+(.*)$/Suites: \1/p" |
cat - <(echo "Types: deb") |
cat - <(echo "Signed-By: /etc/apt/keyrings/docker.gpg") |
cat - <(echo "URIs: https://download.docker.com/linux/ubuntu") |
sudo tee /etc/apt/sources.list.d/docker.sources
# Install docker
sudo apt update
sudo apt install -y docker-ce docker-ce-cli docker-compose
# Enable the service
sudo systemctl enable dockerThere’s one last thing that I do when prepping a new Ubuntu host and it’s completely unnecessary. For some time, Canonical has been putting what can only be called ads into the MOTD. That means every time you ssh into your Ubuntu machine (which, admittedly, shouldn’t be often) you get slapped in the face with a call to action for something they sell. Here’s how to remove that if it bothers you like it does me.
sudo sed -i "s/ENABLED=1/ENABLED=0/g" /etc/default/motd-news
sudo sed -irn \
-e '/".*if esm service was enabled.*"/a \ \ \ \ return # Added by sed' \
-e '/def _output_esm_service_status\(.*\):/a \ \ \ \ return # Added by sed' \
/usr/lib/update-notifier/apt_check.py
sudo chmod a-x /etc/update-motd.d/10-help-text
sudo chmod a-x /etc/update-motd.d/50-motd-news
sudo /usr/lib/update-notifier/update-motd-updates-available --forceBootstrap a k3s cluster
It’s time to bootstrap the new k3s cluster. You can always follow the instructions in the quick-start guide, but some very patient people have put together an Ansible playbook that can reliably bootstrap a cluster with multiple machines without much effort at all. If you don’t already have Ansible installed, instructions for getting it up and running are here. In a nutshell, you’ll need to already have Python 3 installed, then you run pip install ansible.
The Ansible playbook I’m using is distributed by the same people that maintain k3s. You’ll want to download it here. Either download and un-zip the repo or clone it locally. Ansible needs an inventory file so that it knows where to apply the playbook. A sanitized version of me inventory file is included below.
all:
children:
server:
hosts:
node1:
ansible_host: 192.168.0.101
iface_name: eno1
agent:
hosts:
node2:
ansible_host: 192.168.0.102
iface_name: eno1
node3:
ansible_host: 192.168.0.103
iface_name: eno1
node4:
ansible_host: 192.168.0.104
iface_name: enp2s0
k3s_cluster:
children:
server:
agent:
vars:
ansible_user: k3s_user
k3s_version: v1.29.1+k3s2
systemd_dir: /etc/systemd/system/
extra_server_args: >
--node-external-ip {{ ansible_host }}
--flannel-iface {{ iface_name }}
--flannel-backend=host-gw
extra_agent_args: >
--node-external-ip {{ ansible_host }}
--flannel-iface {{ iface_name }}
api_endpoint: "{{ hostvars[groups['server'][0]]['ansible_host'] | default(groups['server'][0]) }}"
token: ansible_k3sMake sure to customize the file to suit your needs:
- Update the node names to match the node names you set previously.
- Update the
iface_nameto match the actual primary ethernet interface names on your hosts. Since my cluster has two different models of pc, the names aren’t the same among hosts. - Update
all.vars.ansible_userto the username you configured on your hosts. - Update
all.vars.k3s_versionto the version of k3s you want installed. - If you’re feeling security conscious, change
all.vars.k3s_versionto something more secret.
You’ve got a prepped Ubuntu host, an Ansible playbook, and an inventory file. Now run your playbook and watch in awe as your cluster comes to life.
ansible-playbook k3s-ansible/playbook/site.yaml -i path/to/your/inventory.yaml -KkIf everything goes well, you’ll see confirmation from Ansible that the playbook ran correctly.
PLAY RECAP *************************************************************************************************
node1 : ok=15 changed=7 unreachable=0 failed=0 skipped=47 rescued=0 ignored=1
node2 : ok=10 changed=6 unreachable=0 failed=0 skipped=35 rescued=0 ignored=0
node3 : ok=10 changed=6 unreachable=0 failed=0 skipped=35 rescued=0 ignored=0
node4 : ok=10 changed=6 unreachable=0 failed=0 skipped=35 rescued=0 ignored=0 If not, you’ll need to troubleshoot a bit. During testing, I had an issue with my agent nodes failing to join the main server due to a 401 error. To resolve this, I ran the k3s uninstall script and re-ran the Ansible playbook. After doing so, my other nodes joined the cluster without issue.
Connect to the cluster
Now that you’ve got a running kubernetes cluster, you’ll probably want to do stuff with it. At this point in the process, the easiest way to interact with your cluster is by connecting to a server node and using kubectl.
k3s_user@node1:~$ kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
local-path-provisioner-849db5d44d-kbh7g 1/1 Running 0 30m
svclb-traefik-e3ae6aaa-mxc9x 2/2 Running 0 30m
svclb-traefik-e3ae6aaa-sd285 2/2 Running 0 30m
svclb-traefik-e3ae6aaa-jgt5g 2/2 Running 0 30m
svclb-traefik-e3ae6aaa-mxts2 2/2 Running 0 30m
traefik-ff44564c4-vfzjc 1/1 Running 0 30m
coredns-6799fbcd5-dhw9c 1/1 Running 0 30mIf you want to connect to your cluster from another machine, you’ll need to install kubectl and create a kubeconfig on that machine. Copy the config from your server node (either from ~/.kube/config or /etc/rancher/k3s/k3s.yaml) to your workstation. If you’re on linux or mac the config should go to ~/.kube/config whereas for windows it should go to %userprofile%/.kube/config. Finally, you’ll need to manually edit the config on your local workstation so that it points to the cluster. Search for server: https://127.0.0.1:6443 and replace 127.0.0.1 with the local ip address of your server node (192.168.0.101 in my example). After that, you should be able to run kubectl commands against the cluster.
A word of caution: guard your kubeconfig carefully. The information in that file is sufficient to run nearly any type of workload on your cluster and could be abused by a bad actor.
Give yourself a pat on the back. You’ve got a real life kubernetes cluster running on your very own home lab. One of an elite few. In the next post, we’ll make the cluster even better by adding some creature comforts such as external dns and a storage provider.