A Personal,Private,Portable Cloud on Kubernetes and RaspberryPis

25 min readSep 30, 2024

Part 2: Create Kubernetes Cluster with K3s and install Cilium and ArgoCD using OpenTofu.

Articles in the series:

1. Requirements, Hardware and Network setup
2. Create Kubernetes Cluster with K3s and install Cilium and ArgoCD using OpenTofu.
3. GitOps with ArgoCD: Install GatewayAPI, CiliumLB and Kube-Prometheus-Stack (coming soon)
4. PiHole, Tailscale and Custom domains: Securely access from everywhere (coming soon)
5. NAS Storage on Kubernetes with NFS Provisioner in RAID 1 mode (coming soon)
6. Back to hardware — Cooling the cluster and monitor with telegraf (coming soon)
7. Private file hosting service with OwnCloud (coming soon)

In Part 1, we explored the requirements, hardware, and network setup for the cluster. In Part 2, we will make the cluster functional by installing the OS on the Raspberry Pis, setting up the Kubernetes cluster using K3s with 1 master and 3 worker nodes, and initiating the automation journey. We will use OpenTofu to install Cilium as the cluster’s CNI. Cilium will provide integrated Gateway API support and Cilium load balancers, which we will further explore in Part 3: GitOps with ArgoCD: Installing GatewayAPI, CiliumLB, and Kube-Prometheus-Stack. Additionally, we will use ArgoCD to implement GitOps for managing the cluster.

The helper repo for this article you can find it here

Disclaimer: The technologies and hardware discussed in this series have been carefully studied and chosen based on their effectiveness in the outlined scenarios and presented without any sponsorship or endorsement. It is highly advisable to conduct thorough research and customize these implementations to align with unique requirements and preferences.

Note: I will not dive deep into the theoretical details of each technology, package, or software used unless it directly serves the purpose of this article. For a more in-depth understanding of each product’s specifications, please refer to their respective documentation to gain a deeper knowledge and insight.

Installing OS on the Pis

The first step is to install the OS on each Raspberry Pi. For this setup, we use four 64GB microSD cards, one for each Pi. Ubuntu Server 22.04.01 LTS (64-bit) will be installed on the three Raspberry Pi 4 devices, while Ubuntu Server 24.04.01 LTS (64-bit) will be used for the Raspberry Pi 5. The reason for using two different Ubuntu versions is that, as mentioned in Part 1, the Raspberry Pi 5 was added to the cluster at a later stage, and version 24.04 was not available when the cluster was initially created. Additionally, 22.04.01 LTS is not supported on Raspberry Pi 5. If you’re starting from scratch, it is recommended to use Ubuntu Server 24.04 LTS for all the Pis to maintain consistency.

Preparing the microSD cards

The Raspberry Pi comes with a convenient tool for installing an operating system: the Raspberry Pi Imager. This imaging utility offers a quick and easy way to install a variety of OS options supported by Raspberry Pi devices. It is available for multiple operating systems, including, but not limited to, macOS, Windows, and Ubuntu Desktop.

Steps are pretty straight forward:

Raspberry Pi Device: Choose in which RaspPi you want to install the OS. This section acts mostly as a filtering for the Operating system list of the next step.

*Image 2.2 — Raspberry Pi Device selection*

Operating System: Here, you choose which OS to install. The Raspberry Pi Imager provides a list of operating systems based on their purpose. To install Ubuntu, navigate to Other general-purpose OS => Ubuntu. The available distributions will depend on the device selected in the first step. For Raspberry Pi 4, you will see Ubuntu versions starting from 20 and above, while for Raspberry Pi 5, only versions 24 and above are available. Choose the version that best suits your needs. For this project, by the writing of this article, the selected versions are Ubuntu Server 22.04.05 LTS (64 bit) for RaspPi 4 and Ubuntu Server 24.04.01 LTS (64 bit) for RaspPi 5

Image 2.3 — Operating system selection list

*Image 2.4 — RaspPi4 general-purpose operating system selection list*

*Image 2.5 — Available Ubuntu versions for RaspPi 4*

*Image 2.6 — Available Ubuntu versions for RaspPi 5*

Storage: Here you select the SD card. The utility automatically detects any external drive/card inserted on your laptop/pc

*Image 2.7 — Available Storage to right the image*

Everything is now ready, and the “Next” button will become active. Upon clicking it, a warning will appear, notifying you that all data on the SD card will be erased. You’ll also be prompted to define custom settings such as hostname, SSH keys, etc. You can skip this step, as we will configure these settings later. Once you confirm, the process of writing the files to the card will begin. After completion, the basic OS will be fully functional and ready to be installed on the Raspberry Pi.

*Image 2.8 — Files in microSD after imager completes*

However, we’re not completely done yet. Before proceeding, we need to configure the network settings and set up cloud-init.

Network allocations

In Part 1, we talked about the architectural design of our project. Let’s focus on the cluster setup

As previously outlined, the cluster consists of three Raspberry Pi 4 devices and one Raspberry Pi 5, all operating within the 10.54.42.0/24 network. One of the initial decisions was to assign static IP addresses to the Pis, ensuring they remain constant. To distinguish between the Raspberry Pi 4 and Raspberry Pi 5 devices at the network level, we decided to allocate IPs for Raspberry Pi 4s in the 10.54.42.1xx range, while Raspberry Pi 5 will be in the 10.54.42.2xx range. With this in mind, the final IP allocation is as follows:

Master Node: 10.54.42.101
Worker1 Node: 10.54.42.102
Worker2 Node: 10.54.42.103
Raspberry Pi 5 Worker1 Node: 10.54.42.201

Network configuration

To apply the necessary configuration, we need to modify the network-config file. The key configurations include setting a static IP address, defining the gateway IP, specifying the nameservers, and disabling DHCP to prevent automatic IP allocation.

If you are using Ubuntu Server 22.04, the syntax for the configuration can be found here:

If you are using Ubuntu Server 24.04, the syntax is different and can be found here:

Be sure to use the correct version, as the configurations are slightly different for each.

Installing Kubernetes

There are various distributions and methods to install Kubernetes, such as MicroK8s or using kubeadm directly. For this implementation, we will be using K3s.

The cloud-init

Before we dive into the Kubernetes installation, let’s first discuss cloud-init. Originally developed for Ubuntu, cloud-init is now available across many Linux distributions. According to the Ubuntu documentation:

cloud-init is a tool for automatically initializing and customizing an instance of a Linux distribution.By adding cloud-init configuration to your instance, you can instruct cloud-init to execute specific actions at the first start of an instance. Possible actions include, for example: Updating and installing packages, Applying certain configurations, Adding users, Enabling services, Running commands or scripts, Automatically growing the file system of a VM to the size of the disk

Cloud-init is the perfect tool to initialize and automate essential components of the cluster, including K3s. Initializing cloud-init is straightforward, requiring modification of the user-data file.

For our cluster, we will use three different cloud-init scripts:

For the master node: Master Node Cloud-init Script
For worker nodes using Ubuntu Server 22.04: Worker Node (Ubuntu 22.04) Cloud-init Script
For the worker node using Ubuntu Server 24.04: Worker Node (Ubuntu 24.04) Cloud-init Script

We use three different cloud-init configurations because each node requires specific settings based on its version and role within the cluster. The files linked above contain all the initial configuration needed to set up the cluster. I will explain the contents of these configurations in this and future articles, depending on their relevance to the topic being covered.

Common configurations for all nodes

Let’s break down the cloud-init common contents in all 3 cloud configs (the parts that are relevant to this article).

ssh_pwauth: false

We aim to make SSH connections to the nodes as secure as possible. By setting ssh_pwauth to false, we disable password authentication for SSH, preventing users from logging in via that method. Instead, we will use SSH key-based authentication, which is significantly more secure.

hostname: your-host-name# replace with the desired hostname
groups:
  - ubuntu: [root, sys]

users:
  - name: replace-me # replace with your own username
    groups: sudo
    shell: /bin/bash
    lock_passwd: true
    ssh_authorized_keys:
      - ssh-rsa replace-me # replace with your own ssh key
    sudo: ALL=(ALL) NOPASSWD:ALL

hostname sets the desired hostname of the machine

groups defines the group ubuntu and specifies that the root and sys groups will be part of it. This controls access and permissions for different user groups on the system.

In users we create the users of the system. Here we create one user which

Is added to the sudo group, granting it administrative privileges
Specifies /bin/bash as its default shell
lock_passwd: true disables the password-based login. So the user cannot create a password
ssh_authorized_keys: This section allows SSH key-based authentication. Should be replaced with the user’s public SSH. It is recommended to create a new key and store it in a new path in our system, for example .ssh/local_cluster/id_rsa to avoid overwriting your default ssh key usually stored directly under the .ssh folder
In sudo, this command grants the user password-less sudo access, allowing the user to execute commands as root without needing to enter a password (NOPASSWD:ALL).

package_update: true
package_upgrade: true
packages:
.....

These configurations execute the apt update and apt upgrade commands to ensure the system’s packages are up to date. They also install various essential packages needed for the system to function properly, such as those required for K3s, Cilium, and other components.

write_files:
......

write_files directive allows us to directly write content to files that are used by various components.

runcmd:
....

runcmd runs arbitrary commands after all the previous steps have finished

Considerations When Executing Cloud-Init

Cloud-init runs only once during the system’s first boot. If an error occurs during execution, cloud-init does not automatically retry or continue with the remaining steps. The process halts at the point of failure. However, you can manually re-execute specific scripts if necessary. For more information on troubleshooting or re-running cloud-init, refer to the official documentation.

To debug issues, cloud-init logs are available in /var/log/cloud-init.log and /var/log/cloud-init-output.log on each node. You can access these logs by SSHing into the node using the command:
ssh <optional path to ssh-key if needed> <node ip>
These logs can help you identify why the process failed.

Cloud-init operates in multiple stages (init, config, final). Reviewing the logs will allow you to pinpoint which stage the failure occurred in.

Installing K3s

Now that we have covered the cloud-init process, let’s move on to installing K3s. I won’t go into detail about the necessary packages, as these have already been installed via the provided cloud-init configuration. For any adjustments or optimizations, please refer to the relevant documentation to review and modify the package installation as needed.

Creating the master node

Master node is created by the following command:

- export MASTER_IP=10.54.42.101
- curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--cluster-cidr=10.54.42.0/24 --flannel-backend=none --disable-network-policy --disable=servicelb --node-ip=${MASTER_IP} --node-external-ip=${MASTER_IP} --bind-address=${MASTER_IP} --token 12345" sh -s -

First we create the env var MASTER_IP where we store the master’s IP. Then we install k3s master node as described by the official documentation and we provide the necessary flags needed for our project.

— cluster-cidr=10.54.42.0/24:

This specifies the CIDR (Classless Inter-Domain Routing) block for pod networking within the cluster. In this case, it defines the pod IP range as 10.54.42.0/24.

— flannel-backend=none:

This disables Flannel, the default network provider for K3s as we will use Cilium instead

— disable-network-policy:

Similar to the previous flag, we disable Kubernetes’ default network policies as we are going to use Cilium

— disable=servicelb:

K3s comes with a built-in service load balancer. We are going also here to use Cilium so we dont need that

— node-ip=${MASTER_IP}:

Sets the internal IP address of the master node.

— node-external-ip=${MASTER_IP}:

Specifies the external IP address for the master node. It’s often used when you need external access to the node from outside the cluster.

— bind-address=${MASTER_IP}:

This option specifies the IP address on which the K3s master will listen for incoming API requests.

— token 12345:

This sets a pre-shared token (in this case, 12345) that is used to secure communication between the master and worker nodes. Worker nodes need to supply this token to join the cluster.

Everything is now ready and configured. Insert the microSD card into the Raspberry Pi and power it on. Cloud-init will begin the initialization process. After a few minutes, once the SSH configuration is set up, you’ll be able to log in to the node.

You can SSH into the node once cloud-init has completed the SSH configuration, although it may still be processing the remaining steps. To check if cloud-init has finished or if it encountered any errors, you can inspect the /var/log/cloud-init-output.log file. Cloud-init execution is complete when you see a line similar to the following, with no further logs being recorded:

Cloud-init v. 24.3.1-0ubuntu0~22.04.1 finished at Sun, 29 Sep 2024 08:18:07 +0000. Datasource DataSourceNoCloud [seed=/dev/mmcblk0p1][dsmode=local].  Up 84.83 seconds

If there are no errors in the log, everything is set up and ready.

Accessing the cluster from the local machine

To access the cluster from your local machine, you first need to have kubectl installed. It’s also recommended to use K9s, a great terminal-based tool for managing Kubernetes clusters.

K3s stores the kube-config file on the master node at /etc/rancher/k3s/k3s.yaml. Copy the contents of this file to your local machine’s .kube/config folder. If the config file doesn’t exist, create it.

To access the cluster, use kubectx to select the correct context. In K3s, the context is named default by default. To avoid conflicts and make it more relevant, it’s recommended to rename the context to something more appropriate. In K9s, you can switch contexts using the ctx command.

Having set everything correct you should see the following with kubectl

=>  kubectl get nodes                                                                                                                                                                                            
NAME                    STATUS   ROLES                  AGE    VERSION
ditc-k8s-master         Ready    control-plane,master   3m   v1.30.3+k3s1

Creating worker nodes

Creating worker nodes is also simple. We use again the K3s cli

- export MASTER_IP=10.54.42.101
- export WORKER_IP=10.54.42.xxx
- curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="agent --server=https://${MASTER_IP}:6443 --node-ip=${WORKER_IP} --node-external-ip=${WORKER_IP} --token 12345" sh -s -

The key difference here is the use of the agent option, which instructs K3s to configure this instance as a worker node. When adding this option, you must specify the master node’s IP address and the token used during the master node setup. Since we want the worker to have specified IP, we also set this value.

From this point forward, the process is identical to that of creating the master node. Once the setup is complete, you’ll see the worker node join the cluster. Simply repeat this process for any additional worker nodes you need to add to the cluster.

*Image 2.10 — Cluster with all nodes joined in K9s*

Creating nodes with specific K3s version

By default, when creating a new node, K3s uses the latest stable version. However, there may be cases where you need to create a node with a specific version, such as when the latest stable version is ahead of the version currently running in your cluster. To ensure consistency across your nodes, you can create the new node with the same version as the others. This is done by specifying the INSTALL_K3S_VERSION environmental variable when running the CLI command.

 - curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION="v1.29.2+k3s1" INSTALL_K3S_EXEC="agent --server=https://${MASTER_IP}:6443 --node-ip=${WORKER_IP} --node-external-ip=${WORKER_IP} --token 12345" sh -s

Ubuntu 24.04 and AppArmor (Kudos to Martín Montes for this tip)

Ubuntu 24.04 comes with AppArmor enabled by default. AppArmor is a Linux security module that restricts programs’ capabilities using per-program profiles. This also impacts the pods running on nodes with Ubuntu 24.04, requiring you to set specific profiles for AppArmor. In this guide, we won’t cover AppArmor, including its configuration or setup. Instead, we will disable AppArmor through cloud-init by adding the following under the write_files section:

 - path: /etc/sysctl.d/60-apparmor-namespace.conf
    content: |
      kernel.apparmor_restrict_unprivileged_userns=0

  - path: /boot/firmware/cmdline.txt
    content: |
      cgroup_enable=cpuset 
      cgroup_enable=memory 
      cgroup_memory=1 
      cgroup_enable=hugetlb 
      cgroup_enable=blki

Provisioning Cloud-native infrastructure components with OpenTofu

Note: All the relevant code for this sections is available on https://github.com/stathis-ditc/ppp-cloud/tree/main/tofu

OpenTofu has emerged as an open-source alternative to Terraform, developed in response to HashiCorp’s decision to change Terraform’s licensing model. Since its inception over a year ago, OpenTofu has grown independently as an official Linux Foundation project.

OpenTofu is a direct fork of Terraform, created after the HashiCorp licensing shift. It guarantees compatibility with existing Terraform configurations and modules for versions up to 1.5.7, but updates may be necessary for future Terraform versions. As OpenTofu continues to evolve, it’s essential to regularly consult its documentation and latest releases for updates on compatibility.

If you’re familiar with Terraform, transitioning to OpenTofu will be straightforward. This article will explore OpenTofu’s syntax and best practices based on the project’s official documentation.

Before getting started, ensure that the OpenTofu CLI is installed on your machine. Installation packages are available for all major operating systems. To run a command using the CLI, simply type tofu followed by the desired options. For help or guidance, you can run tofu -h to display the available commands and options.

Initialising a new OpenTofu project

Initialising a new OpenTofu project is straightforward. While there are no strict rules for structuring files and folders, it’s generally recommended to create a folder named tofu or tf and place all project files inside it. Best practices suggest including three files in this folder: main.tofu, variables.tofu, and outputs.tofu, even if they are initially empty. These filenames are used purely for organisational purposes and do not affect functionality. Additionally, any content in subfolders is ignored, though you can place code in subfolders as modules (more on this below).

The first step is to tell OpenTofu where our infrastructure, in this case our cluster, is located. To achieve this, we need to create a file named providers.tofu and add the following content:

What we’re doing here is adding the Kubernetes provider and configuring it to point to the location of our cluster. Since we’re running OpenTofu locally, we can specify the path to our kube-config file. Optionally, we can also provide the context of our cluster. This is especially useful (and often mandatory) when multiple contexts are present in the kube-config, ensuring that code changes are applied to the correct cluster. With this configuration, we have informed OpenTofu of the cluster’s location, and we’re ready to initialise the project.

To initialise the project run inside the tofu folder `tofu init` . If everything is ok, a new message will appear at the end of the execution of the command in green letters

*Image 2.12 — tofu init success message*

If there are errors, they will appear in red letters and usually will indicate what the problem is so as to be able to fix. Without a successful initialisation, we will not be apple to plan and apply any infrastructure changes

A successful initialisation will produce the following files and folders:

.terraform folder: This folder contains critical components related to the project’s state and backend configuration. While the contents are essential for running plan and apply commands, it’s not recommended to version it (e.g., pushing it to a Git repo). You can always safely delete it and recreate it by running tofu init, which will regenerate it based on the current state of the project.
terraform.lock.hcl: This file is created to manage provider versions, ensuring consistent environments during deployments. Like the .terraform folder, you can delete and recreate this file without risk, as it can be regenerated by running tofu init.
terraform.tfstate: This is the most important file in OpenTofu’s operation and its contents are commonly referred to as the state. It tracks the infrastructure resources that OpenTofu manages and stores the current state of your infrastructure. You must always keep this file safe and backed up. It’s recommended to version it in a GitHub repo or store it remotely, such as in a cloud provider bucket. Never delete this file, as losing it means losing all information about the infrastructure, forcing you to start from scratch. For a deeper understanding, consult the official documentation regarding this file.

Now we’re ready to begin creating our infrastructure. The tofu init command is necessary in various situations, such as when adding a new provider. If reinitialisation is required, OpenTofu will display an error message and prompt you to run the tofu init -upgrade command to reinitialise and update any necessary dependencies.

*Image 2.14 — tofu when needs upgrading initialization*

Plan and Apply

To generate a plan, we use the tofu plan command. This command produces a list of changes that would be applied if the plan is executed. The output is a “diff” between the code changes and the current state recorded in the terraform.tfstate file. It’s important to note that running tofu plan does not apply any changes to the infrastructure — it only shows what will happen if you proceed with the changes.

*Image 2.15 — Sample output when running plan*

If there are no changes, the output will be similar to this:

If an error occurs, the output will provide details about the issue, helping you understand what went wrong.

The tofu apply command performs two tasks. First, it runs the plan and conducts all the necessary checks. If the plan succeeds, it will prompt you to confirm the changes. To proceed, you must type yes; entering any other value will cancel the apply process.

*Image 2.18 — Apply plan with confirmation prompt*

A successful apply will look like this

Removing something from the code will trigger the destruction of the corresponding infrastructure resources.

*Image 2.20 — Successful destruction of a resource*

You can skip the confirmation prompt by using the -auto-approve flag, though it is generally not recommended. Additionally, it’s important to note that the apply command should never be interrupted or forcefully terminated, as doing so can corrupt the state file.

Structuring the project

Having covered the basics of OpenTofu, let’s define the structure of our project. As previously mentioned, OpenTofu operates at the root level, meaning it ignores files located in subfolders. While it’s possible to place all your code in a single file or multiple files in the root directory, this approach can lead to an unmaintainable codebase as the project grows.

Thankfully, OpenTofu supports modules, which are a way to organize and reuse infrastructure code. In simple terms, modules allow you to place code in subfolders and reference them as separate, reusable components within your main project. The structure of a module follows the same conventions as the root.

For our project, we will create our first module named bootstrap. To organize this, we’ll create two folders: first, a modules folder inside the tofu folder, and then a bootstrap folder within the modules folder. In our main.tofu file, we can reference the module like this:

By running now tofu plan and/or apply will execute also anything inside the bootstrap folder which will contain the initial necessary components for our cluster organized, structured and maintainable.

Installing Cilium

Let’s move on to provisioning our CNI for the cluster. For this, we’ll use Cilium as our CNI. Cilium is an open-source project providing advanced networking, security, and observability, specifically designed for cloud-native environments like Kubernetes. While it may have a steeper learning curve compared to other CNI solutions, it is well worth the effort due to its advanced capabilities. We’ve chosen Cilium for this project not only for its core features but also because it offers two key components we want to leverage: the Cilium Load Balancer and built-in GatewayAPI support. We will talk about and configure both on Part3: GitOps with ArgoCD: Install GatewayAPI, CiliumLB and Kube-Prometheus-Stack

The problematic helm provider

We will install Cilium using its Helm package, but we’ll do it through OpenTofu to ensure automation and reusability. Before we proceed, I’d like to share my experiences with the Helm provider. While there is a Helm provider available for Terraform, we won’t be using it here. In my experience, the Terraform Helm provider has proven to be unreliable. I’ve given it many chances, but it consistently turned out to be difficult to maintain, hard to debug, and in several cases, even when no errors were reported, it failed to properly install the Helm packages.

This is exactly what happened in this project. Initially, I decided to give the Helm provider another try and installed Cilium with it. Although there were no errors during the installation, the Cilium agents failed to become ready, and they kept crash looping in the cluster. After days of debugging without success, I tried installing the Helm package manually via the CLI, and everything worked perfectly. Once again, I abandoned the Helm provider for its unreliability and removed it from the project. Instead, I opted for the tried-and-true, old fashioned method of installing a Helm chart through code, which proved to be more reliable.

Creating and applying the Cilium Module

Inside the bootstrap folder we created earlier, we will add a new file named cilium.tofu and include the following content:

Let’s break down the code. We used the null_resource to execute Helm CLI commands that will add the Helm repo and install the chart. The null_resource is a special resource that doesn’t manage any real infrastructure but can be used to trigger actions based on other resources or execute specific tasks in your configuration. It’s often paired with the provisioner block to run local scripts, execute commands, or create dependencies between resources. Once added to the state, it won’t execute again unless it’s provided with triggers.

In our case, the null_resource is configured with triggers that force it to run when a specific value changes. For our Helm installation, we define two triggers: one for changes in the chart’s version and another for changes in the chart’s values file checksum (we’ll discuss in the next paragraph). To install the Helm chart, we use the upgrade — install option. This ensures that the chart will be installed if it hasn’t been installed yet and upgraded if any changes occur. If we had used only the install option, it would have worked for the initial installation, but any subsequent updates would have failed because the install command can’t re-install an already existing chart.

Now, let’s discuss the Helm values provided. To keep the code clean and maintain the structure, it’s a good practice to store Helm chart values in a separate file, retaining the original YAML format. To do this, we create the following directories and file inside the bootstrap folder: helm/cilium/values.yaml. You can find an example here: values.yaml. This file will contain the default values we want to override for our Helm chart.

To reference this file in our code, we use its absolute path within the project. The path.module variable refers to the absolute path of the directory containing the module’s code, and we append the relative path to where the values are stored. This allows us to provide custom values during the Helm installation process.

Additionally, we need to ensure the Helm chart is upgraded if any changes are made to the values.yaml file. To achieve this, we use the local_file data source to retrieve the file content by specifying its path. In the null_resource triggers, we then compute the checksum of the file. If the checksum differs from the one recorded in the state file, OpenTofu will recognize the change and trigger the Helm upgrade.

With everything in place, we are now ready to proceed. You can use the tofu CLI to plan and apply the configuration. If everything runs smoothly, Cilium will be successfully installed on the cluster, and you should see a result similar to the following:

*Image 2.21 — Cilium up and running in the cluster*

To learn more about Cilium, I highly recommend exploring the official documentation and labs. We will revisit Cilium in Part 3 when we will configure Load Balancers and GatewayAPI

Installing ArgoCD

For our GitOps solution, I chose to go with ArgoCD, as I have only had limited experience with it and wanted to explore it more deeply. ArgoCD is a declarative, GitOps-based continuous delivery tool for Kubernetes. It automates the deployment and management of applications in Kubernetes clusters by using Git repositories as the source of truth for application configurations. ArgoCD continuously monitors these repositories for changes and automatically applies updates to the Kubernetes environment, ensuring that the cluster state always aligns with the desired configuration stored in Git.

Creating and applying the ArgoCD Module

The process to install ArgoCD is similar to the one we used for Cilium. We create a new file in the bootstrap folder named argocd.tofu and add the following content:

We also create the relevant values folder `argocd/values` under the `helm` folder. In addition to installing Cilium, this setup also creates a new namespace where we will install ArgoCD.

Triggering the plan will install ArgoCD in our cluster and the result will be similar to this

*Image 2.22 — ArgoCD up and running in the cluster*

One of the major advantages of ArgoCD is that it comes with its own dashboard. To access the dashboard, we need to port forward to the argocd-server service on port 80. You can do this with kubectl by running:

kubectl port-forward -n argocd svc/argocd-server 8080:80

Alternatively, if you’re using K9s, you can press Shift+f on the argocd-server service or pod. Once the port is forwarded, open your browser and go to localhost:8080, which will take you to the ArgoCD dashboard login page. The default credentials are:

Username: admin
Password: The password is encoded in the Kubernetes secret argocd-initial-admin-secret, which is created during the chart’s installation. You can retrieve it using kubectl.

In Part 4: PiHole, Tailscale and Custom domains: Securely access from everywhere we will see how we will be able to access this dashboard with our own, custom urls

The first ArgoCD application

Note: Since I’m not covering the theory, I highly recommend reviewing the official ArgoCD documentation if you’re unfamiliar with the basics. Please ensure you understand the core concepts before proceeding.

It’s time to create our first application with ArgoCD. In Part 3, I’ll dive deeper into application installation with ArgoCD and the architecture and structure of GitOps in this project. For now, we will use OpenTofu once more to install the initial ArgoCD manifests. After this, all other components and packages, including Kubernetes cluster updates, will be managed through ArgoCD.

To begin, we will create a new module and include it in main.tofu.

You can find the implementation here:

We use depends_on to ensure this installation occurs after ArgoCD has been installed.

Once again, we will use null_resource, but in a slightly different way. This time, we will execute the kubectl apply command via a shell script.

To set this up, we’ll create a new folder named argocd inside the modules folder. Inside argocd, we’ll add two more folders: manifests and scripts.

Within the scripts folder, create a new file called kubectl-apply.sh with the following content:

The MANIFEST_PATH is an environment variable that we will populate through null_resource.

In main.tofu, we will define multiple null_resource resources to execute the kubectl-apply.sh script, passing the appropriate values through the environment variable. A sample implementation looks like this:

For a full example, visit: GitHub example.

In this setup, the command runs the kubectl-apply.sh script (ensure it has the appropriate executable permissions). We pass the absolute path of the manifest file into MANIFEST_PATH. This allows us to use kubectl to apply manifests through code execution.

Now that we’ve defined the OpenTofu code and how to install YAML files, let’s take a look at the ArgoCD manifests. We’ll create three manifests: an Application, a Project, and a Kubernetes Secret.

In this project, we’ll adopt the following architecture: we’ll create a root Application that will oversee all other Applications installed in the cluster, with those applications depending on it. This is because we want to configure everything from this point onward using a GitOps approach, without relying on OpenTofu. Essentially, we’re transferring control from OpenTofu to ArgoCD, where the root Application will manage everything. We’ll cover this in more detail in Part 3.

Application in ArgoCD:

An Application in ArgoCD represents a single deployment of resources. It tells ArgoCD where to find the Git repository or Helm chart containing the manifest files for that application.

Here’s an example of an Application manifest:

In this example, we instruct ArgoCD to pull the application configuration from a specific Git repository, under the config path in the main branch. We also specify that this Application belongs to the local-k8s project.

Project in ArgoCD:

A Project in ArgoCD is a higher-level construct that groups and manages multiple applications. It defines boundaries and policies, such as which repositories, clusters, or namespaces the applications can interact with.

Here’s an example of a Project manifest:

For an Application to access a Git repository, the repository must be listed in the Project’s source repositories. This ensures the Application has the necessary permissions to pull its configuration.

Kubernetes Secret:

Lastly, we need to create a Secret that allows the Application to authenticate and access the remote Git repository.

Here’s an example of the Secret manifest:

In this Secret:

We add the label argocd.argoproj.io/secret-type: repository.
The password contains a base64-encoded GitHub token, which gives access to the repository defined in the Application.
The project field is a base64-encoded string representing the Project’s name. This links the Project to the Secret, allowing ArgoCD to authenticate against the repository.
The url is the base64-encoded URL of the repository used in the Application.
The username field is base64-encoded and set to “x” (a common placeholder in Git credentials).

Ready to Apply:

With these manifests, the ArgoCD setup is ready. Use tofu cli to apply the manifests. Full code example for this module is available here https://github.com/stathis-ditc/ppp-cloud/tree/main/tofu/modules/argocd

When the plan finishes, we will be able to see the manifests created in our cluster.

*Image 2.24 — local-k8s project created*

*Image 2.24 — ArgoCD dashboard with our 2 first Application (details in part 3)*

Our ArgoCD setup is fully configured and ready to manage our cluster using a GitOps approach. OpenTofu will still be used for any changes to the infrastructure it has provisioned, ensuring a seamless integration between both tools.

Conclusion

In this part, we’ve successfully set up our Kubernetes cluster with K3s by installing Ubuntu on our Raspberry Pis. We then used OpenTofu to install the base components, Cilium and ArgoCD. Additionally, we created our first application in ArgoCD, setting the stage for our GitOps journey. In Part 3, we will use ArgoCD to configure GatewayAPI and the Cilium Load Balancer, and we’ll install our first Helm chart, kube-prometheus-stack, using ArgoCD. We’ll also demonstrate how to monitor the cluster with Grafana and Prometheus. Finally, we’ll explore how to update K3s through ArgoCD.