A Personal,Private,Portable Cloud on Kubernetes and RaspberryPis

16 min readSep 27, 2024

Part 1: Requirements, Hardware and Network setup

Cloud technologies have emerged as powerful enablers, fundamentally shifting the landscape of data and application management. This is a series with the aim to delve into the realm of creating a personalized, private, own cloud infrastructure operating inside any home network, providing controlled accessibility from the outside. Utilizing Kubernetes and the versatile Raspberry Pis, the focus is on constructing a secure, wireless and modular private cloud tailored to specific requirements.

The project and the steps mentioned in the articles, are an experimentation and trials for over a year. Every guide is a polished version of how to achieve the desired result.

Before starting, a big shout out to my friend Martin Montes who gave me the inspiration to start this project in summer 2023. Martin has written a great article on how to create a home cluster. You can find it in https://itnext.io/securely-ingressing-into-bare-metal-kubernetes-clusters-with-gateway-api-and-tailscale-cc68299b646a.

Disclaimer: The technologies and hardware discussed in this series have been carefully studied and chosen based on their effectiveness in the outlined scenarios and presented without any sponsorship or endorsement. It is highly advisable to conduct thorough research and customize these implementations to align with unique requirements and preferences.

Articles in the series:

1. Requirements, Hardware and Network setup
2. Create Kubernetes Cluster with K3s and install Cilium and ArgoCD using OpenTofu.
3. GitOps with ArgoCD: Install GatewayAPI, CiliumLB and Kube-Prometheus-Stack (coming soon)
4. PiHole, Tailscale and Custom domains: Securely access from everywhere (coming soon)
5. NAS Storage on Kubernetes with NFS Provisioner in RAID 1 mode (coming soon)
6. Back to hardware — Cooling the cluster and monitor with telegraf (coming soon)
7. Private file hosting service with OwnCloud (coming soon)

Defining the Project

Initial Purpose:

The goal of this project is to create a private cloud that can store data from any device, anywhere. The cloud infrastructure will be based on a Kubernetes cluster, initially utilizing NAS storage, with future plans to migrate to a blob storage solution. This setup will also serve as a testing ground for experimenting with various cloud apps and operators before implementing them in larger projects. Additionally, the cluster will function as a smart home control center.

Key Requirements:

Portability:

The cluster should be compact enough to fit into small spaces, such as a bookshelf or similar constrained areas.
It must support wireless access, eliminating the need for Ethernet cables. While this may reduce connection speed, it will enhance the cluster’s modularity and ease of placement.
The power setup should be minimal, requiring as few power cables as possible. Ideally, a single power source should suffice for the entire cluster without needing additional power strips or multiple adapters.
Noise reduction is crucial. Since the cluster will be equipped with cooling fans, it should be placed in a rarely used area, like a storage room. Such rooms typically have limited access to power outlets, and Ethernet connections may be unavailable.

2. Network Isolation:

The cluster should be on a separate, isolated network from the main network while remaining accessible from the primary network when necessary.

3. Mobility:

The cluster should be easy to relocate. It must maintain connectivity to the network when needed, allowing access from outside, or operate locally when external connectivity is not required.

Architecture Design

Note: The whole list of equipment used is at the end of the article with their product names. In the whole article the equipment is referred by their scope and not by product name

Based on the project’s requirements, the following hardware setup was defined:

Raspberry Pi Cluster:

3x Raspberry Pi 4 (8GB RAM): Serving as one master node and two worker nodes.
1x Raspberry Pi 5 (8GB RAM): Later added as a third worker node.
All Raspberry Pis are powered through Ethernet using PoE (Power over Ethernet) HATs for each device.

2. PoE Switch:

The PoE switch serves a dual purpose:

Provides power to the Raspberry Pis via Ethernet.
Facilitates communication between the nodes in the cluster.

3. Wireless Nano Router:

The cluster is connected wirelessly via a nano router operating in shared network mode, offering the following functionality:

Gateway to the Cluster: The router manages network traffic and isolates the cluster within its own private network.
Access Control: Any device connected to the router can access the cluster’s network.
Bridge to Main Network: Since it operates in shared mode, the router can also connect to another network, such as the home network, allowing the cluster to be accessed externally if needed.

The nano router requires a 5V power input, which is drawn directly from a USB port on one of the Raspberry Pis, eliminating the need for a separate power adapter.

For network connectivity, the router is linked to the input of the PoE switch.

4. Storage Setup (NAS):

The cluster uses 3x 2TB SSDs to provide network-attached storage (NAS).

Two of the SSDs are configured in RAID 1 for redundancy.
The third SSD is used as an additional backup for the NAS.

Due to power constraints of the Raspberry Pis, the SSDs cannot be connected directly to the Pi USB ports during heavy operations (e.g., large file transfers), as the Pis may unmount them under load.

Solution: An external powered USB hub is used to connect the SSDs. The hub supplies sufficient power to the SSDs, which are then connected to the desired Raspberry Pi for mounting.

Next step was to visualize the above in an architectural diagram:

Before beginning the hardware build, let’s first cover the theory and configure the networking aspects of the cluster.

Network Architecture

The architecture is divided into two primary networks: the home network and the cluster network.

Home Network:

The home network operates on the 192.168.0.0/24 subnet, a typical configuration for many home setups.
This network includes various devices that connect to a central router, which serves as the gateway to the internet.
Devices connect to the home router either via Ethernet or wirelessly, enabling access to the broader internet.

2. Cluster Network:

The cluster network operates independently on the 10.54.42.0/24 subnet.
Similar to the home network, it has its own router that acts as the gateway for devices within the cluster.
However, unlike the home network, the cluster network is not directly connected to the internet.

How Does the Cluster Access the Outside World?

Since the cluster network is isolated from direct internet access, it relies on the home network for external connectivity. The cluster network’s router bridges the gap, allowing communication between the two networks, enabling the cluster to reach the outside world through the home router. But how is it possible to bridge the 2 routers? The answer is simple but it needs to have the right devices to be able to do the necessary configurations. Let’s see the following simple visualization

Each network requires its own router with distinct characteristics. The home network needs a robust, highly configurable router, offering more features than typical routers provided by internet service providers. For the cluster network, a more compact, minimal router is needed, capable of handling its dual role efficiently. This routers are commonly referred as nano routers.

To implement this network architecture, the following components and configurations must be possible:

Main Router:

Must support Wi-Fi connectivity.
Should allow for the definition of custom routes in its routing table to manage traffic between networks.

2. Nano Router:

Needs to function in dual roles:

Act as a device connected to the main network, obtaining an IP address from it.
Serve as the gateway for the cluster network, with a pre-configured gateway IP for managing traffic within the cluster.

The router should be capable of connecting wirelessly to the main router while providing an Ethernet connection for the PoE switch to connect to.

Router configuration

The first step is to assign static IP addresses to both router gateways in their LAN configurations. The main router will be set to IP 192.168.0.1, while the nano router will use IP 10.54.42.1

With both LANs configured, the next step is to set up the nano router. Nano routers offer several operating modes, including Hotspot, Repeater, Access Point, and Client, but only one mode can be active at a time. In this case, the router needs to operate in Client mode, which will create a wireless bridge between the two networks.

It is essential to configure the WAN settings of the nano router with a static IP to ensure it consistently retains the same IP address on the main network. This eliminates the need to repeatedly update the main router’s routing table whenever the nano router’s IP changes (see below). For our setup, we will assign the nano router a fixed IP of 192.168.0.150.

Additionally, since we are manually assigning a static IP to the nano router’s WAN, we must also specify the subnet mask and the gateway of the main network. In our case, these values are 255.255.255.0 for the subnet mask and 192.168.0.1 for Gateway, which is the IP of the main router set in the previous step, for the gateway. The primary dns server can be configured also to the gateway ip

Now,it is time to define the client settings order to establish the connection to the main router

If everything has been configured correctly, a bridge between the two routers should now be established. You can verify this by checking the main router’s client list to see if the nano router is connected.

*Image 1.4 — Nano router connected as client to main router*

To further verify connectivity, you can run a ping command from any device on the main network to the IP address 192.168.0.150, which was assigned to the nano router.

=> ping 192.168.0.150    
64 bytes from 192.168.0.150: icmp_seq=0 ttl=63 time=56.537 ms
64 bytes from 192.168.0.150: icmp_seq=1 ttl=63 time=3.311 ms
64 bytes from 192.168.0.150: icmp_seq=2 ttl=63 time=9.272 ms
64 bytes from 192.168.0.150: icmp_seq=3 ttl=63 time=88.823 ms
64 bytes from 192.168.0.150: icmp_seq=4 ttl=63 time=102.509 ms
--- 192.168.0.150 ping statistics ---
5 packets transmitted, 5 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 3.311/52.090/102.509/40.309 ms

Now, let’s assume there is a device in the cluster network with the IP address 10.54.42.101 (which we will configure on the master node later), and it is operational. We can then run a ping command to test connectivity to this IP.

=> ping 10.54.42.101   
PING 10.54.42.101 (10.54.42.101): 56 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1
Request timeout for icmp_seq 2
--- 10.54.42.1 ping statistics ---
4 packets transmitted, 0 packets received, 100.0% packet loss

We notice that the ping fails and cannot reach the IP address of the cluster network. To troubleshoot, let’s take a closer look at the routing table of the main router.

*Image 1.5 — Main router’s routing table*

Looking at the routing table, we can immediately see that the main router has not configured a route for the cluster network (10.54.42.0/24). As a result, any request to this IP range is being sent to the internet route by default, rather than being directed to the cluster network. To resolve this, it’s crucial that the main router supports the configuration of static routes. We need to set a static route where any request for the destination network 10.54.42.0/24 is routed through the nano router’s IP, which we configured as a client on the home network. In this case, that IP is 192.168.0.150.

The routing table is now updated and includes the cluster network as a destination, using the nano router’s IP as its gateway. This ensures that any traffic intended for the cluster network is correctly routed through the nano router.

We can also examine the routing table of the nano router. Here, we can observe how the routes are configured to establish proper communication between the networks, allowing the cluster’s network to access the internet through the main network.

*Image 1.8 — Nano router’s routing tables*

Running ping now on IP 10.54.42.101, succeeds

=>  ping 10.54.42.101   
PING 10.54.42.101 (10.54.42.101): 56 data bytes
64 bytes from 10.54.42.101: icmp_seq=0 ttl=62 time=4.091 ms
64 bytes from 10.54.42.101: icmp_seq=1 ttl=62 time=128.852 ms
64 bytes from 10.54.42.101: icmp_seq=2 ttl=62 time=20.806 ms
--- 10.54.42.101 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 4.091/51.250/128.852/55.296 ms

Additional Networking Considerations

With the network bridge now established, there are a few important factors to keep in mind.

First, the nano router should not be placed too close to the cluster cage, as electromagnetic interference from the Raspberry Pis can cause significant signal noise. This can be observed by running a ping command to any IP within the cluster. If the nano router is too close, the ping response times will be unusually high. However, you don’t need to move it very far — just a few centimeters can reduce interference. Use the ping command to monitor response times and adjust the router’s placement accordingly.

Secondly, if your network is equipped with a Wi-Fi mesh, there is no need to directly connect to the main router when within range. A Wi-Fi mesh operates on a different network but functions similarly to a network bridge, though in a wired mode. Devices connected to the mesh will still have access to the cluster network as long as the routing tables are properly updated.

With this topology, we can still access the cluster network because when a request is sent to the 10.54.42.0/24 range, the Wi-Fi mesh’s routing table does not find a match and forwards the request to the gateway, which is the main router. The main router then processes the request based on its own routing table.

An alternative approach would be to connect the nano router directly to the Wi-Fi mesh instead of the main router. However, in this case, the Wi-Fi mesh must support the configuration of custom routes, just as we did with the main router earlier.

Additionally, if you have a Wi-Fi mesh and the main router also provides Wi-Fi, you could configure the main router’s Wi-Fi SSID to be hidden and connect only the nano router to it. All other devices can then connect to the Wi-Fi mesh, simplifying network management.

Building the cluster

The initial project was designed with three Raspberry Pi 4 units, each with 8GB of RAM. One Raspberry Pi was designated as the master node, while the remaining two served as worker nodes. Later, the cluster was upgraded with the addition of a Raspberry Pi 5 (8GB) as an extra worker. At the time of the initial build, the Raspberry Pi 5 had not yet been released and was purchased much later.

The cluster cage

One of the initial challenges was deciding how to build the rack. While there are many solutions available on the market, I had specific requirements in mind:

I preferred a solution made from metal rather than entirely plastic or plexiglass.
Since dust would eventually become an issue, it was important that the components were not fully exposed and had some form of enclosure.
At the same time, proper ventilation was essential, as the Raspberry Pis would generate heat and require adequate airflow.
Additionally, the rack needed to accommodate more than just the Pis. The project also includes 3 SSDs, a PoE switch, a USB hub, and a nano router. Since the cluster was designed to be portable, it would be ideal if most or all of these components could be housed together in a single unit.

After considering these factors, I settled on the following solution:

The cage is specifically designed for Raspberry Pis and can accommodate up to four units. While it’s made for Raspberry Pi 4, the Raspberry Pi 5 also fits perfectly. It has room for four SSD drives and includes additional space at the bottom for devices like switches. The cage features plenty of ventilation holes, providing airflow while keeping the Pis and disks enclosed. Two 80mm fans can be mounted on the back for extra cooling. Although the cage comes with its own fans, they are of lower quality. We’ll discuss fan upgrades and cooling the cluster in more detail later in this guide, in Part 6: Back to Hardware — Cooling the Cluster and Monitoring with Telegraf.

Next, it’s time to mount the Raspberry Pis. Each Pi has its own dedicated mount, which holds the board on one side and an SSD drive on the other. The SSDs are mounted to the mounts holding the worker nodes, while the master node will be positioned first in line. Since the master node will be powered over Ethernet, each Pi is equipped with a PoE hat. Without the PoE hat, the Raspberry Pis cannot receive power through the Ethernet connection.

*Image 1.11 — RaspPi4 mounted to the holder without the PoE Hat attached*

*Image 1.12 — RaspPi4 mounted to the holder with the PoE Hat attached*

*Image 1.13 — RaspPi4 mounted to the holder with the PoE Hat attached (over angle)*

*Image 1.14 — SSD mounted on the other side of the mount, opposite of the board*

*Image 1.15 — Front side of holder, fully mounted*

Before putting the mounts in the cage, every board is equipped with a microSD card of 64GB, in which the OS will be installed. This takes place on the Part 2: Create Kubernetes Cluster with K3s and install Cilium and ArgoCD using OpenTofu.

With all the boards and disks mounted, I placed them inside the cage, which looked like this (the photo was taken before adding the Raspberry Pi 5 and the backup SSD).

*Image 1.16 — The cluster with 3 Pis and 2 SSDs*

Now it was the time for adding the Poe Switch, the nano router, the USB Hub and all the necessary cables. With everything in place, the cluster looks like this:

*Image 1.17 — The cluster with 4 Pis and 3 SSDs,PoE switch, USB Hub, NanoRouter and all cables attached*

The cluster hardware is now fully assembled and ready to be operational.

The PoE Switch

We haven’t discussed the PoE switch yet. The switch must have enough Ethernet ports to connect all the boards, as well as an input port to receive network connectivity. It’s important to note that there are two types of PoE switches: self-powered switches, which receive power from an external source, and switches that are powered through the input Ethernet port but this power can be given only by a self-powered switch. When using only one PoE switch, it must be self-powered, as it provides power to the entire system via PoE. If you decide to extend the network later with an additional PoE switch, that secondary switch doesn’t need to be self-powered and can receive power through another self-powered switch. When purchasing a PoE switch, be mindful of this distinction.

Powering the unit

With the architecture defined, the entire unit can be powered using just one power cable — this cable powers the PoE switch, which in turn provides power to the Raspberry Pi boards via the PoE ports. Since the Pis can’t handle the heavy load required for NAS capabilities, a USB-powered hub is also needed, requiring a second power cable. And that’s it — rather than needing seven separate power plugs (for the switch, nano router, USB hub, and four Raspberry Pi boards), the entire cluster operates with just two power plugs.

Hardware used in the project

3x RaspberryPis 4 8GB
1x RaspberryPi 5 8GB
3x PoE HAT (Power over Ethernet) with fan for Raspberry Pi 4B
1x Waveshare POE HAT (F) with fan for Raspberry Pi 5
4x MicroSD 64GB
2x WD Red 2 TB NAS SSD 2.5 Inch SATA
1x WD Blue SA510 SATA SSD 2 TB 2,5 inch
1x UCTRONICS Upgraded Full Case for Raspberry Pi Cluster
1x TP-Link TL-WR902AC AC750 WLAN Nano Router
1x TP-Link TL-SG105PE 5-Port Gigabit Managed LAN PoE Switch with 4 PoE+ Ports
2x Noctua NF-A8 PWM fans
1x atolla USB Hub with power supply, USB 3.0 Hub Active with 4 Ports, 1 Intelligenter Charging Port and 15W(5V/3A) power supply
3x USB3.0 to SATA cables
5 x 0.25 m CAT6 Network Cable

Conclusion

This project outlines the creation of a portable and efficient private cloud using a Kubernetes cluster built with Raspberry Pis, offering both NAS capabilities and a flexible playground for cloud applications. The architecture was carefully designed to meet specific requirements, such as portability, minimal power consumption, and wireless connectivity, while maintaining a high level of performance and modularity.

The hardware setup, including Raspberry Pis, SSDs, PoE switch, and nano router, was thoughtfully assembled to minimize cable clutter and power needs. By utilizing a PoE switch, the entire cluster is powered with only two power cables instead of the typical seven, greatly simplifying the power management of the system. The rack design not only houses the Pis and SSDs but also ensures proper cooling and protection from dust while keeping the unit compact and easy to move.

Networking was configured to provide seamless integration between the home and cluster networks, with routing tables and static IPs ensuring proper communication. The use of a nano router in client mode created a bridge between the two networks, while the routing setup allows easy access to the cluster from the main network.

In the end, this design successfully meets the goal of creating a compact, efficient, and scalable cluster that can be used for private cloud storage, smart home control, and as a testing ground for cloud applications. The cluster’s flexibility and portability make it a valuable tool for experimentation and practical use, all while being powered efficiently with minimal hardware and space requirements.

On Part 2 we see how to create a Kubernetes cluster with K3s as well as installing the basic components of our cluster, Cilium and ArgoCD using OpenTofu