Containers - Not just boxes on a Ship

This article is about my personal history with application deployment starting from 2010. The evolution of deployment methods, along with my personal experiences and opinions, is shown. At the end, there is a small example of the inner workings of Docker.

PHP on a lonely Laptop​

Since my early years I was fascinated by hosting something which can be accessed by everyone on the internet. And when I finally started technical high school I acquired the knowhow necessary to set up an apache webserver and configure port-forwarding on the router. The web server was hosted on a 2005 Acer Aspire which was running an installation of the current Xubuntu. Apache was installed using the Ubunty Package Manager APT: apt update && apt install apache2

I was not using Git or FTP at this point, so everything hosted on the Acer Aspire was developed directly on it and backed up via a USB drive. Thinking back, I wish the high school teachers would have encouraged us to use Git. Nowadays, I use it for pretty much everything. Even though it was such a simple and stiff setup, it was the greatest thing for me, I loved it and I enjoyed going to school the next day to show my classmates the things I build. This setup was very simple, so to port the whole setup to a new machine was not a big deal. So was updating.

Five Admins and one Hetzner dedicated server​

A year or so passed, and one of my classmates decided to rent a dedicated root server from Hetzner and invited other interested people. I was one of the lucky ones.

We did not use any virtualization, so everything we installed was running on the dedicated server. We hosted SSH, Apache, Apache-PHP and some Minecraft servers. In the following years I played around with hosting Java servers (Tomcat), ASP.NET and other C# applications. There was no backup strategy, everything was pushed onto the server via SFTP. Every server user had its own thing running, so migration was no option.

Next to the Minecraft maps, there was nothing worth migrating anyway because most of it (at least my deployments) where just proofs of concept. I mention this, because portability and reliability was not an issue for me back than, because nobody, not even me, was really depending on any of the deployments, except maybe the minecraft servers (I was not the one maintaining those).

When we graduated from high school we stopped with the Hetzner dedicated server because everybody went their own way, and we sadly lost touch of each other.

Hello from the Cloud​

When I moved to Graz to start studying computer science at Technical University of Graz, I rented my own small cloud server from Hetzner. Different to all of my previous hosting endeavours, this time I wanted to have some production ready service which I can use to replace a bunch of SaaS service, since I was never a fan of vendor-lock-in.

I set up GitLab Community Edition for my GIT repositories, a ownCloud server to replace Dropbox, and a Web shell service to have a Linux shell available during working hours. (I was working at a Company that was using Windows at the time).

I used Apache with Let's Encrypt to TLS terminate the services. There was quite a bit of configuration and tinkering involved, until everything worked properly. Also, this time everything was set up using the systems package manager and the release deployments of the technologies mentioned above.

Why is there a PC under the bed?​

Around the year 2017 I became more security continuous and wanted to change the way I protect my personal data. I really wanted to own my own data. One of my measures was to move my server deployment from the Hetzner cloud to a local server located in my flat.

This was the first time when I was annoyed by the limited portability of my setup. What made it even worst: I did not document how I set up the last deployment on the Hetzner cloud. So this time I set up everything from scratch and imported all the user data manually into the services, because there were version differences and I had problems with the migrations.

During the setup I started documenting what I installed, what commands I executed, and the contents of the config files. In case I would lose the entire server due to a hard drive failure, I could recreate everything. On this server I added additional deployments like a DNS server, a DHCP server and a service which updates my external IP with a DynDNS provider. This was my first real migration from one setup to another, and I felt the pain of missing portability, version and environment mismatches and a lack of documentation.

Docker, Docker what is Docker?​

Around 2018, five years late to the party (Yes, the first release of docker was in March 2013!), I discovered the pleasures of docker. Docker is a container platform which allows entire applications to be packaged in their ideal environment with all dependencies and a stable configuration interface via environment variables.

When I discovered that there were containers for all the applications I was using at the time, my decision was to set everthing up again, but this time using containers. Because the documentation of my last server setup was not really usable any more due to the switch to docker, I decided to use Ansible to set up the server. I used one Docker container per deployment, even certbot and the reverse proxy were hosted via docker.

Unlike the other setups I used, I switched from Apache to nginx due to the structure of the config file. It was really nice to use containers for all of my setups, because I knew it would just work on the next Linux server with an active Docker runtime. Docker supports container networks which enables different containers to communicate with each other. All containers requiring a reverse proxy where connected to the nginx docker network. Event though Ansible was very slow and the very weak typed nature introduces bugs to my deployments the container management worked great.

Due to the ease and portability I added a bunch more deployments:

• A database server which could be used by my applications
• A monitoring stack with Prometheus, Grafana, etc...
• MQTT message brokers for my data aggregation platform
• Docker registry for my own docker containers

At this point I was pretty much hooked on containers, docker worked very well. I mounted all directories containing persistent data into a single directory, which can be backed up in one single job. So if I migrate the server, I only need to move the persistent data and the docker configuration.

Halfway to Kubernetes​

In the next few months, I started creating an ASP.NET service which was able to communicate with the Docker API to change the current running container to a newer version. I used this tool in the CI pipeline to automatically deploy the newly build images onto the server.

This ASP.NET deployment service featured a simple permission management, to restrict CI pipelines to a specific set of containers. In the next view iterations of this service I added the option to define container deployments via a JSON file in a declarative manner. Also, every time I added a new deployment (mosly new services of my data aggregation system) I needed to adjust the reverse proxy config to accommodate the new service.

The reason is simple, it not that hard to set up a kubernetes cluster, but it is hard to keep it updated! It is not impossible, but compared to the docker installation which just gets updated via apt it involves recurring manual tasks, especially if something breaks.

The moment everything changed was when I discovered K3S. K3s is a Kubernetes distribution, it comes prebundled with everything needed to run a Kubernetes cluster in a single standalone executable. Updating is as simple as replacing the k3s executable and restarting the Linux service (something which can be automated), a true no-brainer in my opinion. Almost immediately I started with the migration from Docker to k3s.

Containers everywhere, orchstrated, managed and resilient​

Today I run a home lab style server with full disc encryption using Linux and the KVM Hypervisor.

I have 2 virtual machines:

1. The first is my router VM which handles incoming VPN connections, routing and firewalling for all the home and VPN-networks (server, home, device, iot and guest-network)
2. The second VM is used for my K3S single node cluster.

I use this setup to have more control over the firewall. Kubernetes and also the Docker runtime use the Linux firewall to forward and/or restrict incoming communication. By itself this is not a problem, but some rules (especially forward rules) might interfere with custom rules. To have an extra layer of security, only the router VM can connect the K3S VM with outside networks. With this setup I have full control over every bit of incoming and outgoing traffic. Kubernetes with all amenities (ingress, loadbalance, cert-manager, configs, secrets...) makes deployments and monitoring a breeze.

Kubernetes deployments are super portable and reproduceable.

At RiKuWe we use customized tooling to make Kubernetes (and many other technologies) deployments even more automated, secure and predictable.

Containers: A technical excursion under the hood​

To end this article, we will rip of Dockers mask and look into its divine face. We are going to export a Debian container from Docker and mount it manually using Linux default tools. Everybody should be able to reproduce this (I use Arch btw.).

Pulling and Exporting the container image​

We use the Docker runtime to pull the image and export it into a tar file, this can also be done with other tools. Docker registries work via http protocol.

docker pull debian:trixie
docker image save debian:trixie -o /tmp/debian.tar

ls /tmp/debian.tar

# ╭───┬─────────────────┬──────┬──────────┬───────────────╮
# │ # │      name       │ type │   size   │   modified    │
# ├───┼─────────────────┼──────┼──────────┼───────────────┤
# │ 0 │ /tmp/debian.tar │ file │ 123.8 MB │ 2 minutes ago │
# ╰───┴─────────────────┴──────┴──────────┴───────────────╯

Unpacking and mounting the container image​

We unpack the saved container image into a directory on our machine.

mkdir /tmp/image /tmp/container
tar xf /tmp/debian.tar --directory /tmp/image
ls /tmp/image
# ╭───┬───────────────┬──────┬───────┬──────────────╮
# │ # │     name      │ type │ size  │   modified   │
# ├───┼───────────────┼──────┼───────┼──────────────┤
# │ 0 │ blobs         │ dir  │  60 B │ 3 weeks ago  │
# │ 1 │ index.json    │ file │ 362 B │ 55 years ago │
# │ 2 │ manifest.json │ file │ 459 B │ 55 years ago │
# │ 3 │ oci-layout    │ file │  31 B │ 55 years ago │
# │ 4 │ repositories  │ file │  89 B │ 55 years ago │
# ╰───┴───────────────┴──────┴───────┴──────────────╯

There is a file called manifest.json which contains information about the Debian layers and the order they are mounted.

cat /tmp/image/manifest.json
# [
#   {
#     "Config": "blobs/sha256/047bd8d819400f5dab52d40aa2c424109b6627b9e4ec27c113a308e72aac0e7b",
#     "RepoTags": [
#       "debian:trixie"
#     ],
#     "Layers": [
#       "blobs/sha256/cb9eb84282d037ad85b02cf671ef6ca766c43bc957f88b048d16dd6deb6e68b8"
#     ],
#     ...
#   }
# ]

In the manifest.json, there is only one layer. We extract it into the target root directory of the container.

tar xf ./image/blobs/sha256/cb9eb84282d037ad85b02cf671ef6ca766c43bc957f88b048d16dd6deb6e68b8 --directory /tmp/container
ls /tmp/container
# ╭────┬───────┬─────────┬────────┬────────────────╮
# │  # │ name  │  type   │  size  │    modified    │
# ├────┼───────┼─────────┼────────┼────────────────┤
# │  0 │ bin   │ symlink │    7 B │ 3 months ago   │
# │  1 │ boot  │ dir     │   40 B │ 3 months ago   │
# │  2 │ dev   │ dir     │   60 B │ 45 seconds ago │
# │  3 │ etc   │ dir     │ 1.2 kB │ 3 weeks ago    │
# │  4 │ home  │ dir     │   40 B │ 3 months ago   │
# │  5 │ lib   │ symlink │    7 B │ 3 months ago   │
# │  6 │ lib64 │ symlink │    9 B │ 3 months ago   │
# │  7 │ media │ dir     │   40 B │ 3 weeks ago    │
# │  8 │ mnt   │ dir     │   40 B │ 3 weeks ago    │
# │  9 │ opt   │ dir     │   40 B │ 3 weeks ago    │
# │ 10 │ proc  │ dir     │   40 B │ 3 weeks ago    │
# │ 11 │ root  │ dir     │  100 B │ a minute ago   │
# │ 12 │ run   │ dir     │   60 B │ 3 weeks ago    │
# │ 13 │ sbin  │ symlink │    8 B │ 3 months ago   │
# │ 14 │ srv   │ dir     │   40 B │ 3 weeks ago    │
# │ 15 │ sys   │ dir     │   40 B │ 3 months ago   │
# │ 16 │ tmp   │ dir     │   40 B │ 3 weeks ago    │
# │ 17 │ usr   │ dir     │  240 B │ 3 weeks ago    │
# │ 18 │ var   │ dir     │  260 B │ 3 weeks ago    │
# ╰────┴───────┴─────────┴────────┴────────────────╯

The layer we extracted seems to be the root file system of our container. So far so good, now our container needs to share some of the host systems directoryies. We use mount to mount them into the container. Those directories are used to provide the userspace applications with the required resources to access the hardware of the container. In this example I mounted the entire directories, so everything is mounted. In a real container only a subset of those directories is mounted. To enable networking in the container, docker creates virtual network devices which are mounted into the container.

sudo mount --bind /proc "/tmp/container/proc"
sudo mount --bind /sys "/tmp/container/sys"
sudo mount --bind /dev "/tmp/container/dev"
sudo mount --bind /run "/tmp/container/run"

Now it is time to start a shell within our docker container and test if it is working.

sudo chroot /tmp/container
apt update && apt install screenfetch
screenfetch

The chroot command tells the linux kernel to treat /tmp/container as the new root for the current shell session. We only update the package manager and install screenfetch a tool to print system stats.

root@node01:~# screenfetch
         _,met$$$$$gg.           root@test.container.rikuwe.com
      ,g$$$$$$$$$$$$$$$P.        OS: Debian
    ,g$$P""       """Y$$.".      Kernel: x86_64 Linux 6.16.4-arch1-1
   ,$$P'              `$$$.      Uptime: 3h 26m
  ',$$P       ,ggs.     `$$b:    Packages: 218
  `d$$'     ,$P"'   .    $$$     Shell: bash 5.2.37
   $$P      d$'     ,    $$P     Resolution: No X Server
   $$:      $$.   -    ,d$$'     WM: Not Found
   $$\;      Y$b._   _,d$P'      Disk: 0 / 32G (0%)
   Y$$.    `.`"Y$$$$P"'          CPU: AMD Ryzen AI 7 350 w/ Radeon 860M @ 16x 5.09091GHz
   `$$b      "-.__               GPU:
    `Y$$                         RAM: 10667MiB / 64082MiB
     `Y$$.
       `$$b.
         `Y$$b.
            `"Y$b._
                `""""

The fact that apt works and screenfetch thinks it was executed on a Debian machine tells us that our Debian container is functional.

I hope you enjoyed this little excursion into the realm of docker and linux container images. Have a good day and stay curious.