Like many other agencies, we use CI/CD in our projects. The principle is simple: Developers write code, push it to GitLab, and after each push, automated build and test jobs run—such as static code analysis, acceptance tests, etc.
As soon as a merge is made to the main branch or a tag is created, automatic deployments to test and production environments begin.
How has it gone so far?
For our CI jobs, we used virtual machines from Hetzner Cloud. The platform offers a good API, its own tools for GitLab CI, and is “Made in Europe.”
Our infrastructure consisted of three parts:
- A GitLab server to which the code is committed
- A control node that receives jobs and creates a short-lived Hetzner VM for each job
- The runner VMs, which run briefly for a job and are then cleaned up
To avoid the startup time when launching a VM, we kept a pool of VMs on standby during the day. The setup remained stable and scalable for a long time.
What went wrong?
As Hetzner’s popularity grew, so did the strain on its infrastructure. More and more CI jobs simply failed due to a lack of resources in Hetzner’s data centers—sometimes after waiting for several minutes, which was extremely frustrating.
This is no criticism of Hetzner—they were simply victims of their own success.
The performance limits of the cloud instances also became apparent: full-stack tests took up to 15 minutes, and some test scenarios crashed with OOM errors.
On top of that, Docker Machine—the software used to manage Hetzner VMs—is now deprecated and is set to be replaced by Fleeting, a new, plugin-based product developed by GitLab itself.
However, switching to Fleeting would not have solved our underlying problem—even the developers of the Hetzner plugin for Fleeting are struggling with the issue of limited resources.
What alternatives were there?
Public Cloud
For a short time, we considered switching to the public cloud—specifically GCP, Azure, or AWS. All three providers are officially supported by GitLab’s Fleeting architecture, have a mature API for VM management, and have been tried and tested by many other agencies.
Ultimately, however, we decided against it for the following reasons:
- Costs in the public cloud are opaque and likely higher than with our current Hetzner solution
- Even though VM provisioning in the public cloud might be more stable (larger resource pools), provisioning VMs remains a fundamentally slow process
- All three major providers are based in the U.S.—we prefer a provider within the EU
Kubernetes
As IT professionals, we know that almost any problem can be solved with Kubernetes 🙂
Still, we decided against it.
In our day-to-day work, we use a completely different stack. Our day-to-day operations revolve around FreeBSD and Linux servers—we use containers almost exclusively in local development environments or for specific edge cases.
Setting up a new Kubernetes-based CI infrastructure would have meant a steep learning curve, a significant time investment, and additional setup and maintenance effort.
Inspiration: TYPO3 Core Testing Infrastructure
Unsatisfied with the alternatives, we spoke with TYPO3 core developer Stefan Bürk about their testing infrastructure, how it came to be, and the challenges involved.
At first glance, their approach seems almost old-fashioned: instead of cloud VMs, TYPO3.org uses Runners on dedicated bare-metal servers.
The result: no wait times for VM provisioning and enough power to continuously run hundreds of CI tests.
Instead of Docker, they rely on Podman, an OCI runtime with key advantages:
- no daemon required, making nested setups (e.g., podman-in-podman) simpler—important for our full-stack tests
- runs by default as an unprivileged user—more secure, smaller attack surface
Infrastructure Migration
Inspired by TYPO3, we rented a dedicated AX102 server from Hetzner: 128 GB of RAM, 16 CPU cores.
Instead of a GitLab server + control node + cloud VMs, we now have: a GitLab server + a high-performance runner.
This offers immediate benefits:
- Caches are immediately available and do not need to be copied between the server and VMs
- Images are pulled once and are immediately available to all jobs
Technical Differences
TYPO3.org makes its Core Testing Infrastructure available to the community as a public service. Any contributor can use the CI infrastructure to test and validate their code.
This naturally requires a number of security measures:
- Only a limited number of trusted OCI images may be used
- The storage area for the OCI images is mounted as read-only in the runner—meaning existing images cannot be modified
In addition to these security aspects, there is a strong focus on a reproducible test workflow that can run both locally and in CI.
For this purpose, there is a standardized test script called ./runTests.sh, which can also be run on developers’ machines.
These measures work very well in the context of the TYPO3 testing infrastructure, since only a single software project (TYPO3) is tested there within clearly defined environments.
At punkt.de , however, we use our CI to build and test a wide variety of projects based on different technologies—including, of course, TYPO3, but also Neos, Keycloak, Sylius, and Ansible.
However, limiting ourselves to specific OCI images and using a uniform test script would, in our case, mean having to build in numerous exceptions and manually handle a large number of edge cases for the various projects and technologies.
Solution: GitLab CI/CD Components
As it happened, we had already migrated most of our CI pipelines to GitLab CI/CD Components.
Instead of writing a completely new pipeline for each project, this allows us to define standardized, reusable, and customizable building blocks that can be imported into any project.
Examples: We have our own CI components for Composer, npm, PHP-Stan, PHP-CS, etc.
As a result, the switch to Podman was essentially just a matter of adapting these components to Podman. This meant that only minimal changes were required within the individual projects themselves.
Challenges
Full-Stack Testing
Our full-stack tests run using Docker Compose. Although Podman offers `podman-compose`, it is not fully compatible.
Missing features: health checks and the --wait parameter. Solution: an additional `docker-compose.override.yml` file in which health checks and ` service_healthy` dependencies are disabled, e.g.,
services:
php:
depends_on: !reset null
postgres_schema_modify: !reset null
keycloak:
depends_on: !reset null
Since we couldn't rely on health checks, we turned to simpler methods, such as repeatedly sending `curl` requests to endpoints until they returned "ready."
Apart from that, we were able to migrate the tests to Podman with only minor adjustments.
Podman 4.x vs. 5.x
At the time ofpunkt de's setup, only Podman 4.x was available in the Ubuntu/Debian repositories—which had known bugs with DNS resolution. We had to hard-code domain names to avoid problems.
These bugs have been fixed in Podman 5.x, which, however, is only included in Ubuntu 25.04. Our solution: include packages from 25.04 and pin the Podman-dependent packages. Not ideal, but it works.
Results
The switch was worth it—the numbers speak for themselves:
- Next.js project (build, tests, push to external registry): 26 → 8 minutes
- Large Neos project (PHP & Node.js with full-stack tests): 20 → 3 minutes
- TYPO3 project (~600 full-stack tests with Codeception): 25 → 5 minutes
And most importantly: no more unstable jobs due to resource shortages.
Costs
Previously, we paid between €200 and €240 per month for cloud infrastructure.
Now: €104 for an AX102 Runner—with significantly more performance.
To this day, the machine is still far from reaching its limits.
Restrictions
Of course, this setup isn't perfect either. Things to keep in mind:
- No cloud-specific features (firewalls, load balancing, networking)
- A single runner = single point of failure (can be resolved by using multiple machines)
- Scaling only in larger increments (104 → 208 €, etc.)
- podman-compose is not fully compatible with docker-compose
- Major public cloud providers like Azure, AWS, and GCP are officially supported by GitLab and may be easier to set up
Advantages
On the other hand, there are clear advantages:
- no wait time due to VM provisioning
- half the price, significantly better performance
- theoretically also possible on-premises with your own hardware
- thanks to GitLab CI Components, only minimal pipeline adjustments are needed
Conclusion
After a few months of use, we’re convinced: The switch from cloud VMs to bare metal was absolutely worth it.
To help others make the switch more easily, we’ve published the Ansible role we used to set up our GitLab Runner. Give it a try, set up your own runners, and let us know what you think.
And if you’d also like to speed up your pipelines while reducing costs, we offer GitLab consulting— feel free to contact us for details.
Thanks to
- Stefan Bürk
- Christian “Lolli” Kuhn
