leaving FreeBSD for Archlinux

After close to 10 years running FreeBSD and trying to make it work for how I like to run my services, I am now finally giving up.

It mostly comes down to the way I want to run jails and how there is still no proper tooling for FreeBSD to make it work.

But before I get into more details, some history to make clear where I am coming from and why I want to run things how I want to do.

2008 - 2012 - the OpenSolaris Epoch

Back in 2008 I was introduced to OpenSolaris. It was a very stable OS with ZFS, zones, a virtual network stack, resource management and service management framework.

The tooling in many cases was just very intuitive to use and the man pages full of examples on how to use them.

Sadly Sun was bought by Oracle and OpenSolaris was killed near instantly. Forks were created in Illumos distributions, but the momentum was mostly lost.

Despite that I set up a server running OpenIndiana in 2010 for a project on a mixed used server and was running without issues for more than 2 years. Many applications were split into different zones, resources were pinned to different containers to make sure everything had room to breath, network was managed via the virtual network stack and flows to ensure ssh traffic was always possible.

2012 - 2022 - the FreeBSD era

In 2012 I was a bit lost, because Illumos almost ground to a halt regarding development and support from other open source projects. I liked every part of OpenSolaris/OpenIndiana, most of all its stability in every load I could throw at it. The only other distribution that offered even part of that stability was FreeBSD.

FreeBSD had jails, which I saw first around the same time I found OpenSolaris and it supported ZFS in a very fresh release.

So I started migrating all my stuff to a new server now running FreeBSD. In some cases it get very impressive tools like gpart, camcontrol, mfiutil and most of all pf.

But FreeBSD was missing a properly working virtual network stack and making jails work mostly consisted either of using ezjail or completely manual management via jail.conf. I fiddled around with jails to make it even a bit how I used zones before but to no avail.

I left the services running directly on the root system for some years and tried again multiple times in the last 10 years. Some things changed, e.g. multiple jail management tools were created, but none was quite as close to what zonecfg/zoneadm could do. The virtual network stack evolved but even in 2020 I still had crashes when using epair devices.

At one point I even set up a huge virtual network lab with virtual routers and failover, BGP, OSPF and the works. At that time I wrote carp_failover, a small tool to make carp failovers work with jails with only epair devices.

But after close to 10 years of waiting, something else happend - systemd.

2012 till now - systemd getting better

In 2012 Archlinux was migrated to systemd. Since then systemd gained a lot of things. The most important for me though is systemd-networkd and systemd-machined.

Both these components make network and container management much easier to use on linux. Both of these tools need config files and then do what is told them to do.

The most interesting part is, that it is pretty close in how zone and network management worked in OpenSolaris (not from the actual config file format, but how the components interact).

It took systemd a long time, but it is now at a point where the tooling is close to being on par with the OpenSolaris tools I knew.

Now FreeBSD also has a very good init system in rc. Sadly FreeBSD didn’t recognize systemd as a hint, that other tooling apart from the init system is also important.

2022 - what now?

Now in 2022, after trying for 3 years to migrate my FreeBSD and porting it to a setup similar to my OpenIndiana setup, I am giving up. FreeBSD doesn’t offer the tooling that I need to make this work.

I can’t run jails with resource limits, configured in multiple text files with the network architecture I want without investing massive amounts of time.

What I decided to do was to go with Archlinux. For one it provides the software I need without patches as best they can and update in a very timely manner.

With OpenZFS providing ZFS modules and systemd providing the container tooling, there is just nothing left on FreeBSD that could make me stay and invest more time. My private live just doesn’t allow that anymore.

Will I miss anything from FreeBSD? Yes, pf. Probably the best firewall tool that I have ever used. The same is true for gpart which, at one point, I even used to rescue one of my linux systems. Both will stay in my memory in the same happy place as zoneadm and dladm and many other OpenSolaris tools.

Now I will start to move all my stuff onto my Archlinux server, which already contains a couple containers, nicely separated and secured via systemd-machined.

I systemd without flaws? No, but the tooling I require and need to make my stuff work just works.

My old FreeBSD server now has an uptime of 504 days and I am afraid to reboot it. My new server has an uptime of 2 days and is running Archlinux for half a year.

Thank you FreeBSD for all your hard work. I will remember you.