the Illumos eco system

After my openindiana server is already running for 4 months straight, I thought, I write a bit about the ecosystem of Illumos and its state.

Illumos ecosystem

Illumos is the base system of which every distribution uses. It’s more or less the base system, like FreeBSD. With Solaris 11 being the original OpenSolaris, Illumos is a fork of what was open source of OpenSolaris in 2010.

The development on Illumos is pretty active and at the moment, there is no merge with the Solaris code base planned. Oracle distributed code after the Solaris 11 release, but it was mostly code which had to be distributed either way. So there were no updates on kernel or ZFS code.

This has a huge impact on the future development of Illumos as everything has to be developed by contributors like Joyent, Nexenta and others. But it also has implications for most of the core features of Solaris, the most important ZFS. These are already noticeable with Solaris 11 having ZFS version 31 and FreeBSD and Illumos having version 28. This means, that neither FreeBSD nor Illumos can do something with a zpool created on a Solaris 11. This already makes a switch from one system to another difficult.

But nevertheless the contributors to Illumos work to make it better. The largest part at the moment is to get Illumos compiling with GCC 4.6.1. At the first look, it seems like a minor problem, but OpenSolaris was not written to be built with GCC but with the proprietary SunStudio. As far as I could see, this has some major implications and raised huge holes in the code, which has to get fixed. With that the base system is also upgraded from older versions of Perl and python, which also will be a longer process.

Another huge part is the process of building packages. Solaris 10 and older used the SVR4 format. That was pretty simple and looked like rpm. OpenSolaris introduced a new format named IPS - Image Packaging System. This is also compatible with the SVR4 format. OpenSolaris had a pretty big infrastructure for building IPS packages, but it was lost when oracle acquired sun and shut it down. The problem now is, how to build new packages. Some are using SVR4 to build the IPS packages, which works good and the repository already has a bunch of newer releases of many projects. Another attempt was to use pkgsrc. This is a project of NetBSD and already supports Solaris. This attempt died pretty fast. They were not used like FreeBSD ports and also not for compiling the packages. The third approach is to build a packing system on top of dpkg/apt. It is a collaboration between Nexenta, OpenIndiana and others. There is also a plan to build a new distribution out of it - named illumian.

One major difference between Solaris 11 and Illumos is that Illumos has KVM. It got ported from Linux by Joyent and works pretty good. With this step, Illumos not only had Zones for virtualization but also a full virtualization to get Linux running.

distribution ecosystem

There are a bunch of distributions out there, trying to solve different problems.

Solaris 11 - the first cloud os

Not so much a distribution of Illumos, but of the old OpenSolaris. Solaris 11 is a pretty good allround distribution. It is used from small systems to huge ones, running one application or some hundred on one machine. Some use it for storage and others to virtualize the hell out of it with zones and crossbow.

OpenIndiana - open source and enterprise

OpenIndiana was one of the first distributions using the Illumos core. It is available as a server distribution and a desktop one. The server one is targeted for the same usage as Solaris 11. As OpenIndiana uses Illumos it also has support for KVM and therefore can be used as a platform to host many fully virtualized instances on top of ZFS and crossbow infrastructure.

A problem at the moment is the pretty old software it offers. Most of the packages are from OpenSolaris and therefore nearly 2 years old. Most of them don’t even get security patches. The reason for that is the packaging topic mentioned above. As long as they don’t have a strategy, nothing will change here. The only option is to use the sfe repo at the moment.

This may change in the future, because of the joint effort with Nexenta of packaging releases.

OpenIndiana also has a desktop part which is targeted at ubuntu users wanting ZFS and time machine. As I used OpenSolaris already on a laptop I can only say “Yes, it works”. But you have to decide yourself, if you can live with pretty old but stable software. And many projects are not even available in package form, so that one would have to compile it yourself.

Nexenta - enterprise storage for everyone

Nexenta is another distribution who switched to Illumos core pretty fast. It is intended to be used for storage systems, but can also be used for other kinds of servers. It also uses the debian package system and a gnu userland. It is available as a community edition and “enterprise” edition.

The packages are a bit more up to date than the OpenIndiana ones. With the combined effort of both projects, they may keep closer to the actual releases.

illumian - illumos + debian package management

Illumian is a new project and collaboration work between Nexenta and OpenIndiana. It will provide packages through the debian package management dpkg/apt. The target audience seems to be the same as OpenIndiana. The plan at the moment is to release all packages in the same version as in OpenIndiana, so that the ultimate choice will just be, if you want to use dpkg or IPS.

SmartOS - the complete modern operating system

This is not so much a distribution as a live image. Its purpose is to use all disks in the server to create a zpool and use that to provide storage for virtual machines, be it zones or KVM instances. The KVM instances are also put into zones to attach dtrace to the virtual instances to see, what’s going on in that instance. SmartOS offers also pretty nice wrappers around the VM operating to get new instances up fast.

The company behind SmartOS is Joyent, more known for building node.js. They use SmartOS as the central pillar of their own JoyentCloud, where they host node.js applications, databases and also Linux machines.


OmniOS is a very new distribution and from OmniIT. It offers not much at the moment apart from an ISO image and a small wiki. It is intended to be used much like FreeBSD. They provide a very stripped down Illumos core with updated packages as far as possible and nothing more. Every other package one might need has to be built and distributed through a package repository. The reason behind this is, that they only want to provide the basic image, which everybody needs, but not the packages needed only by themselves. And even these packages may be one or two versions behind. And let me tell you - the packages they already updated may be considered bleeding edge by many debian stable users.

What next?

This was the excursion into the world of Illumos based distributions. I myself will switch away from OpenIndiana. It’s great, that Illumos lives and breathes more than 4 months ago, but there is much work left to do. SmartOS had a huge impact for me and others and Joyent and Nexenta do great work on improving the Ecosystem. But it will be hard to get back to the times where OpenSolaris was. Too much time went by unused. But I’m looking forward what else might come up of Illumos land.