poudriere in jails with zfs

There are tons of tutorials out there on how to get poudriere running in a jail. But most of them have in common, that they either miss options or have too many of them. So what I try with this, is to get the most condensed version of the whole process to get poudriere up and running, serving the generated packages.

In the following for 4 sections, we create a jail with the name poudriere1, using partitions tank/jails/poudriere1 and tank/jails/ppoudriere1/data. For connectivity the jail will get the IP 192.168.1.10.

setting up zfs

The first step is to create both ZFS partitions on the host system with the following commands:

$ zfs create -o mountpoint=/jails/poudriere1 tank/jails/poudriere1
$ zfs create -o jailed=on tank/jails/poudriere1

The option jailed=on makes the partition completely available for the jail to manipulate, so that poudriere can create new partitions. This also makes it unavailable for the host system to mount as the jail can change the mountpoint.

system preparations

In addition to the ZFS partitions, we also need a separate network address to listen on, so that we can make the packages available using nginx.

For this, we define a new network interface lo1. This is then used by the jail to add its IP.

To make this work, place the following line in rc.conf:

cloned_interfaces="lo1"

and start the new interface with the command

service netif cloneup

Next the jail must be able to reach the internet. As I had pf already in place, I added the following NAT rule

nat on em0 inet from 192.168.1.0/24 to any -> (em0)

which redirects all traffic from 192.168.1.0/24 to the external interface.

Reload pf to make this change working

$ service pf reload

But to make this work, the host must also be told to do packet forwarding in the rc.conf

gateway_enable="YES"

After that restart the routing service using

$ service routing restart

configuring the jail

The next step is to configure the jail, so that it can start and run poudriere. This is done in /etc/jail.conf. The following section shows the settings needed and a short explanation. More can be looked up in main 8 jail.

poudriere1 {
  # first we set the permissions for the jail
  # enforce_statfs allows the jail to get information about the mountpoint. With
  # 1 it is able to see its root mountpoint and below. This is needed to be able
  # to mount any file system.
  enforce_statfs=1;
  # This option allows the jail to mount file systems.
  allow.mount;
  # The following options enable mounting the specific file systems, needed to
  # get poudriere running.
  allow.mount.devfs;
  # nullfs is used for remounting the ports tree into the child jails.
  allow.mount.nullfs;
  # tmpfs can be disabled when poudriere is told to not use it.
  allow.mount.tmpfs;
  allow.mount.procfs;
  # This is needed to mount ZFS file systems.
  allow.mount.zfs;
  # As poudriere is using chflags, the jails needs to be allowed its usage.
  allow.chflags;
  # This option needs to be set, as poudriere grants that permission its jails.
  allow.raw_sockets;
  allow.socket_af;
  allow.sysvipc;
  # Allow this jail to run its own child childs up to the number.
  children.max=20;

  # Set a hostname visiable through jls.
  host.hostname = "$name";
  # Set the path to the jails root directory.
  path = "/jails/$name";

  # Automatically mount and unmount the dev file system, needed for ZFS and also
  # used in poudriere jails.
  mount.devfs;
  # Set the IPs to use. 192.168.1.10 is handled automatically, localhost
  # is reused from the host system. 
  ip4.addr=lo1|192.168.1.10, 127.0.0.1;
  ip6.addr=::1;

  # Boot up the jail at start using the RC system. This enables the use of rc.conf.
  exec.start += "/bin/sh /etc/rc";
  # After the jail is started, grant the ZFS partition to the jail, so that it
  # can manage the work partition itself.
  exec.poststart += "zfs jail $name rpool/jails/$name";
  # On stopping the jail, go through the RC system.
  exec.stop += "/bin/sh /etc/rc.shutdown";
  # This option makes sure, that the jail is running without any environment
  # variables set.
  exec.clean;
}

installing and starting the jail

To install the jail we need to fetch a release, extract it into the jail root, make some last adjustments and then start it up.

To fetch a release the following command can be used (adjust the version to your need):

$ fetch ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/amd64/10.2-RELEASE/base.txz -o /tmp/base.txz

Then extract the base package into the root using the following command

$ tar -xf /tmp/base.txz -C /jails/poudriere1

Next the timezone has to be set, the resolv.conf copied and the hostname set.

$ echo 'hostname="poudriere1"' > /jails/poudriere1/etc/rc.conf
$ cp /etc/localtime /jails/poudriere1/etc/localtime
$ cp /etc/resolv.conf /jails/poudriere1/etc/resolv.conf

Instead of using the resolv.conf of the host system it would also be possible to use unbound on the host and use that as the DNS server in the jail.

With that done, we can now start the jail using the command:

$ jail -c poudriere1

If the command fails with the message that it can’t find /jails/poudriere1, check that the ZFS partition hasn’t jailed set to on and that it is mounted in the correct place.

Accessing the jail using jexec it is possible to check if the basic setup works.

$ jexec poudriere1
root@poudriere1:/ # echo 'GET index.html' | nc zero-knowledge.org 80
root@poudriere1:/ # zfs list

If you do not get html output, check if the interface has the IPs defined and that you set up the routing correctly. If the ZFS partitions are missing, check if the permissions are set in /etc/jail.conf.

configuring poudriere

To install poudriere, build it from ports or install it through pkg.

root@poudriere1/ # portsnap fetch extract
root@poudriere1/ # cd /usr/ports/ports-mgmt/poudriere
root@poudriere1/ # make install clean

Before starting with setting up poudriere, we have to mount the work directory somewhere where we can actually use it:

root@poudriere1/ # zfs set mountpoint=/poudriere tank/jails/poudriere1/work
root@poudriere1/ # zfs mount tank/jails/poudriere1/work

Now we can set up poudriere the environment. Change the following settings in /usr/local/etc/poudriere.conf:

ZPOOL=tank
# relative to the zpool
ZROOTFS=/jails/poudriere1/work
BASEFS=/poudriere
# enable when you have set mount.allow.tmpfs in the jail.conf
USE_TMPFS=yes
# size in GB to allow for the ram drive
TMPFS_LIMIT=2
# set to no when you have the linux driver enabled
NOLINUX=yes
# set to no when you do not want to keep old versions of packages around
KEEP_OLD_PACKAGES=yes
KEEP_OLD_PACKAGES_COUNT=10
PRESERVE_TIMESTAMP=yes
BUILD_AS_NON_ROOT=yes

With that done, we can build the first jail for poudriere to work with. I mostly follow the FreeBSD handbook

root@poudriere1/ # # create a jail with the 10.2-RELEASE
root@poudriere1/ # poudriere jail -c -j 102amd64 -v 10.2-RELEASE
root@poudriere1/ # # list available jails
root@poudriere1/ # poudriere jail -l

If there is a problem with the jail creation, you can run the command using -x to get the debug output.

poudriere -x jail -c -j 102amd64 -v 10.2-RELEASE

If it happens that you get the error Unable to execute id(1) in jail. a permission is missing in /etc/jail.conf. To find out which is missing, check the debug output for the jail command. All permissions are added on the command line, so it is easier to compare the list of permissions with what poudriere wants to grant its jails.

Next we create the ports tree for poudriere to use:

root@poudriere1/ # # create a new ports tree
root@poudriere1/ # poudriere ports -c -p local
root@poudriere1/ # # list the installed port trees
root@poudriere1/ # poudriere ports -l

The next step is to create the list of packages poudriere should build into /usr/local/etc/poudriere.d/base-pkglist:

ports-mgmt/pkg
www/nginx

It is also possible to use sets, for example for build options. The next code would be the make.conf for the base set, when placed in /usr/local/etc/base-make.conf:

OPTIONS_UNSET=DOCS EXAMPLES X11 DOCBOOK NLS CUPS
DEFAULT_VERSIONS+=ssl=openssl
DEFAULT_VERSIONS+=pgsql=9.5

Using these files, the ports can be configured using the command:

root@poudriere1/ # poudriere options -j 102amd64 -p local -z base -f /usr/local/etc/poudriere.d/base-pkglist

To start a bulk run, which build all packages in the list, use the bulk command

root@poudriere1/ # poudriere bulk -j 102amd64 -p local -z base -f /usr/local/etc/poudriere.d/base-pkglist

You can find the created packages in the directory /poudriere/data/packages.

configuration of nginx in the jail

The configuration of nginx in the jail is done in a moment.

For that nginx has to be installed. Using the freshly built packages us the following command (adjust the path according to your setup):

pkg install /poudriere/data/packages/102amd64-local-base/All/nginx-1.10.1.2.txz

After that you can configure nginx in the file /usr/local/etc/nginx/nginx.conf.

The server configuration needs adjustment and nginx must be told where the data resides:

server {
  listen      192.168.1.10:80;
  server_name 192.168.1.10;

  location / {
    root      /poudriere/data/packages;
    autoindex on;
  }
}

This will create an automatic index of the directory content and make it available for download. With this, it can be consumed by pkg on other systems.

If you also want to serve the logs, you can enable them with the following code

location /logs {
  root      /poudriere/data/logs/bulk;
  autoindex on;
}

configuration of nginx outside of the jail

To forward incoming requests to the jail nginx instance, the following location option can be used:

location / {
  proxy_pass http://192.168.1.10:80;
  include proxy_params;
}

more information

This should help to get things up and running. If you need further information, please see the following man pages

• man jail
• man jail.conf
• man poudriere
• man zfs

There is also good documentation found on

• FreeBSD handbook ports section
• FreeBSD handbook jails section
• nginx

There are also some tools to run jails, instead of making it raw like I did in this entry.

• cbsb
• ezjail
• iocage (unsupported from 10.3 onwards)