poudriere in jails with zfs
There are tons of tutorials out there on how to get poudriere running in a jail. But most of them have in common, that they either miss options or have too many of them. So what I try with this, is to get the most condensed version of the whole process to get poudriere up and running, serving the generated packages.
In the following for 4 sections, we create a jail with the name poudriere1,
using partitions tank/jails/poudriere1
and tank/jails/ppoudriere1/data
. For
connectivity the jail will get the IP 192.168.1.10.
setting up zfs
The first step is to create both ZFS partitions on the host system with the following commands:
$ zfs create -o mountpoint=/jails/poudriere1 tank/jails/poudriere1
$ zfs create -o jailed=on tank/jails/poudriere1
The option jailed=on makes the partition completely available for the jail to manipulate, so that poudriere can create new partitions. This also makes it unavailable for the host system to mount as the jail can change the mountpoint.
system preparations
In addition to the ZFS partitions, we also need a separate network address to listen on, so that we can make the packages available using nginx.
For this, we define a new network interface lo1. This is then used by the jail to add its IP.
To make this work, place the following line in rc.conf:
cloned_interfaces="lo1"
and start the new interface with the command
service netif cloneup
Next the jail must be able to reach the internet. As I had pf already in place, I added the following NAT rule
nat on em0 inet from 192.168.1.0/24 to any -> (em0)
which redirects all traffic from 192.168.1.0/24 to the external interface.
Reload pf to make this change working
$ service pf reload
But to make this work, the host must also be told to do packet forwarding in the
rc.conf
gateway_enable="YES"
After that restart the routing service using
$ service routing restart
configuring the jail
The next step is to configure the jail, so that it can start and run poudriere.
This is done in /etc/jail.conf
. The following section shows the settings
needed and a short explanation. More can be looked up in main 8 jail
.
poudriere1 {
# first we set the permissions for the jail
# enforce_statfs allows the jail to get information about the mountpoint. With
# 1 it is able to see its root mountpoint and below. This is needed to be able
# to mount any file system.
enforce_statfs=1;
# This option allows the jail to mount file systems.
allow.mount;
# The following options enable mounting the specific file systems, needed to
# get poudriere running.
allow.mount.devfs;
# nullfs is used for remounting the ports tree into the child jails.
allow.mount.nullfs;
# tmpfs can be disabled when poudriere is told to not use it.
allow.mount.tmpfs;
allow.mount.procfs;
# This is needed to mount ZFS file systems.
allow.mount.zfs;
# As poudriere is using chflags, the jails needs to be allowed its usage.
allow.chflags;
# This option needs to be set, as poudriere grants that permission its jails.
allow.raw_sockets;
allow.socket_af;
allow.sysvipc;
# Allow this jail to run its own child childs up to the number.
children.max=20;
# Set a hostname visiable through jls.
host.hostname = "$name";
# Set the path to the jails root directory.
path = "/jails/$name";
# Automatically mount and unmount the dev file system, needed for ZFS and also
# used in poudriere jails.
mount.devfs;
# Set the IPs to use. 192.168.1.10 is handled automatically, localhost
# is reused from the host system.
ip4.addr=lo1|192.168.1.10, 127.0.0.1;
ip6.addr=::1;
# Boot up the jail at start using the RC system. This enables the use of rc.conf.
exec.start += "/bin/sh /etc/rc";
# After the jail is started, grant the ZFS partition to the jail, so that it
# can manage the work partition itself.
exec.poststart += "zfs jail $name rpool/jails/$name";
# On stopping the jail, go through the RC system.
exec.stop += "/bin/sh /etc/rc.shutdown";
# This option makes sure, that the jail is running without any environment
# variables set.
exec.clean;
}
installing and starting the jail
To install the jail we need to fetch a release, extract it into the jail root, make some last adjustments and then start it up.
To fetch a release the following command can be used (adjust the version to your need):
$ fetch ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/amd64/10.2-RELEASE/base.txz -o /tmp/base.txz
Then extract the base package into the root using the following command
$ tar -xf /tmp/base.txz -C /jails/poudriere1
Next the timezone has to be set, the resolv.conf copied and the hostname set.
$ echo 'hostname="poudriere1"' > /jails/poudriere1/etc/rc.conf
$ cp /etc/localtime /jails/poudriere1/etc/localtime
$ cp /etc/resolv.conf /jails/poudriere1/etc/resolv.conf
Instead of using the resolv.conf of the host system it would also be possible to use unbound on the host and use that as the DNS server in the jail.
With that done, we can now start the jail using the command:
$ jail -c poudriere1
If the command fails with the message that it can’t find /jails/poudriere1
,
check that the ZFS partition hasn’t jailed set to on and that it is mounted
in the correct place.
Accessing the jail using jexec it is possible to check if the basic setup works.
$ jexec poudriere1
root@poudriere1:/ # echo 'GET index.html' | nc zero-knowledge.org 80
root@poudriere1:/ # zfs list
If you do not get html output, check if the interface has the IPs defined and that you set up the routing correctly. If the ZFS partitions are missing, check if the permissions are set in /etc/jail.conf.
configuring poudriere
To install poudriere, build it from ports or install it through pkg.
root@poudriere1/ # portsnap fetch extract
root@poudriere1/ # cd /usr/ports/ports-mgmt/poudriere
root@poudriere1/ # make install clean
Before starting with setting up poudriere, we have to mount the work directory somewhere where we can actually use it:
root@poudriere1/ # zfs set mountpoint=/poudriere tank/jails/poudriere1/work
root@poudriere1/ # zfs mount tank/jails/poudriere1/work
Now we can set up poudriere the environment. Change the following settings in
/usr/local/etc/poudriere.conf
:
ZPOOL=tank
# relative to the zpool
ZROOTFS=/jails/poudriere1/work
BASEFS=/poudriere
# enable when you have set mount.allow.tmpfs in the jail.conf
USE_TMPFS=yes
# size in GB to allow for the ram drive
TMPFS_LIMIT=2
# set to no when you have the linux driver enabled
NOLINUX=yes
# set to no when you do not want to keep old versions of packages around
KEEP_OLD_PACKAGES=yes
KEEP_OLD_PACKAGES_COUNT=10
PRESERVE_TIMESTAMP=yes
BUILD_AS_NON_ROOT=yes
With that done, we can build the first jail for poudriere to work with. I mostly follow the FreeBSD handbook
root@poudriere1/ # # create a jail with the 10.2-RELEASE
root@poudriere1/ # poudriere jail -c -j 102amd64 -v 10.2-RELEASE
root@poudriere1/ # # list available jails
root@poudriere1/ # poudriere jail -l
If there is a problem with the jail creation, you can run the command using -x to get the debug output.
poudriere -x jail -c -j 102amd64 -v 10.2-RELEASE
If it happens that you get the error Unable to execute id(1) in jail.
a
permission is missing in /etc/jail.conf
.
To find out which is missing, check the debug output for the jail command. All
permissions are added on the command line, so it is easier to compare the
list of permissions with what poudriere wants to grant its jails.
Next we create the ports tree for poudriere to use:
root@poudriere1/ # # create a new ports tree
root@poudriere1/ # poudriere ports -c -p local
root@poudriere1/ # # list the installed port trees
root@poudriere1/ # poudriere ports -l
The next step is to create the list of packages poudriere should build into
/usr/local/etc/poudriere.d/base-pkglist
:
ports-mgmt/pkg
www/nginx
It is also possible to use sets, for example for build options. The next code
would be the make.conf for the base set, when placed in /usr/local/etc/base-make.conf
:
OPTIONS_UNSET=DOCS EXAMPLES X11 DOCBOOK NLS CUPS
DEFAULT_VERSIONS+=ssl=openssl
DEFAULT_VERSIONS+=pgsql=9.5
Using these files, the ports can be configured using the command:
root@poudriere1/ # poudriere options -j 102amd64 -p local -z base -f /usr/local/etc/poudriere.d/base-pkglist
To start a bulk run, which build all packages in the list, use the bulk command
root@poudriere1/ # poudriere bulk -j 102amd64 -p local -z base -f /usr/local/etc/poudriere.d/base-pkglist
You can find the created packages in the directory /poudriere/data/packages
.
configuration of nginx in the jail
The configuration of nginx in the jail is done in a moment.
For that nginx has to be installed. Using the freshly built packages us the following command (adjust the path according to your setup):
pkg install /poudriere/data/packages/102amd64-local-base/All/nginx-1.10.1.2.txz
After that you can configure nginx in the file /usr/local/etc/nginx/nginx.conf
.
The server configuration needs adjustment and nginx must be told where the data resides:
server {
listen 192.168.1.10:80;
server_name 192.168.1.10;
location / {
root /poudriere/data/packages;
autoindex on;
}
}
This will create an automatic index of the directory content and make it available for download. With this, it can be consumed by pkg on other systems.
If you also want to serve the logs, you can enable them with the following code
location /logs {
root /poudriere/data/logs/bulk;
autoindex on;
}
configuration of nginx outside of the jail
To forward incoming requests to the jail nginx instance, the following location option can be used:
location / {
proxy_pass http://192.168.1.10:80;
include proxy_params;
}
more information
This should help to get things up and running. If you need further information, please see the following man pages
There is also good documentation found on
There are also some tools to run jails, instead of making it raw like I did in this entry.