When you want to start with cfengine, it is not exactly obvious how some stuff works. To make it easier for others, I will write about some stuff I find out in the process.

For the start, here is the first thing I found out. By default cfengine logs to files in the work directory. This can get a bit ugly, when the agent is running every 5min. As I use cf-execd, I added the option executorfacility to the exed section.

body executor control {
  executorfacility => "LOG_LOCAL7";
}

After that a restart of execd will result in logs appearing through syslog.

The new blog is finally online. It took us nearly more than a year to finally get the new design done.

First we replaced thin with puma. Thin was getting more and more a bother and didn’t really work reliable anymore. Because of the software needed, it was pinned to a specific version of rack, thin, rubinius and some other stuff. Changing one dependency meant a lot of working getting it going again. Puma together with rubinius make a pretty nice stack and in all the time it worked pretty well. We will see, how good it can handle running longer than some hours.

The next part we did was throw out sinatra and replace it with zero, our own toolkit for building small web applications. But instead of building yet another object spawning machine, we tried something different. The new blog uses a chain of functions to process a request into a response. This has the advantage that the number of objects kept around for the livetime of a request is minimized, the stack level is smaller and in all it should now need much less memory to process a request. From the numbers, things are looking good, but we will see how it will behave in the future.

On the frontend part we minimized the layout further, but found some nice functionality. It is now possible to view one post after another through the same pagination mechanism. This should make a nice experience when reading more a number of posts one after another.

We hope you like the new design and will enjoy reading our stuff in the future too.

Postfix’ policy system is a bit confusing. There are so many knobs to avoid receiving mails which do not belong to any account on the system and most of them check multiple things at once, which makes building restrictions a bit of a gamble.

After I finally enabled the security reports in freebsd the amount of mails in the mailqueue hit me. After some further investigation I found even error messages of dspam, having trouble to rate spam for receivers which were not even in the system.

To fix it, I read into the postfix documentation again, build new and hopefully better restrictions. The result was even more spam getting through. After a day went by and my head was relaxed I read the documentation again and found the following in the postfix manual

The virtual_mailbox_maps parameter specifies the lookup table with all valid recipient addresses. The lookup result value is ignored by Postfix.

So instead of one of the many restrictions a completely unrelated parameter is responsible for blocking mails for unknown users. Another parameter related is smtpd_reject_unlisted_recipient. This is the only other place I could find, which listed virtual_mailbox_maps and I only found it when looking for links for this blog entry.

So if you ever have problems with receiving mails for unknown users, check smtpd_reject_unlistef_recipient and virtual_mailbox_maps.

As I was setting up a firewall on my freebsd server I had to choose between one of the three firewalls available.

There is the freebsd developed firewall ipfw, the older filter ipf and the openbsd developed pf. As for features they have all their advantages and disadvantages. Best is to read firewall documentation of freebsd.

In the end my decision was to use pf for one reason - it can check the syntax before it is running any command. This was very important for me, as I’m not able to get direct access to the server easily.

ipf and ipfw both get initialized by a series of shell commands. That means the firewall controll program gets called by a series of commands. Is one command failing, the script may fail and the firewall ends up in a state undefined by the script. You may not even get into the server by ssh anymore and needs a reboot.

This is less of a problem with pf, as it does a syntax check on the configuration beforehand. It is not possible to throw pf into an undefined state because of a typo. So the only option left would be to forget ssh access or anything else.

I found the syntax of pf a bit weird, but I got a working firewall up and running which seems to work pretty well. ipfw looks similar, so maybe I try it the next time.

After more than a year working on my mail setup, I think I have it running in a pretty good way. As some of the stuff is not documented at all in the wide of the internet, I will post parts here to make it accessible to others.

Many setups use the MTA (postfix, exim) to store mails on the filesystem. My setup lets dovecot take care of that. That way it is the only process able to change data on the filesystem.

to make this work, we first need a lmtp socket opened by dovecot. The configuration part looks like this

service lmtp {
  unix_listener /var/spool/postfix/private/delivery.sock {
    mode = 0600
    user = postfix
    group = postfix
  }
}

LMTP is a lightweight smtp protocol and most mail server components can speak it.

Next we need to tell postfix to send mails to this socket instead storing it on the filesystem. This can be done with with the following setting

mailbox_transport = lmtp:unix:/var/spool/postfix/private/delivery.sock

or for virtal accounts with

virtual_transport = lmtp:unix:/var/spool/postfix/private/delivery.sock

Now postfix will use the socket to deliver the mails.

It is also possible to use other services between these two like dspam. In my case postfix delivers the mails to dspam and that will deliver them to dovecot.

For dovecot change the path of the socket to something dspam can reach. I’m using /var/run/delivery.sock.

Then change the dspam.conf to use that socket as a delivery host

DeliveryProto LMTP
DeliveryHost  "/var/run/delivery.sock"

As postfix needs to speak to dspam, we set dspam to create a socket too

ServerMode auto
ServerDomainSocketPath "/var/run/dspam.sock"

ServerMode should be set to either auto or standard.

Now the only thing left to do is to tell postfix to use that socket to deliver its mails. For that, set the options from before to the new socket

virtual_transport = lmtp:unix:/var/run/dspam.sock

And with that, we have a nice setup where only dovecot stores mails.

This weekend I had a small problem with my omnios installation. The installation is now more than a year old and back then, the feature flags for zfs were really fresh. So as time went on, zfs got better but somehow it was missed to update the grub installation. When I then booted my server on friday, I did not get up again as grub was unable to load my zpool.

Thanks to Rich Lowe from illumos project the bug got fixed. But I had to somehow run installgrub on my system to get it up again. For that to work, I had to find a system which either boots a current illumos kernel or lets me enter a kernel paramter.

To make things more complicated, the current omnios installation medium did not like my system and just rebooted with a kernel panic without me having a chance to get my kernel paramter from the zpool. FreeBSD 9.1 was not able to read the zpool because of the feature flags. After half a day of try and error I just went with SmartOS and lo and behold - it worked!

So I imported my zpool with zpool import -o /mnt rpool, read my menu.lst to get the parameter and restarted with omnios to enter that command to the kernel in grub and I got a live environment up to do installgrub!

For me, the parameter in question is -B acpi-user-options=0x2,$ZFS-BOOTFS to switch off ACPI on my Sandy Bridge Celeron system.

A very nice feature on Solaris was the possibility to initialize new zones with a sysidcfg file. This does not exist on omnios. With kayak, omnitis deployment server, a way to run postboot scripts was created. The way is the file /.initialboot. This is just a shell script which gets executed on the first boot and gets removed afterwards. Nothing much but already very useful to make the initial setup for dns and the ip.

A little example. I have a zone foo1 with a vnic barnic1. I want to setup dns and dhcp for the interface. The zone is installed in /zones/foo1/ so we have to root file system mounted as /zones/foo1/root.

We create the /.initialboot file (which ends up as /zones/foo1/root/.initialboot with the following content

ipadm create-if foonic1
cp /etc/nsswitch.dns /etc/nsswitch.conf
ipadm create-addr -T dhcp foonic1/ipv4
echo "nameserver 192.168.56.1" >> /etc/resolv.conf

Now boot the zone and after 5minutes or so everything is setup and ready to go. Makes it really easy.

I had the need to filter logs from different programs into different places - in this case the postgres and nginx logs. The man page of syslog.conf describes it pretty good, but misses some examples to make it more clear. So here is how I configured it, to make it easier.

First, I edited the syslog.conf

# filter everything apart from postgres and nginx
!-postgres,nginx
*.err;kern.warning;auth.notice;mail.crit    /dev/console
# and all the other stuff

# filter only postgres
!postgres
*.*        /var/log/postgresql.log

# filter only nginx
!nginx
*.*        /var/log/nginx.log

The next step is to setup the log rotate. This happens in /etc/newsyslog.conf. The man page is very helpful, so if you want to adjust something, take a peek into it.

# postgresql                                                                                                                                                                                  
/var/log/postgresql.log     640  5     100  * JC                                                                                                                                              
                                                                                                                                                                                          
# nginx                                                                                                                                                                                       
/var/log/nginx.log      640  5     100  * JC

And that is all. If you want to add more program filtes, you have to define them in the syslog.conf as notfilter and filter and add the rotate to newsyslog.conf.

I got an interesting question regarding zones on Solaris in #omnios.

scarcry: Does anyone know how to move a zone from one zpool to another?

There are some guides out there on how to move a zone from one machine to another, but most of them install the zone in the same place as before.

But instead of moving it from one machine to another, this small guide will just show what to do, when only the location is chaning.

preparations

First, we need to setup the partitions and zones for our little experiment. For this example, I will use the pool rpool and the following partitions

• rpool/zones/old mounted to /zones/old/
• rpool/zones/new mounted to /zones/new/

We also need the zone config, so here is it.

create -b
set zonepath=/zones/old/zone1
set ip-type=exclusive
set autoboot=false
add net
set physical=zone1
end
commit

Just install the zone with the normal commands

$ zonecfg -z zone1 < zone.config
$ zoneadm -z zone1 install
$ zoneadm -z zone1 boot

Check if the zone is running and write a file, just to make sure, we have the same zone at the end.

moving the zone

For this guide, we will assume, that the zone is in production use and can’t be offline too long. For that to work, we will do a first snapshot, when the zone is still running.

$ zfs snapshot -r rpool/zones/old/zone1@move1

After that, we can replay that snapshot into the new location.

$ zfs send -R rpool/zones/old/zone1@move1 | zfs recv rpool/zones/new/zone1

This step will take some time, depending on the size of your zone. Now we stop the the zone and detach it.

$ zoneadm -z zone1 halt
$ zoneadm -z zone1 detach

This frees the zfs partition from the zone and makes it accessible. We need that a bit later. Now we need an incremental snapshot and move that data to the new location.

$ zfs snapshot -r rpool/zones/old/zone1@move2
$ zfs send -R -i move1 rpool/zones/old/zone1@move2 | zfs recv rpool/zones/new/zone1

When we now list all zfs partitions, we see, that a partition zbe is mounted two times into the same location.

rpool/zones/old/zone1/ROOT/zbe    724M  1.59T   723M  /zones/old/zone1/root
rpool/zones/new/zone1/ROOT/zbe    724M  1.59T   723M  /zones/old/zone1/root

To fix that, issue the following command.

zfs set mountpoint=/zones/new/zone1/root rpool/zones/new/zone1/ROOT/zbe

Now the partition has to be mounted, so that zoneadm can find it for the attach. You can do that with the following command

zfs mount rpool/zones/new/zone1/ROOT/zbe

Now with the partition in the correct place, we have to tell the zone, where to look for its new partition.

$ zonecfg -z zone1
zonecfg:zone1> set zonepath=/zones/new/zone1
zonecfg:zone1> verify
zonecfg:zone1> commit
zonecfg:zone1> exit

With the zone reconfigured, attach the zone.

$ zoneadm -z zone1 attach

This may take a bit of time, as the content of the zone gets checked for compatibility. When it is back, check the zone is installed.

$ zoneadm list -cv
  ID NAME             STATUS     PATH                           BRAND    IP
   - zone1            installed  /zones/new/zone1               ipkg     excl

Now boot the zone and we are done.

$ zoneadm -z zone1 boot

Now check if everything is where you expect it to be and start your services and everything is good.

ideas

Here are some ideas, what can be done differently in the process.

iterative snapshots

If you zone has a lot of traffic, where many changes aggregate between the first snapshot and the second, do some more iterative snapshots before taking down the zone. This has the advantage, that you can close the gap of changes to a minimum size and therefore make the move at the end a bit faster. But check the available disk space in the process to avoid a full disk.

create a new zone

Instead of chaning the old zone and therefore making a rollback more complicated, create a new zone, which looks exactly like the old one. Instead of chaning the old one, do instead

$ zonecfg -z zone2
zonecfg:zone2> create -a /zones/new/zone1

This will set everything from the old zone with the new zonepath. Keep in mind, that this will also use the old interface. If you don’t want that, create a new interface before and change it in the config step.

You can also restore that zfs partition in a partition which has the correct.

I hope it helps and you have some fun playing with it.

This is some kind of hint for others, which may have the same problems I had.

I wanted to compile llvm 3.1 on omnios, an illumos distribution but it did not work out like I wanted it to. One of the first errors I got was a linking error.

Text relocation remains                         referenced
    against symbol                  offset      in file
llvm::LoopBase<llvm::MachineBasicBlock, llvm::MachineLoop>::getLoopPredecessor() const 0x149a           /tmp/build_gibheer/llvm-3.1.src/Release/lib/libLLVMCodeGen.a(MachineLICM.o)
llvm::LoopBase<llvm::MachineBasicBlock, llvm::MachineLoop>::getExitBlocks(llvm::SmallVectorImpl<llvm::MachineBasicBlock*>&) const 0x6200  /tmp/build_gibheer/llvm-3.1.src/Release/lib/libLLVMCodeGen.a(MachineLICM.o)
ld: fatal: relocations remain against allocatable but non-writable sections

The problem in this case is, that parts of the llvm code are not compiled position independent (PIC). As I learned, this can be solved with the following setting.

LDFLAGS="-mimpure-text -Wl,-ztextwarn"

This changes the way of linking to only warn about position independent code, but still link it all together. It is not a nice solution, but with this, it is possible to find out, where it is gooing wrong.

After that problem partially solved, I had another problem. Solaris supports 32bit and 64bit programs on the same environment, just like you can do on linux with multilib. The first compile of llvm produced 32bit binaries. When trying to compile llvm for 64bit, it was just impossible. I tried different things, like setting CFLAGS, LDFLAGS, OTHER_OPTIONS whatever there was and the only thing to get it compiled for 64bit is to overwrite CC and CXX. It seems like the Makefile just ignores the CFLAGS and therefore does only compile the code for the hostsystem seemingly bitness.

But both of these problems got solved with 3.2, which I tried from svn and they work. The release date of 3.2 is only 7 days away, so hopefully it will still work by then. Nice thing is, Rubinius can already use it :D