After more than a year working on my mail setup, I think I have it running in a pretty good way. As some of the stuff is not documented at all in the wide of the internet, I will post parts here to make it accessible to others.

Many setups use the MTA (postfix, exim) to store mails on the filesystem. My setup lets dovecot take care of that. That way it is the only process able to change data on the filesystem.

to make this work, we first need a lmtp socket opened by dovecot. The configuration part looks like this

service lmtp {
  unix_listener /var/spool/postfix/private/delivery.sock {
    mode = 0600
    user = postfix
    group = postfix
  }
}

LMTP is a lightweight smtp protocol and most mail server components can speak it.

Next we need to tell postfix to send mails to this socket instead storing it on the filesystem. This can be done with with the following setting

mailbox_transport = lmtp:unix:/var/spool/postfix/private/delivery.sock

or for virtal accounts with

virtual_transport = lmtp:unix:/var/spool/postfix/private/delivery.sock

Now postfix will use the socket to deliver the mails.

It is also possible to use other services between these two like dspam. In my case postfix delivers the mails to dspam and that will deliver them to dovecot.

For dovecot change the path of the socket to something dspam can reach. I'm using /var/run/delivery.sock.

Then change the dspam.conf to use that socket as a delivery host

DeliveryProto LMTP
DeliveryHost  "/var/run/delivery.sock"

As postfix needs to speak to dspam, we set dspam to create a socket too

ServerMode auto
ServerDomainSocketPath "/var/run/dspam.sock"

ServerMode should be set to either auto or standard.

Now the only thing left to do is to tell postfix to use that socket to deliver its mails. For that, set the options from before to the new socket

virtual_transport = lmtp:unix:/var/run/dspam.sock

And with that, we have a nice setup where only dovecot stores mails.

This weekend I had a small problem with my omnios installation. The installation is now more than a year old and back then, the feature flags for zfs were really fresh. So as time went on, zfs got better but somehow it was missed to update the grub installation. When I then booted my server on friday, I did not get up again as grub was unable to load my zpool.

Thanks to Rich Lowe from illumos project the bug got fixed. But I had to somehow run installgrub on my system to get it up again. For that to work, I had to find a system which either boots a current illumos kernel or lets me enter a kernel paramter.

To make things more complicated, the current omnios installation medium did not like my system and just rebooted with a kernel panic without me having a chance to get my kernel paramter from the zpool. FreeBSD 9.1 was not able to read the zpool because of the feature flags. After half a day of try and error I just went with SmartOS and lo and behold - it worked!

So I imported my zpool with zpool import -o /mnt rpool, read my menu.lst to get the parameter and restarted with omnios to enter that command to the kernel in grub and I got a live environment up to do installgrub!

For me, the parameter in question is -B acpi-user-options=0x2,$ZFS-BOOTFS to switch off ACPI on my Sandy Bridge Celeron system.

A very nice feature on Solaris was the possibility to initialize new zones with a sysidcfg file. This does not exist on omnios. With kayak, omnitis deployment server, a way to run postboot scripts was created. The way is the file /.initialboot. This is just a shell script which gets executed on the first boot and gets removed afterwards. Nothing much but already very useful to make the initial setup for dns and the ip.

A little example. I have a zone foo1 with a vnic barnic1. I want to setup dns and dhcp for the interface. The zone is installed in /zones/foo1/ so we have to root file system mounted as /zones/foo1/root.

We create the /.initialboot file (which ends up as /zones/foo1/root/.initialboot with the following content

ipadm create-if foonic1
cp /etc/nsswitch.dns /etc/nsswitch.conf
ipadm create-addr -T dhcp foonic1/ipv4
echo "nameserver 192.168.56.1" >> /etc/resolv.conf

Now boot the zone and after 5minutes or so everything is setup and ready to go. Makes it really easy.

I had the need to filter logs from different programs into different places - in this case the postgres and nginx logs. The man page of syslog.conf describes it pretty good, but misses some examples to make it more clear. So here is how I configured it, to make it easier.

First, I edited the syslog.conf

# filter everything apart from postgres and nginx
!-postgres,nginx
*.err;kern.warning;auth.notice;mail.crit    /dev/console
# and all the other stuff

# filter only postgres
!postgres
*.*        /var/log/postgresql.log

# filter only nginx
!nginx
*.*        /var/log/nginx.log

The next step is to setup the log rotate. This happens in /etc/newsyslog.conf. The man page is very helpful, so if you want to adjust something, take a peek into it.

# postgresql                                                                                                                                                                                  
/var/log/postgresql.log     640  5     100  * JC                                                                                                                                              

# nginx                                                                                                                                                                                       
/var/log/nginx.log      640  5     100  * JC

And that is all. If you want to add more program filtes, you have to define them in the syslog.conf as notfilter and filter and add the rotate to newsyslog.conf.

I got an interesting question regarding zones on Solaris in #omnios.

scarcry: Does anyone know how to move a zone from one zpool to another?

There are some guides out there on how to move a zone from one machine to another, but most of them install the zone in the same place as before.

But instead of moving it from one machine to another, this small guide will just show what to do, when only the location is chaning.

preparations

First, we need to setup the partitions and zones for our little experiment. For this example, I will use the pool rpool and the following partitions

• rpool/zones/old mounted to /zones/old/
• rpool/zones/new mounted to /zones/new/

We also need the zone config, so here is it.

create -b
set zonepath=/zones/old/zone1
set ip-type=exclusive
set autoboot=false
add net
set physical=zone1
end
commit

Just install the zone with the normal commands

$ zonecfg -z zone1 < zone.config
$ zoneadm -z zone1 install
$ zoneadm -z zone1 boot

Check if the zone is running and write a file, just to make sure, we have the same zone at the end.

moving the zone

For this guide, we will assume, that the zone is in production use and can't be offline too long. For that to work, we will do a first snapshot, when the zone is still running.

$ zfs snapshot -r rpool/zones/old/zone1@move1

After that, we can replay that snapshot into the new location.

$ zfs send -R rpool/zones/old/zone1@move1 | zfs recv rpool/zones/new/zone1

This step will take some time, depending on the size of your zone. Now we stop the the zone and detach it.

$ zoneadm -z zone1 halt
$ zoneadm -z zone1 detach

This frees the zfs partition from the zone and makes it accessible. We need that a bit later. Now we need an incremental snapshot and move that data to the new location.

$ zfs snapshot -r rpool/zones/old/zone1@move2
$ zfs send -R -i move1 rpool/zones/old/zone1@move2 | zfs recv rpool/zones/new/zone1

When we now list all zfs partitions, we see, that a partition zbe is mounted two times into the same location.

rpool/zones/old/zone1/ROOT/zbe    724M  1.59T   723M  /zones/old/zone1/root
rpool/zones/new/zone1/ROOT/zbe    724M  1.59T   723M  /zones/old/zone1/root

To fix that, issue the following command.

zfs set mountpoint=/zones/new/zone1/root rpool/zones/new/zone1/ROOT/zbe

Now the partition has to be mounted, so that zoneadm can find it for the attach. You can do that with the following command

zfs mount rpool/zones/new/zone1/ROOT/zbe

Now with the partition in the correct place, we have to tell the zone, where to look for its new partition.

$ zonecfg -z zone1
zonecfg:zone1> set zonepath=/zones/new/zone1
zonecfg:zone1> verify
zonecfg:zone1> commit
zonecfg:zone1> exit

With the zone reconfigured, attach the zone.

$ zoneadm -z zone1 attach

This may take a bit of time, as the content of the zone gets checked for compatibility. When it is back, check the zone is installed.

$ zoneadm list -cv
  ID NAME             STATUS     PATH                           BRAND    IP
   - zone1            installed  /zones/new/zone1               ipkg     excl

Now boot the zone and we are done.

$ zoneadm -z zone1 boot

Now check if everything is where you expect it to be and start your services and everything is good.

ideas

Here are some ideas, what can be done differently in the process.

iterative snapshots

If you zone has a lot of traffic, where many changes aggregate between the first snapshot and the second, do some more iterative snapshots before taking down the zone. This has the advantage, that you can close the gap of changes to a minimum size and therefore make the move at the end a bit faster. But check the available disk space in the process to avoid a full disk.

create a new zone

Instead of chaning the old zone and therefore making a rollback more complicated, create a new zone, which looks exactly like the old one. Instead of chaning the old one, do instead

$ zonecfg -z zone2
zonecfg:zone2> create -a /zones/new/zone1

This will set everything from the old zone with the new zonepath. Keep in mind, that this will also use the old interface. If you don't want that, create a new interface before and change it in the config step.

You can also restore that zfs partition in a partition which has the correct.

I hope it helps and you have some fun playing with it.

This is some kind of hint for others, which may have the same problems I had.

I wanted to compile llvm 3.1 on omnios, an illumos distribution but it did not work out like I wanted it to. One of the first errors I got was a linking error.

Text relocation remains                         referenced
    against symbol                  offset      in file
llvm::LoopBase<llvm::MachineBasicBlock, llvm::MachineLoop>::getLoopPredecessor() const 0x149a           /tmp/build_gibheer/llvm-3.1.src/Release/lib/libLLVMCodeGen.a(MachineLICM.o)
llvm::LoopBase<llvm::MachineBasicBlock, llvm::MachineLoop>::getExitBlocks(llvm::SmallVectorImpl<llvm::MachineBasicBlock*>&) const 0x6200  /tmp/build_gibheer/llvm-3.1.src/Release/lib/libLLVMCodeGen.a(MachineLICM.o)
ld: fatal: relocations remain against allocatable but non-writable sections

The problem in this case is, that parts of the llvm code are not compiled position independent (PIC). As I learned, this can be solved with the following setting.

LDFLAGS="-mimpure-text -Wl,-ztextwarn"

This changes the way of linking to only warn about position independent code, but still link it all together. It is not a nice solution, but with this, it is possible to find out, where it is gooing wrong.

After that problem partially solved, I had another problem. Solaris supports 32bit and 64bit programs on the same environment, just like you can do on linux with multilib. The first compile of llvm produced 32bit binaries. When trying to compile llvm for 64bit, it was just impossible. I tried different things, like setting CFLAGS, LDFLAGS, OTHER_OPTIONS whatever there was and the only thing to get it compiled for 64bit is to overwrite CC and CXX. It seems like the Makefile just ignores the CFLAGS and therefore does only compile the code for the hostsystem seemingly bitness.

But both of these problems got solved with 3.2, which I tried from svn and they work. The release date of 3.2 is only 7 days away, so hopefully it will still work by then. Nice thing is, Rubinius can already use it :D

Today someone told me about natural and inner joins. As I'm using SQL for many years already, I was a bit puzzled at first. I heard of the terms, but thought till now, that they were meaning the same.

The first thing I did was looking in the PostgreSQL documentation and yes, they are not the same. But they are also the same.

The inner join is the default for doing joins. It just joins two tables using the on clause.

# select *
  from tableA A
  join tableB B
    on A.id = B.id;

| a.id | b.id |
|------|------|
|    3 |    3 |

Now an interesting thing is, that the on clause can be replaced by a using clause when both tables provide the same columns. This not only makes the select a bit shorter, but also reduces the number of columns in the result. All columns listed in the using clause will be left out from the result and replaced with a new column with the name used in the using clause. The select from above would then look like this

# select *
  from tableA A
  join tableB B
  using (id);

| id |
|----|
|  3 |

The natural join goes one step further and tries to search for common columns itself and generate a using clause itself. The resulting query then looks like this

# select *
  from tableA A
  natural join tableB B;

| id |
|----|
|  3 |

As nice as this seems it can backfire pretty fast, when one has two tables with a column of the same name, but completely different content not meant to be joined. Then it is possible just to get nothing.

# select * from foo limit 1;
 id | test | bar 
----+------+-----
  1 | 1    | 3

# select * from baz limit 1;
 id | test | bar 
----+------+-----
  1 | 1    | 20

# select * from foo natural join baz
 id | test | bar 
----+------+-----

As all columns are named the same, but the content is different in column bar, no common row is found and therefore returned.

For further information, the PostgreSQL documentation is pretty good.

Today we release a small project from me - zero 0.1.0.

It is aimed at being a toolkit for building web services. It is build around the idea of abstracting away what is tedious work and work with the information clearer. With that in mind, some modules are already included. These are the following.

Request

This class provides an interface to information regarding the request and making them available grouped together in other parts.

One example is the grouping of parameters in one attribute request.params which makes get and post parameters available separately. As many developers also regard URLs as some kind of parameters it is possible to define custom key/value pairs.

Another example is the grouping of all accept headers under #accept, which makes accessing type information nice to read: request.accept.types.

Response

The response class helps in building responses in a easy way. It provides similar interface as the Rack::Response. It also does a small check already for status code correctness and will probably get more helpers in the future to set common headers and status codes.

Router

Zero has a small router which takes routes and pushes the request to other applications. A similar implementation is Rack::URLMap, but this router is also able to extract variables from the URL and puts them into the Request.

Renderer

The renderer is a facility to render templates according to the accept type wanted by the client. That is possible by providing a map of short type names to the actual mimetypes which is then used internally to search for the right template. With this, developers have only to provide the template and mapping and then the rendering takes care of actually using it.

The router does not take care of the actual method of the request. This can be done before the router with a small application defining routers for every possible method or working with the method in the applications called by the routers.

Controller

This component is still a "Work in Progress" but can already be used to glue all these parts together. It splits the work into two steps - processing the request and rendering the resulting data. If that workflow does not fit, it is also possible to extend or modify the controller to adapt other work flows.

Status and Future of the toolset

The idea is to make web service development easier than before and this is only the beginning of zero. The plan is to extend the toolkit with more modules and make all modules possibly available as stand alone modules. That way, zero can be used in other projects to replace parts or help in making developing easier. It is a Work in Progress at the moment but the results so far are very promising.

We also have a repository for example applications and we will extend it with the time to show as many aspects of the toolkit as possible.

If you are interested, checkout zero on github or have a look at the examples.

Just out of curiosity I tried to build a service for PostgreSQL and the systemd init system. Before that, I only read the service files of postgres and dhcp delivered with Archlinux. What I wanted to build is a service file able to start multiple instances of postgres with separate configuration files.

This was much easier than I thought it would be.

Systemd supports that pretty well and the only thing to do, is add an '@' to the service file name. Everything after '@' is then put into a variable %I, which can be used in the service file. So my service file was named 'pg@.service' and I put it into /etc/systemd/system. Another possible location is /usr/lib/systemd/system/.

The service file looks like an .ini file. It has the three sections Unit, Service and Install. The section Install in which target the service is installed. Targets are like run levels in other init systems. The postgres service gets installed into the multi-user target, which is started after the network:

[Install]                                                                       
WantedBy=multi-user.target

The next part is Unit. This section describes the service with a short description and a description of the dependencies. Postgres just needs the network up, so this section looks like this:

[Unit]                                                                          
Description=run PostgreSQL instance %I                                          
After=network.target

There you can also see the %I, which is replaced with the part after '@' from the name in systemd.

The next section is a bit larger and describes everything needed to manage the service itself, like start, stop and reload.

[Service]                                                                       
User=postgres                                                                   
Group=postgres                                                                  
TimeoutSec=120                                                                  
Type=forking                                                                       

EnvironmentFile=/etc/conf.d/pg.%I                                                  

SyslogIdentifier=postgres-%i                                                       

ExecStartPre=/usr/bin/postgresql-check-db-dir ${pgdata}                            
ExecStart= /usr/bin/pg_ctl -s -D ${pgdata} start -w -t 120                         
ExecReload=/usr/bin/pg_ctl -s -D ${pgdata} reload                                  
ExecStop=  /usr/bin/pg_ctl -s -D ${pgdata} stop -m fast                                       

OOMScoreAdjust=-200

Okay, this is a bit longer than the other parts. The first Couple of options handle the user to start with and he startup timeout. The timeout can't be replaced with a variable because all options from the config will be loaded as environment variables on execution. The Type option is very important, because it can't be set to anything else as forking for postgres, because it will fork to the background. So if you start it as a simple service systemd would loose the handler to postgres and stop it immediately.

The next options are EnvironmentFile and SyslogIdentifier. The first is for a small config file in /etc/conf.d/pg.instance where you replace instance with the instance name. As you can see with the %I in place, it will fill up the full name with the instance identifier. So you can use different config files for different instances. The same happens to the SyslogIdentifier. I thought it would be awesome if the log can be showed per instance and this is what you need to make it happen.

The option OOMScoreAdjust is just an option for the OOMKiller, that it should leave postgres alone as much as possible.

The option ExecStartPre calls a script which is delivered with postgres on Archlinux and does a check for the data dir. If it does not exist, it will log a line on how to create it. Pretty neat. ExecStart, ExecStop and ExecReload describe the actions to be done, when the service should be started, stopped or reloaded. As you can see, the script uses ${pgdata} to determine where to look and that variable comes from the EnvironmentFile, which looks for my first instance like this

pgdata=/tmp/ins1

The file is saved as /etc/conf.d/pg.ins1 and is really nothing more than this. The rest can handle postgres itself.

Now how do we get the service file into systemd? You do a

systemctl --system daemon-reload

and then

systemctl start pg@ins1.service

This creates your first service and tries to start it. You will get an error message like the following

Job for pg@ins1.service failed. See 'systemctl status pg@ins1.service' and 'journalctl' for details.

If you run the status, you will see that it failed, how it failed and the log message from the check script. After that, you can create the instance and start it anew there it is.

# systemctl status pg@ins1.service
pg@ins1.service - PostgreSQL database server
          Loaded: loaded (/etc/systemd/system/pg@ins1.service; disabled)
          Active: active (running) since Tue, 25 Sep 2012 09:27:54 +0200; 3 days ago
         Process: 372 ExecStop=/usr/bin/pg_ctl -s -D ${PGROOT}/data stop -m fast (code=exited, status=0/SUCCESS)
         Process: 624 ExecStart=/usr/bin/pg_ctl -s -D ${PGROOT}/data start -w -t 120 (code=exited, status=0/SUCCESS)
         Process: 619 ExecStartPre=/usr/bin/postgresql-check-db-dir ${PGROOT}/data (code=exited, status=0/SUCCESS)
        Main PID: 627 (postgres)
          CGroup: name=systemd:/system/postgresql.service
                  ├ 627 /usr/bin/postgres -D /var/lib/postgres/data
                  ├ 629 postgres: checkpointer process   
                  ├ 630 postgres: writer process   
                  ├ 631 postgres: wal writer process   
                  ├ 632 postgres: autovacuum launcher process   
                  └ 633 postgres: stats collector process

Now if you want to see some logging, you can ask journcalctl and give it the log string.

journcalctl SYSLOG_IDENTIFIER=postgres-instance1

That's all there is to multi instance services on syslog. To figure everything out actually took not even much time, as the documentation is pretty good. Just a hint, don't look in the web for documentation but in the man pages. The best starting point to look for documentation is man systemd and then take a look at the SEE ALSO section.

Have fun!

After I switched everywhere to a tiling wm, I wondered how everybody else locks his screen. Sure, you can lock the screen with a keybinding, but what when you leave the pc for talking and then leave it be?

The tool I found is xautolock and works pretty good. After a configurable time span it starts the lock and after another time it can also start suspend, hibernate or whatever. I use it with the following settings:

xautolock -locker slock -time 2 -killer "systemctl suspend" -killtime 10 &

This starts slock, the simple locker, after two minutes and sends the pc into suspend after 10 minutes in activity. As it runs in the background, you can either start it through .xinitrc or with your wm of choice.

To lock the screen by command, bind xautolock -locknow to your keys and it calls the deamon which then calls the locker.