exim4 upgrades and configuration fragility

Last night I decided I'd catch up on sysadmin tasks. Some of that was trying to tighten up my spam filtering again. I had got in place a per-user Bayesian filter on spamassassin, which essentially should allow it to learn a much more individual pattern of what each user considers spam. I also had configuration for having my mail server (running exim) reject mail within the SMTP session for very egregious examples. That configuration hasn't been working for a while - I had to revert lots of custom changes to my config for a significant exim4 upgrade a while back, and I haven't had the time and patience to try and reinstate it all. So I thought I'd look at that.

I ran a standard apt update and upgrade before I started. I noted exim4 and its various binary packages were marked for upgrade which isn't unusual and proceeded (the upgrade was between versions 4.95-RC2-1 and 4.95-1). Debian warned me of a change to exim4.conf.template and I examined the diff briefly, didn't see anything extraordinary and retained my config. In any case, I am using a distributed config in conf.d so expected to see a more targetted diff on one of those. I didn't. exim restarted without complaint.

I started using tail to follow my exim logs, and could immediately see that every single inbound message was being temporarily rejected, albeit, there was no information as to why. I spent probably the guts of an hour checking the changes between my configuration files and .dpkg-dist versions (the ones shipped as the new changes) and couldn't see the problem. I tried copying over the exim4.conf.template and updating the configuration with update-exim4.conf and still I had the same problem. I checked the changelog and didn't see anything profound that should really be a worry.

In the end I had to downgrade all the exim4 packages, and my mail started to be delivered again. Of course, this only buys me a little time to either find the problem, or hope it's something upstream in the Debian package. I maybe should report this as a bug, but I feel I don't yet have enough information.

However, it really got me thinking just how fragile exim4 configuration seems to be. I need to add a few tweaks to the shipped configuration if I want more effective handling of some things, such as my multiple domains, or being able to handle email addresses with extra bits like <name>-<website>@<domain> so I can filter email from various sites (and determine who sold my data). But in the main the challenges are in more effective spam filtering. All of those tweaks can be really easily disrupted by an update. I've been thinking of trying to generate a patch set against the default configuration in case that makes it easier to accept all new config files and applying the patch set, but that will have its own problems. greylist - a package that temporarily rejects initial attempts to deliver mail from unknown servers uses this patch approach.

In the meantime, if you upgrade exim4 and suddenly have all inbound messages being temporarily rejected you are not alone, but I can't yet explain why. For me a temporary downgrade was the only solution.

aptitude install exim4=4.94.2-7 exim4-base=4.94.2-7 exim4-config=4.94.2-7 exim4-daemon-heavy=4.94.2-7