Oh, The Huge Manatee

A blog about technology, open source, and the web... from someone who works with all three.

My War on Systemd-resolved

I run ubuntu as the base for my daily driver machine – heavily customized though it is – because Canonical’s choices are, by definition, mainstream. That makes them easy to support, easy to understand, and generally easy to work with. So what I’m about to describe is exceptional in how frustrating it is for me. Seriously, this one issue keeps is enough to drive me into the arms of another distro.

Ubuntu has a built in DNS cache, which it checks first when trying to resolve anything. This Makes Sense For The User in that DNS queries are resolved faster if they come from local cache. It Makes Sense for the network admin, in that repetitive DNS queries don’t take up bandwidth. But it really doesn’t Make Sense for the web developer.

The local DNS cache takes up port 53, which is a problem if you’re trying to run any different kind of DNS service locally. For example, the DNS service that practically any integrated docker-based development environment (docksal, ddev, pygmy, etc etc etc) needs. Even my own haproxy-based setup needs to control DNS. Not to mention, the system randomly fails all on its own! So: in order to do my job, I need to disable the Ubuntu DNS proxy. That’s where the trouble begins.

There are a hundred variations on how to disable systemd-resolvd on askubuntu and stackoverflow. All of them have different suggestions. Partly this is just the flexibility of *nix systems. But it’s largely because in the 5 years since Ubuntu introduced the local DNS cache, it’s changed which system it’s using (twice), how it starts up, how its config is stored, and the overall startup daemon control process. Every year, there are new tricks to learn for how to disable this totally extraneous system. And every time you update, it is magically re-enabled (or enabled in a new way). This is enough that it’s kept me from bothering with dist-upgrades for the last year. So I run the latest kernel version, but an old version of the OS, partly because I really hate fighting this problem.

For my own notes, and hopefully to help some others, here is how I disable the local DNS cache on Ubuntu 17.04.

# confirm that it's running
$ sudo netstat -tulpn |grep 53
udp        0      0 *                           28168/systemd-resolved
$ sudo ps -aux |grep resolved
systemd+ 28168  0.0  0.0  50024  5300 ?        Ss   Jan24   0:01 /lib/systemd/systemd-resolved
# try to kill it.
$ sudo killall systemd-resolved
$ ps -aux |grep resolved
systemd+ 28168  0.0  0.0  50024  5300 ?        Ss   Jan24   0:01 /lib/systemd/systemd-resolved
# roll your eyes, and try to disable it through the normal startup routes.
$ sudo service resolvconf disable-updates
$ sudo update-rc.d resolvconf disable
$ sudo service resolvconf stop
# pray that it survives a reboot.

Note that if you do this, you have to manage your own nameservers in /etc/resolved.conf! In my case networkmanager does it for me, but that won’t be true for everyone.