Dump all Ansible hostvars for a host
If you want to dump all the hostvars for an Ansible host, try this:
ansible -i inventory -m command -a "echo \"{{ hostvars | to_nice_json }}\"" yourhostRunning the nextcloud client under systemd
Since my nextcloud sync client kept crashing due to too many open files, I wanted to have something auto-restart it. Also, I recently stumbled across the awesome fact that nowadays, every Linux Desktop session also includes its very own systemd user instance, so I put the nextcloud client under its supervision. Here’s how.
Ceph Bluestore/Filestore latency
We’ve been looking deeply into Ceph Storage latency, comparing BlueStore and FileStore, and looking at methods how to get below the magic 2ms write latency mark in our Proxmox clusters. Here’s what we found.
The endeavour was sparked by our desire to run ZooKeeper on our Proxmox Clusters. ZooKeeper is highly sensitive to IO latency: If writes are too slow, it will log messages like this one:
fsync-ing the write ahead log in SyncThread:1 took 1376ms which will adversely effect operation latency.File size is 67108880 bytes. See the ZooKeeper troubleshooting guideSubsequently, ZooKeeper nodes will consider themselves broken and restart. If the thing that’s slow is your Ceph cluster, this means that all three VMs will be affected at the same time, and you’ll end up losing your ZooKeeper cluster altogether.
We mitigated this by moving ZooKeeper to local disks, and getting rid of the Ceph layer in between. But that is obviously not a satisfactory solution, so we’ve spent some time looking into Ceph latency.
Unfortunately, there’s not a lot of advice to be found other than “buy faster disks”. This didn’t seem to cut it for us: Our hosts were reporting 0.1ms of disk latency, while the VMs measured 2ms of latency. If our hosts had weighed in at 1.8ms, I’d be willing to believe that we have a disk latency issue - but not with the discrepancy that we were seeing. So let’s dive in and see if we can find other issues.
Enforcing correct DNS upstreams for internal zones
When you’re frequently working with internal DNS zones of a company whose DNS server sits behind a VPN, you’ll probably soon encounter DNS shenanigans where you’ll find that resolving internal domain names is a lot more tricky than it should be. I’ve found a way that works using dnsmasq, but I also found that you need to be careful to keep an overly-eager NetworkManager in check.
Revisiting Samba RODC + Bind
Ok, so here’s another step in the evolution of my Samba4-RODC-based DNS setup. First steps were setting up a Samba4 Read-Only DC in my remote locations, so that DNS would be replicated to that location so that DNS doesn’t fail in case the VPN connection dies. Then we discovered that the SAMBA_INTERNAL DNS backend does not support caching, which unsurprisingly lead to performance problems, so we switched to Samba AD DC with Bind as DNS backend. This setup is quite a bit more complex though, and it seems a bit unstable in the sense that Samba lost its ability to update records in Bind for some reason and we have to “fix” that manually by re-joining the RODC to the domain. Rumor has it that the SAMBA_INTERNAL backend is a lot more stable. So, here’s step three in our evolution: Let’s allow Samba to use SAMBA_INTERNAL, but only run on 127.0.0.1, while communication with the outside world is handled by a bind instance that handles caching and forwards queries for the company domain records to Samba.
HKP port number
The port number used by GnuPG’s HKP protocol is 11371.
Just saying.
LetsEncrypt DNS verification using a local BIND instance
I’ve been looking into Let’s Encrypt DNS verification for a while. Not only because you’re able to obtain wildcard certificates through this method, freeing you from the necessity to obtain an individual certificate for every single one of your subdomains: It also allows you to get a certificate for stuff running on your LAN, provided you’re running it on a subdomain that belongs to you. The problem is though, how do you enable Certbot to automate the DNS server update, without putting a credential in place that would allow full access to all your domains? And what to do if you’re running a server for a domain that doesn’t even belong to you: How can the owner delegate permissions for the verification TXT records to you, without having to give you full access to all their domains? Today I stumbled across a solution: Delegate the _acme-challenge subdomain to a local BIND instance and have Certbot update that. Here’s how.
Monitoring DNS replication
Suppose you’re running a few DNS servers spread across multiple locations in your infrastructure, and you want to monitor DNS replication to make sure they’re all on the same page. Here’s an approach on how to do that using Prometheus and a bit of scripting.
Samba AD DC with Bind as DNS backend
How to set up a Samba AD DC to use Bind as its DNS backend in 2020 on Ubuntu 18.04. I’m writing this down as I go, because I feel a lot of chaos incoming and I kindof want that documented for some reason. TL;DR skip to “Configuration”.
How to run an ARM VM on a x86 host
A few weeks ago, I wasted a lot of time (and hair) trying to run Raspbian in a VM, so that I could run it as part of my CI system and build software for it. I eventually managed to get something running that counts as good enough, both for 32 and 64 bit.