I once again ended up having to edit a somewhat large Python file. Since I don’t believe in Clean Code’s methodology of splitting things up into thousands of micro-functions and then having to pass tons of variables around as arguments all the time, I’m actually happy with that. Also it’s kinda what code folding was invented for. Unfortunately I usually don’t find it all that helpful, because it doesn’t fold the way I want it to.

I like what C# does with #region and #endregion, where you can freely define a section of code to be folded away, exactly how it makes sense to you the human, who this feature is designed for, rather than the machine that gets to execute the code and thinks in different patterns than you do.

So, I found a way to enable Kate to do this in Python. Here’s how.

Read more…

Here’s a clean way to migrate an SQLite DB to MariaDB.

Read more…

Sometimes you’d like to span a layer2 network across two locations, but you don’t want to (or can’t) use VLANs. An excellent alternative in this situation is VXLAN, which transports the payload traffic via standard ole UDP packets on port 4789. This works easily with unmanaged switches (or switches you don’t have admin access to), and can even be routed across other networks or VPNs. Here’s how to set things up between two Linux boxes, and a FortiGate firewall to top it off.

Read more…

When debugging network connectivity issues, sometimes it would be helpful to know what’s the largest packet that can be transferred from one host to another without being fragmented. This is especially helpful when Jumbo Frames or VPNs are involved. The method to find this size is called Path MTU Discovery (PMTUD). It is performed by your OS of choice in the background, but you can also do it manually. In Linux this is easily done, but it’s also possible from Windows. Here’s how.

Read more…

I recently came across smbnetfs. Initially I had only been looking for a way to mount a Windows file share from userspace, so that I wouldn’t have to deal with permissions for my non-root user accessing a mountpoint. smbnetfs can do a lot more than just that though: It gives you the equivalent of Windows’s “Network” feature, allowing you to access the whole network via a single mount. Since the documentation is a bit sparse, here’s how I set it up.

Read more…

I recently wanted to react to events detected by some obscure hardware monitoring device, using NodeRED. I was not exactly sure how to achieve this, because the device doesn’t really offer any method of accessing it via an API or something. Of course these devices often can send mail or SNMP traps, but I didn’t want to involve a mail server or deal with OIDs, so I kept looking for a different solution.

Another protocol that is actually quite widely supported is syslog. Since I had a syslog-ng instance running to send logs to Loki, I wondered if it would be possible to also send the logs to MQTT. Indeed this works, and it allowed me to implement my use-case quite easily. But ever since, I kept finding new applications for this: The syslog protocol is ubiquitous, and even the cheapest routers support it.

I’m going to lay out a few details on how it works.

Read more…

If you want to dump all the hostvars for an Ansible host, try this:

ansible -i inventory -m command -a "echo \"{{ hostvars | to_nice_json }}\"" yourhost

Read more…

Since my nextcloud sync client kept crashing due to too many open files, I wanted to have something auto-restart it. Also, I recently stumbled across the awesome fact that nowadays, every Linux Desktop session also includes its very own systemd user instance, so I put the nextcloud client under its supervision. Here’s how.

Read more…

We’ve been looking deeply into Ceph Storage latency, comparing BlueStore and FileStore, and looking at methods how to get below the magic 2ms write latency mark in our Proxmox clusters. Here’s what we found.

The endeavour was sparked by our desire to run ZooKeeper on our Proxmox Clusters. ZooKeeper is highly sensitive to IO latency: If writes are too slow, it will log messages like this one:

fsync-ing the write ahead log in SyncThread:1 took 1376ms which will adversely effect operation latency.File size is 67108880 bytes. See the ZooKeeper troubleshooting guide

Subsequently, ZooKeeper nodes will consider themselves broken and restart. If the thing that’s slow is your Ceph cluster, this means that all three VMs will be affected at the same time, and you’ll end up losing your ZooKeeper cluster altogether.

We mitigated this by moving ZooKeeper to local disks, and getting rid of the Ceph layer in between. But that is obviously not a satisfactory solution, so we’ve spent some time looking into Ceph latency.

Unfortunately, there’s not a lot of advice to be found other than “buy faster disks”. This didn’t seem to cut it for us: Our hosts were reporting 0.1ms of disk latency, while the VMs measured 2ms of latency. If our hosts had weighed in at 1.8ms, I’d be willing to believe that we have a disk latency issue - but not with the discrepancy that we were seeing. So let’s dive in and see if we can find other issues.

Read more…

When you’re frequently working with internal DNS zones of a company whose DNS server sits behind a VPN, you’ll probably soon encounter DNS shenanigans where you’ll find that resolving internal domain names is a lot more tricky than it should be. I’ve found a way that works using dnsmasq, but I also found that you need to be careful to keep an overly-eager NetworkManager in check.

Read more…