Sysadmin Life Hack: Syslog and NodeRed, a match made in heaven

I recently wanted to react to events detected by some obscure hardware monitoring device, using NodeRED. I was not exactly sure how to achieve this, because the device doesn’t really offer any method of accessing it via an API or something. Of course these devices often can send mail or SNMP traps, but I didn’t want to involve a mail server or deal with OIDs, so I kept looking for a different solution.

Another protocol that is actually quite widely supported is syslog. Since I had a syslog-ng instance running to send logs to Loki, I wondered if it would be possible to also send the logs to MQTT. Indeed this works, and it allowed me to implement my use-case quite easily. But ever since, I kept finding new applications for this: The syslog protocol is ubiquitous, and even the cheapest routers support it.

I’m going to lay out a few details on how it works.

Here’s a config snippet for syslog-ng that defines an MQTT destination:

destination d_mqtt {
    mqtt (
        address("tcp://127.0.0.1:1883")
        topic("syslog")
        fallback-topic("syslog_fallback")
        template("$(format-json --scope nv_pairs --scope core --scope all_macros --scope selected_macros --scope everything)")
        qos(0)
    );
};

Then just use an MQTT input node to connect to the syslog topic and you’re golden. You’ll receive structured data formatted as JSON that you can decode and then process. This requires syslog-ng version 3.33 though, so it’s not always an option. On older versions, another option that works reasonably well is to define a standard syslog destination like this:

destination d_nodered {
    network("127.0.0.1" transport("udp") port("20514"));
};

This destination will then use the older RFC-3164-style message format. If you don’t care too much about parsing message severities and facilities correctly, it works pretty well to just use a UDP input node, bind it to port 20514 and then use regular expressions or similar methods to parse the incoming messages.

I’m not sure if, in this case, it’s even worth going through syslog-ng when you could just as well configure the NodeRED UDP endpoint directly as a network syslog host. In my case, I need it also to send logs to Loki, so I have it running anyway.

You’ll then receive structured messages like this:

<189>1 2024-02-24T22:41:47+01:00 192.168.23.1 Something interesting happened

The numbers in the first field denote the facility and severity. The RFC says that the timestamp should follow immediately after the > character, but for some reason it doesn’t. Next is the IP address or hostname of the host the message originated from. The remainder of the data is the log message payload.

Once you have the messages in Node-RED, you can do whatever you like with them: Use it to control lights, track presence of people by evaluating log messages from the DHCP server or even trigger alarms in case something bad happens. NodeRED makes these things very easy, and tapping into the syslog protocol means you can connect to pretty much anything :)