BOFH Meeting 19 July 2025
Jump to navigation
Jump to search
Meeting notes
Note (r3boot): I added a priority to the points from my PoV. 1 == highest, 10 is lowest
Monitoring
- Monitoring stack with alerting (we have uptime-kuma for non-bofh), but what else do we want?
- Folkert mentioned Icinga2 and that is also what we ran before.
- Prometheus + Grafana + Alertmanager etc is also a good solution.
- The latter also gives us Grafana Loki (log aggregation system).
- Prometheus gives us both metrics & monitoring
- Writing alerts is easier then deploying a nagios/icinga check
- Ready support for all kinds of notification systems
- Prio: 5/10
- Action: r3boot is building this
Kitchen Rack & Cabling
- Moving the kitchen rack before renovations begin.
- We should probably check what cabling is still in use and which ones are defunct at this point due to mentioned renovations. (Think any link going to zaal 2).
- Perhaps we could extend the cabling going to zaal 1 with RJ45 extenders for the time being?
- Prio: 3/10
- Action: Rubberhark (and r3boot when available)
SSL Certificates
- Rolling out Let's Encrypt certificates for on and off site infra.
- For example, SSL certificates for uptime-kuma running at docker-infra.
- Make it easy for members to deploy something that needs HTTPS?
- This already works via revproxy.vm.nurd.space
- Scripts for internal PKI can be documented on the wiki
- Prio: 8/10
- No action
Script Management
- We seem to have a lot of little scripts running but they mostly get deployed on the harkbot LXC.
- Maybe we could make a proper VM for this instead and some sort of system that can monitor these scripts and also provide a description of what it actually does.
- Having it all consolidated in one system will simplify management greatly.
- Think Docker but more simplified. Maybe solutions for this already exist?
- Deploy on a VM using ansible OR store all of them in a git repo and deploy that on a host
- By creating a "chokepoint" you get a centralized place to combine all functionality required for a script (unit file, monitoring, etc)
- Prio: 5/10
Harkbot Migration
- Switch harkbot from LXC (Debian) to a VM (AlmaLinux)?
- Could also roll it out via OpenTofu & Ansible stack then.
- Prio: 5/10
MikroTik Licensing
- Our MikroTik trial license will expire this month.
- What shall we do?
- Switch back to OPNsense?
- Buy the license?
- Switch to a hardware MikroTik router?
- We were not happy with how blocked down the Nightmare (dream) machine was.
- OPNsense is OK but also has its quirks.
- Buying a P1 CHR license (1Gbit) is around €38.
- https://www.dectdirect.nl/nl/crs310-8g2sin.html (€190 MikroTik router for example)
- Meh .. +1 for OPNsense .. It works, and its predictable. Its also boring, but thats okay.
- Prio: 2/10
BOFH Password Management
- Start using Bitwarden for BOFH password management PLEASE.
- It would be great if we stopped using the standard insecure passwords, lol.
- Can it also contain people's SSH certs and auto-deploy them to BOFH-managed devices?
- See vaultwarden.vm.nurd.space, but we need to start moving & resetting passwords
- Prio: 3/10
EPC MQTT Bridge
- The EpcMqttBridge probably needs a rewrite and not yeet itself out of existence the moment one of the EPCs goes offline.
- They are also fairly slow so maybe we could look into replacing the brains with a network-based ESP32 instead?
- Prio: 5/10
Erratic Server Power Usage
- Erratic is gobbling up power like Mia Khalifa on OnlyFans. Wat do?
- I'd like to keep two systems for redundancy, but we could replace the hardware with something that gobbles less power while keeping or even increasing the performance, if we were to switch to consumer hardware.
- Make BOM and we can discuss it.
- Prio: 2/10
Consolidation Considerations
- If we upgrade Erratic, do we also want to move Harmony's drives to Erratic?
- Right now Harmony does about 50~60W (estimated), we could probably reduce the rack total with a few watts if we consolidate them.
- We probably would need an extra SATA controller for 6 extra drives; this might end up not reducing the power usage by much in the end.
- Giving Erratic a 10Gbps NIC would mean we can increase max space storage throughput.
- Camera recording would also need to be handled by Erratic then (right now it goes to the 2×2TB in Harmony).
- Though if we are going with more than two cameras, we'll need to anyway.
- Prio: 4/10
GPON Power
- The GXPN power supply still needs to be connected to the door UPS system.
- Prio; 8/10
Matrix Setup
- We still have to roll out the Matrix setup.
- A start has been made here: https://git.nurd.space/bofh/opentofu/machines/matrix
- Prio: 4/10
MinIO S3
- Minio S3 pulled a "Everyone disliked that".
- Are we going to switch to something else? Garage?
- There is also https://github.com/OpenMaxIO/openmaxio-object-browser that we could roll out and keep using Minio.
- Prio: 7/10
Home Assistant
- Currently our Home Assistant setup is pretty clunky.
- What if we redeployed a new instance to `sporadic` and started anew?
- Prio: 8/10
Rack Monitoring
- Rack monitoring? It might be nice to have a temp sensor at the back and front?
- Prio: 7/10
UPS & NUT
- NUT-ing onto the UPS still needs to be done.
- There is a Pi connected to the UPS via serial but everything still needs to be setup.
- Prio: 6/10
Portal
- We are still missing the portal, what will we do about that?
- There is a PoC with plugin support, written in python + easygui
- Prio: 6/10
Ticketing & Kanban board
- A ticketing system that can be both used by BOFH as well as other members/committees would be very nice to have
- A kanban board would be nice as well, although for BOFH we could technically use the one integrated into Gitlab
- Prio: 2/10
Gitlab
- We could probably upgrade the version to 18.1?
- Requires some steps likely, beware
- Also update gitlab-runners
- Also beware of ACL changes in 18
- Prio: 3/10
Keycloak
- Newer version breaks with our ancient SAML setup
- Switching to oidc2 is possible but can be tricky to get user mapping correct
- Prio: 3/10
The Wiki
- There have been often complains of how the wiki currently is.
- Do we want to do something about that? Or put someone on it?