BOFH Meeting 19 July 2025
Jump to navigation
Jump to search
Meeting notes
Monitoring
- Monitoring stack with alerting (we have uptime-kuma for non-bofh), but what else do we want?
- Folkert mentioned Icinga2 and that is also what we ran before.
- Prometheus + Grafana + Alertmanager etc is also a good solution.
- The latter also gives us Grafana Loki (log aggregation system).
- Prometheus gives us both metrics & monitoring
- Writing alerts is easier then deploying a nagios/icinga check
- Ready support for all kinds of notification systems
- Prio: 5/10
Kitchen Rack & Cabling
- Moving the kitchen rack before renovations begin.
- We should probably check what cabling is still in use and which ones are defunct at this point due to mentioned renovations. (Think any link going to zaal 2).
- Perhaps we could extend the cabling going to zaal 1 with RJ45 extenders for the time being?
- Prio: 3/10
SSL Certificates
- Rolling out Let's Encrypt certificates for on and off site infra.
- For example, SSL certificates for uptime-kuma running at docker-infra.
- Make it easy for members to deploy something that needs HTTPS?
- This already works via revproxy.vm.nurd.space
- Scripts for internal PKI can be documented on the wiki
Script Management
- We seem to have a lot of little scripts running but they mostly get deployed on the harkbot LXC.
- Maybe we could make a proper VM for this instead and some sort of system that can monitor these scripts and also provide a description of what it actually does.
- Having it all consolidated in one system will simplify management greatly.
- Think Docker but more simplified. Maybe solutions for this already exist?
Harkbot Migration
- Switch harkbot from LXC (Debian) to a VM (AlmaLinux)?
- Could also roll it out via OpenTofu & Ansible stack then.
MikroTik Licensing
- Our MikroTik trial license will expire this month.
- What shall we do?
- Switch back to OPNsense?
- Buy the license?
- Switch to a hardware MikroTik router?
- We were not happy with how blocked down the Nightmare (dream) machine was.
- OPNsense is OK but also has its quirks.
- Buying a P1 CHR license (1Gbit) is around €38.
- https://www.dectdirect.nl/nl/crs310-8g2sin.html (€190 MikroTik router for example)
BOFH Password Management
- Start using Bitwarden for BOFH password management PLEASE.
- It would be great if we stopped using the standard insecure passwords, lol.
- Can it also contain people's SSH certs and auto-deploy them to BOFH-managed devices?
KV Store
- What about using a KV store for certificates etc?
EPC MQTT Bridge
- The EpcMqttBridge probably needs a rewrite and not yeet itself out of existence the moment one of the EPCs goes offline.
- They are also fairly slow so maybe we could look into replacing the brains with a network-based ESP32 instead?
Erratic Server Power Usage
- Erratic is gobbling up power like Mia Khalifa on OnlyFans. Wat do?
- I'd like to keep two systems for redundancy, but we could replace the hardware with something that gobbles less power while keeping or even increasing the performance, if we were to switch to consumer hardware.
- The AMD Ryzen 5 4600G is cheap CPU (€117) from 2020 with 6 cores/12 threads and does 65 watts TDP. Its multithread rating is 15,987 and single 2,652.
- Currently Erratic is a 6 core / 12 thread from 2014 with a TDP of 85W and a multithread of 7,776 and a single of 1,688.
- It's double the performance for less power!
- If we want _more_ we could opt for a better Ryzen CPU.
- The AMD Ryzen 7 5800X is €138 and a multithread of 27,740 and a single of 3,446 which is a 256.7% increase but at 105W TDP.
- Motherboard would be something like €132.
- We could maybe take our 64GB DDR4 ECC with us if we get a motherboard that supports ECC (the 5800X supports ECC).
- The 5800X does _not_ have a GPU. And only CPUs with no iGPU support ECC.
- Melan does have 64GB non-ECC that he maybe is willing to donate.
- (Maybe in exchange for the Erratic guts? *wink wink nudge nudge*)
Consolidation Considerations
- If we upgrade Erratic, do we also want to move Harmony's drives to Erratic?
- Right now Harmony does about 50~60W (estimated), we could probably reduce the rack total with a few watts if we consolidate them.
- We probably would need an extra SATA controller for 6 extra drives; this might end up not reducing the power usage by much in the end.
- Giving Erratic a 10Gbps NIC would mean we can increase max space storage throughput.
- Camera recording would also need to be handled by Erratic then (right now it goes to the 2×2TB in Harmony).
- Though if we are going with more than two cameras, we'll need to anyway.
GPON Power
- The GXPN power supply still needs to be connected to the door UPS system.
Matrix Setup
- We still have to roll out the Matrix setup.
- A start has been made here: https://git.nurd.space/bofh/opentofu/machines/matrix
MinIO S3
- Minio S3 pulled a "Everyone disliked that".
- Are we going to switch to something else? Garage?
- There is also https://github.com/OpenMaxIO/openmaxio-object-browser that we could roll out and keep using Minio.
Home Assistant
- Currently our Home Assistant setup is pretty clunky.
- What if we redeployed a new instance to `sporadic` and started anew?
Rack Monitoring
- Rack monitoring? It might be nice to have a temp sensor at the back and front?
UPS & NUT
- NUT-ing onto the UPS still needs to be done.
- There is a Pi connected to the UPS via serial but everything still needs to be setup.
Portal
- We are still missing the portal, what will we do about that?
Ticketing & Kanban board
- A ticketing system that can be both used by BOFH as well as other members/committees would be very nice to have
- A kanban board would be nice as well, although for BOFH we could technically use the one integrated into Gitlab
Gitlab
- We could probably upgrade the version to 18.1?