BOFH Meeting 19 July 2025: Difference between revisions
Jump to navigation
Jump to search
(→Gitlab) |
|||
| (13 intermediate revisions by 2 users not shown) | |||
| Line 1: | Line 1: | ||
= Meeting notes = | = Meeting notes = | ||
Note (r3boot): I added a priority to the points from my PoV. 1 == highest, 10 is lowest | |||
== Monitoring == | == Monitoring == | ||
| Line 10: | Line 12: | ||
** Ready support for all kinds of notification systems | ** Ready support for all kinds of notification systems | ||
* Prio: 5/10 | * Prio: 5/10 | ||
* Action: r3boot is building this | |||
== Kitchen Rack & Cabling == | == Kitchen Rack & Cabling == | ||
| Line 15: | Line 18: | ||
* We should probably check what cabling is still in use and which ones are defunct at this point due to mentioned renovations. (Think any link going to zaal 2). | * We should probably check what cabling is still in use and which ones are defunct at this point due to mentioned renovations. (Think any link going to zaal 2). | ||
* Perhaps we could extend the cabling going to zaal 1 with RJ45 extenders for the time being? | * Perhaps we could extend the cabling going to zaal 1 with RJ45 extenders for the time being? | ||
* Prio: 3/10 | |||
* Action: Rubberhark (and r3boot when available) | |||
== SSL Certificates == | == SSL Certificates == | ||
| Line 20: | Line 25: | ||
** For example, SSL certificates for uptime-kuma running at docker-infra. | ** For example, SSL certificates for uptime-kuma running at docker-infra. | ||
** Make it easy for members to deploy something that needs HTTPS? | ** Make it easy for members to deploy something that needs HTTPS? | ||
* This already works via revproxy.vm.nurd.space | |||
* Scripts for internal PKI can be documented on the wiki | |||
* Prio: 8/10 | |||
* No action | |||
== Script Management == | == Script Management == | ||
| Line 26: | Line 35: | ||
* Having it all consolidated in one system will simplify management greatly. | * Having it all consolidated in one system will simplify management greatly. | ||
* Think Docker but more simplified. Maybe solutions for this already exist? | * Think Docker but more simplified. Maybe solutions for this already exist? | ||
* Deploy on a VM using ansible OR store all of them in a git repo and deploy that on a host | |||
** By creating a "chokepoint" you get a centralized place to combine all functionality required for a script (unit file, monitoring, etc) | |||
* Prio: 5/10 | |||
== Harkbot Migration == | == Harkbot Migration == | ||
* Switch harkbot from LXC (Debian) to a VM (AlmaLinux)? | * Switch harkbot from LXC (Debian) to a VM (AlmaLinux)? | ||
* Could also roll it out via OpenTofu & Ansible stack then. | * Could also roll it out via OpenTofu & Ansible stack then. | ||
* Prio: 5/10 | |||
== MikroTik Licensing == | == MikroTik Licensing == | ||
| Line 42: | Line 55: | ||
** https://www.mikrotik-store.eu/en/mikrotik-cloud-hosted-router-chr-p1-license | ** https://www.mikrotik-store.eu/en/mikrotik-cloud-hosted-router-chr-p1-license | ||
* https://www.dectdirect.nl/nl/crs310-8g2sin.html (€190 MikroTik router for example) | * https://www.dectdirect.nl/nl/crs310-8g2sin.html (€190 MikroTik router for example) | ||
* Meh .. +1 for OPNsense .. It works, and its predictable. Its also boring, but thats okay. | |||
* Prio: 2/10 | |||
== BOFH Password Management == | == BOFH Password Management == | ||
| Line 47: | Line 62: | ||
* It would be great if we stopped using the standard insecure passwords, lol. | * It would be great if we stopped using the standard insecure passwords, lol. | ||
* Can it also contain people's SSH certs and auto-deploy them to BOFH-managed devices? | * Can it also contain people's SSH certs and auto-deploy them to BOFH-managed devices? | ||
* See vaultwarden.vm.nurd.space, but we need to start moving & resetting passwords | |||
* Prio: 3/10 | |||
* | |||
== EPC MQTT Bridge == | == EPC MQTT Bridge == | ||
* The EpcMqttBridge probably needs a rewrite and not yeet itself out of existence the moment one of the EPCs goes offline. | * The EpcMqttBridge probably needs a rewrite and not yeet itself out of existence the moment one of the EPCs goes offline. | ||
* They are also fairly slow so maybe we could look into replacing the brains with a network-based ESP32 instead? | * They are also fairly slow so maybe we could look into replacing the brains with a network-based ESP32 instead? | ||
* Prio: 5/10 | |||
== Erratic Server Power Usage == | == Erratic Server Power Usage == | ||
* Erratic is gobbling up power like Mia Khalifa on OnlyFans. Wat do? | * Erratic is gobbling up power like Mia Khalifa on OnlyFans. Wat do? | ||
* I'd like to keep two systems for redundancy, but we could replace the hardware with something that gobbles less power while keeping or even increasing the performance, if we were to switch to consumer hardware. | * I'd like to keep two systems for redundancy, but we could replace the hardware with something that gobbles less power while keeping or even increasing the performance, if we were to switch to consumer hardware. | ||
* | * Make BOM and we can discuss it. | ||
* See https://nurdspace.nl/Erratic_V2_BOM | |||
* Prio: 2/10 | |||
* | |||
* | |||
== Consolidation Considerations == | == Consolidation Considerations == | ||
| Line 77: | Line 86: | ||
* Camera recording would also need to be handled by Erratic then (right now it goes to the 2×2TB in Harmony). | * Camera recording would also need to be handled by Erratic then (right now it goes to the 2×2TB in Harmony). | ||
** Though if we are going with more than two cameras, we'll need to anyway. | ** Though if we are going with more than two cameras, we'll need to anyway. | ||
* Prio: 4/10 | |||
== GPON Power == | == GPON Power == | ||
* The GXPN power supply still needs to be connected to the door UPS system. | * The GXPN power supply still needs to be connected to the door UPS system. | ||
* Prio; 8/10 | |||
== Matrix Setup == | == Matrix Setup == | ||
* We still have to roll out the Matrix setup. | * We still have to roll out the Matrix setup. | ||
* A start has been made here: https://git.nurd.space/bofh/opentofu/machines/matrix | * A start has been made here: https://git.nurd.space/bofh/opentofu/machines/matrix | ||
* Prio: 4/10 | |||
== MinIO S3 == | == MinIO S3 == | ||
| Line 89: | Line 101: | ||
* Are we going to switch to something else? Garage? | * Are we going to switch to something else? Garage? | ||
* There is also https://github.com/OpenMaxIO/openmaxio-object-browser that we could roll out and keep using Minio. | * There is also https://github.com/OpenMaxIO/openmaxio-object-browser that we could roll out and keep using Minio. | ||
* Prio: 7/10 | |||
== Home Assistant == | == Home Assistant == | ||
* Currently our Home Assistant setup is pretty clunky. | * Currently our Home Assistant setup is pretty clunky. | ||
* What if we redeployed a new instance to `sporadic` and started anew? | * What if we redeployed a new instance to `sporadic` and started anew? | ||
* Prio: 8/10 | |||
== Rack Monitoring == | == Rack Monitoring == | ||
* Rack monitoring? It might be nice to have a temp sensor at the back and front? | * Rack monitoring? It might be nice to have a temp sensor at the back and front? | ||
* Prio: 7/10 | |||
== UPS & NUT == | == UPS & NUT == | ||
* NUT-ing onto the UPS still needs to be done. | * NUT-ing onto the UPS still needs to be done. | ||
* There is a Pi connected to the UPS via serial but everything still needs to be setup. | * There is a Pi connected to the UPS via serial but everything still needs to be setup. | ||
* Prio: 6/10 | |||
== Portal == | == Portal == | ||
* We are still missing the portal, what will we do about that? | * We are still missing the portal, what will we do about that? | ||
* There is a PoC with plugin support, written in python + easygui | |||
* Prio: 6/10 | |||
== Ticketing & Kanban board == | == Ticketing & Kanban board == | ||
* A ticketing system that can be both used by BOFH as well as other members/committees would be very nice to have | * A ticketing system that can be both used by BOFH as well as other members/committees would be very nice to have | ||
* A kanban board would be nice as well, although for BOFH we could technically use the one integrated into Gitlab | * A kanban board would be nice as well, although for BOFH we could technically use the one integrated into Gitlab | ||
* Prio: 2/10 | |||
I've found the following: | |||
<nowiki> | |||
# Ticketing | |||
- https://github.com/znuny/Znuny?tab=readme-ov-file | |||
- https://github.com/makeplane/plane | |||
- https://zammad.org/ | |||
- https://www.jetbrains.com/youtrack/ | |||
# Kanban / todo | |||
- https://www.focalboard.com/ | |||
- https://github.com/donetick/donetick | |||
- https://vikunja.io/ | |||
- https://wekan.fi/ | |||
- https://kanboard.org/ | |||
- [x] https://planka.app/ | |||
- https://leantime.io/ | |||
# Wiki | |||
- https://www.xwiki.org/xwiki/bin/view/Main/WebHome | |||
- https://js.wiki | |||
- https://otterwiki.com/ | |||
- https://docmost.com/ | |||
- https://www.bookstackapp.com/ | |||
https://github.com/Unitech/pm2 | |||
</nowiki> | |||
== Gitlab == | == Gitlab == | ||
* We could probably upgrade the version to 18.1? | * We could probably upgrade the version to 18.1? | ||
* Requires some steps likely, beware | |||
* Also update gitlab-runners | |||
* Also beware of ACL changes in 18 | |||
* Prio: 3/10 | |||
* See https://gitlab-com.gitlab.io/support/toolbox/upgrade-path/ | |||
== Keycloak == | |||
* Newer version breaks with our ancient SAML setup | |||
* Switching to oidc2 is possible but can be tricky to get user mapping correct | |||
* Prio: 3/10 | |||
== The Wiki == | |||
* There have been often complains of how the wiki currently is. | |||
* Do we want to do something about that? Or put someone on it? | |||
Latest revision as of 21:26, 19 July 2025
Meeting notes
Note (r3boot): I added a priority to the points from my PoV. 1 == highest, 10 is lowest
Monitoring
- Monitoring stack with alerting (we have uptime-kuma for non-bofh), but what else do we want?
- Folkert mentioned Icinga2 and that is also what we ran before.
- Prometheus + Grafana + Alertmanager etc is also a good solution.
- The latter also gives us Grafana Loki (log aggregation system).
- Prometheus gives us both metrics & monitoring
- Writing alerts is easier then deploying a nagios/icinga check
- Ready support for all kinds of notification systems
- Prio: 5/10
- Action: r3boot is building this
Kitchen Rack & Cabling
- Moving the kitchen rack before renovations begin.
- We should probably check what cabling is still in use and which ones are defunct at this point due to mentioned renovations. (Think any link going to zaal 2).
- Perhaps we could extend the cabling going to zaal 1 with RJ45 extenders for the time being?
- Prio: 3/10
- Action: Rubberhark (and r3boot when available)
SSL Certificates
- Rolling out Let's Encrypt certificates for on and off site infra.
- For example, SSL certificates for uptime-kuma running at docker-infra.
- Make it easy for members to deploy something that needs HTTPS?
- This already works via revproxy.vm.nurd.space
- Scripts for internal PKI can be documented on the wiki
- Prio: 8/10
- No action
Script Management
- We seem to have a lot of little scripts running but they mostly get deployed on the harkbot LXC.
- Maybe we could make a proper VM for this instead and some sort of system that can monitor these scripts and also provide a description of what it actually does.
- Having it all consolidated in one system will simplify management greatly.
- Think Docker but more simplified. Maybe solutions for this already exist?
- Deploy on a VM using ansible OR store all of them in a git repo and deploy that on a host
- By creating a "chokepoint" you get a centralized place to combine all functionality required for a script (unit file, monitoring, etc)
- Prio: 5/10
Harkbot Migration
- Switch harkbot from LXC (Debian) to a VM (AlmaLinux)?
- Could also roll it out via OpenTofu & Ansible stack then.
- Prio: 5/10
MikroTik Licensing
- Our MikroTik trial license will expire this month.
- What shall we do?
- Switch back to OPNsense?
- Buy the license?
- Switch to a hardware MikroTik router?
- We were not happy with how blocked down the Nightmare (dream) machine was.
- OPNsense is OK but also has its quirks.
- Buying a P1 CHR license (1Gbit) is around €38.
- https://www.dectdirect.nl/nl/crs310-8g2sin.html (€190 MikroTik router for example)
- Meh .. +1 for OPNsense .. It works, and its predictable. Its also boring, but thats okay.
- Prio: 2/10
BOFH Password Management
- Start using Bitwarden for BOFH password management PLEASE.
- It would be great if we stopped using the standard insecure passwords, lol.
- Can it also contain people's SSH certs and auto-deploy them to BOFH-managed devices?
- See vaultwarden.vm.nurd.space, but we need to start moving & resetting passwords
- Prio: 3/10
EPC MQTT Bridge
- The EpcMqttBridge probably needs a rewrite and not yeet itself out of existence the moment one of the EPCs goes offline.
- They are also fairly slow so maybe we could look into replacing the brains with a network-based ESP32 instead?
- Prio: 5/10
Erratic Server Power Usage
- Erratic is gobbling up power like Mia Khalifa on OnlyFans. Wat do?
- I'd like to keep two systems for redundancy, but we could replace the hardware with something that gobbles less power while keeping or even increasing the performance, if we were to switch to consumer hardware.
- Make BOM and we can discuss it.
- Prio: 2/10
Consolidation Considerations
- If we upgrade Erratic, do we also want to move Harmony's drives to Erratic?
- Right now Harmony does about 50~60W (estimated), we could probably reduce the rack total with a few watts if we consolidate them.
- We probably would need an extra SATA controller for 6 extra drives; this might end up not reducing the power usage by much in the end.
- Giving Erratic a 10Gbps NIC would mean we can increase max space storage throughput.
- Camera recording would also need to be handled by Erratic then (right now it goes to the 2×2TB in Harmony).
- Though if we are going with more than two cameras, we'll need to anyway.
- Prio: 4/10
GPON Power
- The GXPN power supply still needs to be connected to the door UPS system.
- Prio; 8/10
Matrix Setup
- We still have to roll out the Matrix setup.
- A start has been made here: https://git.nurd.space/bofh/opentofu/machines/matrix
- Prio: 4/10
MinIO S3
- Minio S3 pulled a "Everyone disliked that".
- Are we going to switch to something else? Garage?
- There is also https://github.com/OpenMaxIO/openmaxio-object-browser that we could roll out and keep using Minio.
- Prio: 7/10
Home Assistant
- Currently our Home Assistant setup is pretty clunky.
- What if we redeployed a new instance to `sporadic` and started anew?
- Prio: 8/10
Rack Monitoring
- Rack monitoring? It might be nice to have a temp sensor at the back and front?
- Prio: 7/10
UPS & NUT
- NUT-ing onto the UPS still needs to be done.
- There is a Pi connected to the UPS via serial but everything still needs to be setup.
- Prio: 6/10
Portal
- We are still missing the portal, what will we do about that?
- There is a PoC with plugin support, written in python + easygui
- Prio: 6/10
Ticketing & Kanban board
- A ticketing system that can be both used by BOFH as well as other members/committees would be very nice to have
- A kanban board would be nice as well, although for BOFH we could technically use the one integrated into Gitlab
- Prio: 2/10
I've found the following:
# Ticketing - https://github.com/znuny/Znuny?tab=readme-ov-file - https://github.com/makeplane/plane - https://zammad.org/ - https://www.jetbrains.com/youtrack/ # Kanban / todo - https://www.focalboard.com/ - https://github.com/donetick/donetick - https://vikunja.io/ - https://wekan.fi/ - https://kanboard.org/ - [x] https://planka.app/ - https://leantime.io/ # Wiki - https://www.xwiki.org/xwiki/bin/view/Main/WebHome - https://js.wiki - https://otterwiki.com/ - https://docmost.com/ - https://www.bookstackapp.com/ https://github.com/Unitech/pm2
Gitlab
- We could probably upgrade the version to 18.1?
- Requires some steps likely, beware
- Also update gitlab-runners
- Also beware of ACL changes in 18
- Prio: 3/10
Keycloak
- Newer version breaks with our ancient SAML setup
- Switching to oidc2 is possible but can be tricky to get user mapping correct
- Prio: 3/10
The Wiki
- There have been often complains of how the wiki currently is.
- Do we want to do something about that? Or put someone on it?