KlikoAI: Difference between revisions

From NURDspace
Jump to navigation Jump to search
No edit summary
No edit summary
 
(5 intermediate revisions by 2 users not shown)
Line 6: Line 6:
|Tool=No
|Tool=No
}}
}}
The goal of this project is to have an AI detection of the paper and waste containers inside the space. So that we can detect if they are inside our outside. Previous methods using sensors (distance sensor and a ble sensor) have proven to be unsatisfying.


KlikoAI uses ministral-3:14b, using ollama as the backend. Previously moondream was used with interference running within KlikoAI, however the model had issues with detecting certain container configurations so the decision was made to switch to ministral instead as it appeared to have better performance with this task. It is running on gpu.vm.nurd.space (via docker, in ~/klikoai) and makes use of the Nvidia Geforce RTX 3080.
KlikoAI detects whether the paper and waste containers at NURDspace are inside or outside. Previous approaches using distance sensors (often caused spam on IRC) and BLE sensors proved unreliable (they fell right off after the container got emptied), so the project switched to AI-based vision detection instead.


KlikoAI uses the homeassistant calendar integration to detect when it's a trash pick-up day to switch to a more frequent checking schedule and switching back to once every 6 hours on non trash pickup days. It should also be able to detect when a big chance happens in frame, however this is not yet tested.
KlikoAI uses <code>mistral3:14b</code> via llama-swap with llama.cpp as the backend. Before that, moondream was used with inference running inside KlikoAI itself, but it struggled with certain container configurations. <code>mistral3:14b</code> showed better performance on this task.
 
It runs on gpu.vm.nurd.space (as a systemd service, in <code>/opt/klikoai</code>) on an Nvidia GeForce RTX 3080. (See https://git.nurd.space/bofh/nixos/gpu)
 
KlikoAI uses the Home Assistant calendar integration to detect trash pick-up days and switches to a more frequent scan schedule on those days, falling back to once every 6 hours otherwise. There is also logic to detect significant changes in frame, though this has not yet been tested.
 
State is published as an MQTT sensor to Home Assistant, which allows the "PUT THE TRASH OUTSIDE" notification to be muted for both the space's Signal group and IRC channel once the correct container has been detected outside. There is also IRC integration via the <code>!klikos</code> and <code>!klikoupdate</code> commands (the latter triggers a manual scan).
 
A couple of bright lights controlled by an ESP8266 and a MOSFET have been added to illuminate the containers, as they sit in a fairly dark spot. As a bonus, they make for a good jumpscare when people walk past the space at night. A strobe effect has also been added for partying to gabber in the kliko corner.
 
<gallery>
KlikoAI_klikos.jpg|Klikos in their natural habitat.
KlikoAI_showcasing_lights.jpg|Lights on when inference is running
KlikoAI_lights_controller.jpg|ESP8266 + N-channel MOSFET controlling the lights
KlikoAI_lights_and_camera.jpg|Camera and light mounting
1b925f63993602e21b367f81270d818c1c4a0860b896b46ce5be2684d0a33952.png|What the camera sees
</gallery>
 
<syntaxhighlight>
<@Melan> !klikoupdate
<@nurdbot> [Klikos] Running kliko inference task...
<@nurdbot> [Klikos] Paper: Visible Waste: Visible | Last Update: 2026-04-14 19:52:04 | Avg time: 41.48 secs
</syntaxhighlight>
 
The inference time is expected: it runs on a shared GPU machine and the model is loaded and unloaded dynamically, which is included in the total inference time.
 
We use the following prompt:
<syntaxhighlight>
Look carefully at this image.
I can see garbage containers. Tell me which colored LIDS are visible:
 
1. Is there a YELLOW or cream colored lid?
2. Is there a BLUE or dark blue colored lid?
 
Reply ONLY with valid JSON using double quotes:
{"yellow": true, "blue": true}
</syntaxhighlight>


KlikoAI has hass-integration and outputs it's states as a mqtt sensor. This makes it possible to mute the "PUT THE TRASH OUTSIDE" notifications once the correct container has been put outside. There is also IRC integration using the command !klikos and !klikoupdate (to trigger a scan).


== Previous Iteration ==
== Previous Iteration ==
The goal at first was to have a YOLO model trained on an existing dataset containing containers. However, due to a "limited" way of mounting the camera, detection results were very poor which made me decide to use an LLM vision model instead.


=== dataset ===
The original plan was to train a YOLO model on an existing garbage container dataset. Due to the limited camera mounting options at the space, detection results were poor, which led to switching to an LLM vision model.
We want to detect either one of the bins, our bins have a blue lid and a yellow lid while the rest of the bin is green.
 
=== Dataset ===
 
The bins at NURDspace have a blue lid and a yellow lid on an otherwise green body. Finding training images with those specific lid colours turned out to be harder than expected.


Though Its seems that gathering images with containers with these lid colours appear to be more difficult than initially hoped.
YOLO returns bounding boxes, so one option would have been to inspect just the top portion of the detected box and check the colour range, or determine bin position based on where it appears in frame.
But since Yolo returns a bounding box, we could pretty easily look at the top of the container and calculate in which colour range it falls,  
or determine the location of the bin based on where it is in view.


The default Yolo models don't support garbage bin detection, so I trained my own model on my 5090 using this dataset https://universe.roboflow.com/finance-insitut/garbage-container-detection-sam7i
Since the default YOLO models do not support garbage bin detection, a custom model was trained on a 5090 using [https://universe.roboflow.com/finance-insitut/garbage-container-detection-sam7i this dataset from Roboflow].


<gallery>
<gallery>
2026-01-14_17-08-31.jpg|Before training
2026-01-14_17-08-31.jpg|Before training
2026-01-14_18-02-49.jpg|After training  
2026-01-14_18-02-49.jpg|After training
</gallery>
</gallery>

Latest revision as of 20:08, 14 April 2026

KlikoAI
KlikoAI.jpg
Participants Melan
Skills
Status Active
Niche Smart stuff
Purpose Home Automation
Tool No
Location
Cost
Tool category

KlikoAI Property "Tool Image" (as page type) with input value "File:{{{Picture}}}" contains invalid characters or is incomplete and therefore can cause unexpected results during a query or annotation process. {{{Picture}}} {{#if:No | [[Tool Owner::{{{ProjectParticipants}}} | }} {{#if:No | [[Tool Cost::{{{Cost}}} | }}


KlikoAI detects whether the paper and waste containers at NURDspace are inside or outside. Previous approaches using distance sensors (often caused spam on IRC) and BLE sensors proved unreliable (they fell right off after the container got emptied), so the project switched to AI-based vision detection instead.

KlikoAI uses mistral3:14b via llama-swap with llama.cpp as the backend. Before that, moondream was used with inference running inside KlikoAI itself, but it struggled with certain container configurations. mistral3:14b showed better performance on this task.

It runs on gpu.vm.nurd.space (as a systemd service, in /opt/klikoai) on an Nvidia GeForce RTX 3080. (See https://git.nurd.space/bofh/nixos/gpu)

KlikoAI uses the Home Assistant calendar integration to detect trash pick-up days and switches to a more frequent scan schedule on those days, falling back to once every 6 hours otherwise. There is also logic to detect significant changes in frame, though this has not yet been tested.

State is published as an MQTT sensor to Home Assistant, which allows the "PUT THE TRASH OUTSIDE" notification to be muted for both the space's Signal group and IRC channel once the correct container has been detected outside. There is also IRC integration via the !klikos and !klikoupdate commands (the latter triggers a manual scan).

A couple of bright lights controlled by an ESP8266 and a MOSFET have been added to illuminate the containers, as they sit in a fairly dark spot. As a bonus, they make for a good jumpscare when people walk past the space at night. A strobe effect has also been added for partying to gabber in the kliko corner.

<@Melan> !klikoupdate
<@nurdbot> [Klikos] Running kliko inference task...
<@nurdbot> [Klikos] Paper: Visible Waste: Visible | Last Update: 2026-04-14 19:52:04 | Avg time: 41.48 secs

The inference time is expected: it runs on a shared GPU machine and the model is loaded and unloaded dynamically, which is included in the total inference time.

We use the following prompt:

Look carefully at this image.
I can see garbage containers. Tell me which colored LIDS are visible:

1. Is there a YELLOW or cream colored lid?
2. Is there a BLUE or dark blue colored lid?

Reply ONLY with valid JSON using double quotes:
{"yellow": true, "blue": true}


Previous Iteration

The original plan was to train a YOLO model on an existing garbage container dataset. Due to the limited camera mounting options at the space, detection results were poor, which led to switching to an LLM vision model.

Dataset

The bins at NURDspace have a blue lid and a yellow lid on an otherwise green body. Finding training images with those specific lid colours turned out to be harder than expected.

YOLO returns bounding boxes, so one option would have been to inspect just the top portion of the detected box and check the colour range, or determine bin position based on where it appears in frame.

Since the default YOLO models do not support garbage bin detection, a custom model was trained on a 5090 using this dataset from Roboflow.