Naaimachine: Difference between revisions

From NURDspace
Jump to navigation Jump to search
No edit summary
No edit summary
 
Line 8: Line 8:
|Category=Infrastructure
|Category=Infrastructure
}}
}}
(Na)AI machine, is a system to be used for AI at the space located inside the rack near the door of zaal 1.
(Na)AI machine, was a dedicated system to be used for AI at the space located inside the rack near the door of zaal 1.  
It's recently been replaced by a GPU inside Erratic that's available 24/7


It’s mainly used to run an LLM and Stable Diffusion but it can also be used for other things that need GPU power as well. If you want to use it for something, don’t hesitate to ask Melan about it!
Address: gpu.vm.nurd.space (10.208.1.20)
 
Address: gpu.lan.nurd.space (10.208.1.20) # suspend &amp; waking up The system is designed to suspend 15 minutes after there has been no GPU load. You can also put the system to sleep with <code>systemctl suspend</code>
 
Waking up is done by a small script running in an LXC on Erratic which provides a small API-endpoint. You can see the state of the AI-machine and it’s open ports by going to <code>http://gpu-wake.dhcp.nurd.space:8000/</code>
 
Waking up is done by making a GET request to <code>http://gpu-wake.dhcp.nurd.space:8000/wake</code> or sending a magic WOL packet to <code>08:62:66:81:14:6f</code>
 
<span id="specs"></span>
= Specs =
 
The system is equipped with a <code>Intel Xeon E5-2643 v4 @ 3.700GHz</code> which is a six-core processor. The system also has <code>32GB</code> of DDR4 memory available. Primary storage is provided by a fast <code>500GB NVME</code>, while the secondary storage is a <code>512GB SSD</code> mounted under <code>/mnt/storage</code> all powered by an <code>Asus SABERTOOTH X99</code>


The system has the following GPUs available:
The system has the following GPUs available:


- NVIDIA GeForce RTX 3080 (10GB vram)  
- NVIDIA GeForce RTX 3080 (10GB vram)  
- NVIDIA P104-100 (8GB vram)


The system is running <code>Archlinux</code>, with the latest kernel and Nvidia drivers as well as <code>Cuda 12</code>.
The system is running <code>Archlinux</code>, with the latest kernel and Nvidia drivers as well as <code>Cuda 12</code>.
Line 37: Line 25:
=== LLM (Large Language Model) ===
=== LLM (Large Language Model) ===


One of the AI machine’s primary usages is the LLM. It’s integrated with Nurdbot. For this we use https://github.com/LostRuins/koboldcpp
One of the AI machine’s primary usages is the LLM. It’s integrated with Nurdbot. For this we use Ollama.
 
You can talk to the LLM by going to <code>http://gpu.lan.nurd.space:5001/</code>the api-endpoint is available at <code>http://gpu.lan.nurd.space:5001/api</code>
 
As of right now, the model we are running is https://huggingface.co/TheBloke/Wizard-Vicuna-13B-Uncensored-GGML (<code>Wizard-Vicuna-13B-Uncensored.ggmlv3.q4_0.bin</code>)
 
(At the time of writing, the extra parameters used right now are <code>-usecublas 0 --threads 10 --gpulayers 40</code>. However, more tweaking is required. It’s also running on the first GPU but I want to test moving it to the second GPU)
 
<span id="stable-diffusion"></span>
=== Stable Diffusion ===
 
Right now this is still work in progress.
 
<span id="other"></span>
= Other =
 
<span id="setup-notes"></span>
== Setup notes ==
 
In order for the system to be able come back out of suspend properly with the Nvidia cards remembering their state, these settings have to be enabled.
 
<syntaxhighlight lang="bash">sudo systemctl enable nvidia-suspend.service
sudo systemctl enable nvidia-hibernate.service
sudo systemctl enable nvidia-resume.service
 
echo "options nvidia NVreg_PreserveVideoMemoryAllocations=1" >> /lib/modprobe.d/systemd.conf
 
sudo dracut --force</syntaxhighlight>

Latest revision as of 17:00, 17 January 2025

gpu
Owner Space
Status Infra
Hostname gpu.lan.nurd.space
Tool Yes
Tool category Infrastructure

gpu

naaimachine.jpg {{{InventoryOwner}}}Property "Tool Owner" (as page type) with input value "{{{InventoryOwner}}}" contains invalid characters or is incomplete and therefore can cause unexpected results during a query or annotation process. {{{Location}}}Property "Tool Location" (as page type) with input value "{{{Location}}}" contains invalid characters or is incomplete and therefore can cause unexpected results during a query or annotation process.

(Na)AI machine, was a dedicated system to be used for AI at the space located inside the rack near the door of zaal 1. It's recently been replaced by a GPU inside Erratic that's available 24/7

Address: gpu.vm.nurd.space (10.208.1.20)

The system has the following GPUs available:

- NVIDIA GeForce RTX 3080 (10GB vram)

The system is running Archlinux, with the latest kernel and Nvidia drivers as well as Cuda 12.

AI models

LLM (Large Language Model)

One of the AI machine’s primary usages is the LLM. It’s integrated with Nurdbot. For this we use Ollama.