(→How) |
(→What) |
||
| Line 12: | Line 12: | ||
== What == | == What == | ||
Build a [https://www.gnu.org/software/guix/ GNU Guix] managed cluster that uses the open source [https://arvados.org/ Arvados] environment. | Build a [https://www.gnu.org/software/guix/ GNU Guix] managed cluster that uses flexible on-demand installable nodes and distributed file storage, such as the open source [https://arvados.org/ Arvados] environment. | ||
== How == | == How == | ||
Revision as of 09:39, 13 February 2016
| GNU Guix with Arvados testbed | |
|---|---|
 | |
| Participants | |
| Skills | Sysadmin stuff |
| Status | Active |
| Niche | Software |
| Purpose | Infra |
| Tool | No |
| Location | Super01-08 |
| Cost | |
| Tool category | General |
arvados.png {{#if:No | [[Tool Owner::{{{ProjectParticipants}}} | }} {{#if:No | [[Tool Cost::{{{Cost}}} | }}
What
Build a GNU Guix managed cluster that uses flexible on-demand installable nodes and distributed file storage, such as the open source Arvados environment.
How
The cluster is exactly meant for experiments and we are happy to share.
The concept is a throwaway setup. That means you should write your installation in such a way that it can be regenerated every time. This is done through GNU Guix scripts which allow for VMs, containers and PXE installs on the nodes.
To run experiments needs some co-ordination with Pjotr and perhaps others. Every software deployment and configuration should be captured in Guix.
The idea is NOT to have ad hoc installation of software on the main node (though you are free to mess within your VM) and also NOT to have stuff running full-time.
Think of it as proof-of-concepts which can run for a while. I'll be happy to facilitate with 8x nodes with 8 cores and soon another 8x8.
The base software installation for the master is Debian Wheezy with backports. Otherwise, every tool installed on the cluster will be deployed using GNU Guix, probably inside KVM containers or using PXE. The idea is that this a 'throwaway' setup so everything should be rebuilt on the fly using Guix expressions.
We are very interested in running compute (slurm) and distributed file systems (moosefs, Arvados Keep etc.). If anyone is interested in that... I just submitted a software package definition for slurm to GNU Guix ML.
Who
Pjotr Prins, Joep de Ligt and Roel Janssen are going to work on packaging Arvados and other tools. Quite a few research groups take an active interest in this initiative, e.g., from Wageningen, Utrecht, Groningen, Leiden and Rotterdam.
Progress
Milestones
- The machines are up and running with special thanks to Theo and Joep (Dec 2015)
- Base install of machines by Pjotr (Oct 2015)
Next steps
- Tighten up ssh server and provide an external login (Pjotr, Theo)
- Install GNU Guix on the master node (Pjotr, Roel)
- Mount GNU Guix on the other nodes (Pjotr, Roel)
- Get KVM running (Pjotr)
- Start writing Guix expressions for deployment (Pjotr, Roel)
- Get Arvados captured in Guix expressions (Roel)