brandon/oam
1
0
Fork 0
mirror of https://gitea.com/mcereda/oam.git synced 2026-02-09 05:44:23 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
16f68d0819d901617e16d39ceeff3ff58e2e4be1
oam/knowledge base/beowulf cluster.md

5.9 KiB
Raw Blame History

Beowulf cluster

Multi-computer architecture which can be used for parallel computations.
It is usually composed of commodity, non custom hardware and software components and is trivially reproducible, like any PC capable of running a Unix-like operating system with standard Ethernet adapters and switches.

The cluster usually consists of one server node, and one or more client nodes connected via some kind of network.
The server controls the whole cluster, and provides files to the clients. It is also the cluster's console and gateway to the outside world. Large Beowulf machines might have more than one server node, and possibly other nodes dedicated to particular tasks like consoles or monitoring stations.
In most cases, client nodes in a Beowulf system are dumb, and the dumber the better. Clients are configured and controlled by the server, and do only what they are told to do.

Beowulf clusters behave more like a single machine rather than many workstations: nodes can be thought of as a CPU and memory package which is plugged into the cluster, much like a CPU or memory module can be plugged into a motherboard.

Beowulf is no more than a technology of clustering computers to form a parallel, virtual supercomputer. One can build a Beowulf class machine using a standard Linux distribution without any additional software; two networked computers sharing a folder via NFS and which trust each other to execute remote shells can be considered a two node Beowulf machine.

Table of contents

  1. Scheduler
  2. Create a quick and dirty cluster
  3. Further readings
  4. Sources

Scheduler

Takes care of scheduling the jobs and juggling the resources in the cluster.
The most used one at the time of writing is Slurm.

Create a quick and dirty cluster

Uses MPICH on Linux.
Just follow this procedure:

  1. prepare at least 2 Linux hosts
  2. assign a fixed, known IP address to all the hosts
  3. create a file on the server node listing the IP addresses of all the client nodes (e.g. machines_file)
  4. install, enable and start SSH on all the hosts
  5. configure SSH on all the hosts to let the server node connect to all the client nodes without using a password
  6. install MPICH on all the hosts, possibly the same version
  7. test the installation: 
    # execute `hostname` on all hosts
mpiexec -f 'machines_file' -n 'number_of_processes' 'hostname'

See the Vagrant example.

Further readings

Sources