Using the Darwin cluster

Darwin Cluster Wiki

Setting up passwordless SSH
Darwin specifications
Logging in
Compiling MPI and OpenMP programs
Running MPI programs
Timing your runs
MPI example
OpenMP example
Julia example
Emacs setup
Julia MPI

Setting up passwordless SSH

Being able to move among nodes without entering a password will make your life much more enjoyable.
  1. Run ssh-keygen on a Unix machine you typically use (can be Athena, or a Darwin cluster head node). Accept the default settings and leave the passphrase blank. This creates files in ~/.ssh
  2. Append the contents of ~/.ssh/ to the file ~/.ssh/authorized_keys on each machine you want to be able to connect to. The Darwin cluster nodes share a filesystem, so you only need to do this once on one of the head nodes.
  3. Copy the file ~/.ssh/id_rsa (private key) to the same location on each machine you want to connect from. Again, once this file is in ~/.ssh on Darwin you are all set for the cluster.
  4. The file id_rsa needs to have strict permissions. If you get errors about this, run chmod 400 id_rsa.
You can copy files to remote machines using scp:
scp file user@machine:path

Darwin specifications

  • Darwin compute cluster
    • Head node:
    • Nodes: 128
    • Per-node CPU: 2 x 2-core 3.00GHz Intel(R) Xeon(TM), 4MB L2 cache
    • Per-node memory: 8 GB
  • Darwin visualization cluster
    • Head node:
    • Nodes: 60
    • Per-node CPU: 2 x 2-core 2.66GHz Intel(R) Xeon(TM), 4MB L2 cache
    • Per-node memory: 6 GB
On Linux, hardware info is available via less /proc/cpuinfo and less /proc/meminfo.
The top command is useful to see how much CPU and memory everybody's jobs are using. In top, hit "M" to sort by memory use. Hit "u" and type your username to see just your processes.

Logging In

ssh to or with your athena username and password.

If you are on a windows machine, you can download SecureCRT or Putty from MIT, MIT certificate required). Or you can use any other SSH client.

Compiling MPI and OpenMP programs

For MPI programs, use mpicc for C programs, mpiCC for C++ programs, mpif77 for fortran programs, as follows:
mpicc mytest.c -o test
mpiCC -o test
mpif77 mytest.f -o test

For OpenMP programs you need to use the Intel Fortran/C Compilers. Do the following on beagle:
source /opt/intel/cc/9.1.051/bin/
icc -O -openmp -o test
ifc -O -openmp -o test mytest.f

Running MPI programs

After compiling MPI programs, use qsub and mpirun to execute:

echo '/opt/openmpi/bin/mpirun -np # /path/to/yourMPIProgram' | qsub -pe mpich_mx #

where # is the number of processors and yourMPIProgram is the MPI executable to be run. qsub is the interface to the Sun Grid Engine batch queue, which ensures that the cluster is shared fairly.

Anything printed to standard out will be available in a file in your home directory called STDIN.oID where ID is the job ID. STDIN refers to the fact that the job description was provided on standard input via a pipe instead of in a shell script file. The standard error stream is in a similar file, named with an e instead of the o.

When using a private cluster, one can specify machines to use manually using a hostfile, as follows:

mpirun -np # -machinefile hostfile ./yourMPIProgram

"hostfile" is a text file listing all the hosts you want to use. It might look like this:


Look at /etc/hosts to see the names of the compute nodes.

Timing your runs

You can use the command time to time your runs. For example,

time ./myprog

returns something like

real 0m0.958s
user 0m0.180s
sys 0m0.170s

real is the wall clock time elapsed.

Timing MPI programs is similar,

time mpirun -np 4 ./hello++

This method is not very accurate. For accurate timing, use the function MPI_WTime

MPI example

#include "mpi.h"
#include "unistd.h"
#include "stdio.h"

int main(int argc, char **argv)
  char hostname[256];
  int size,rank;

  MPI_Init(&argc, &argv);

  MPI_Comm_size(MPI_COMM_WORLD, &size);
  MPI_Comm_rank(MPI_COMM_WORLD, &rank);

  gethostname(hostname, sizeof(hostname));

  printf("Hi, I am %d of %d and my hostname is %s\n", rank, size, hostname);


OpenMP example

#include <math.h>
#include <stdio.h>

#define N 16384
#define M 10

double dotproduct(int, double *);

double dotproduct(int i, double *x)
  double temp=0.0, denom;
  int j;

  for (j=0; j<N; j++) {
    // zero based!!
    denom = (i+j)*(i+j+1)/2 + i+1;
    temp = temp + x[j]*(1/denom);  

  return temp;

int main()
  double *x = new double[N];
  double *y = new double[N];
  double eig = sqrt(N);
  double denom,temp;
  int i,j,k;

  for (i=0; i<N; i++) {
    x[i] = 1/eig;

  for (k=0;k<M;k++) {


    // compute y = Ax
#pragma omp parallel for shared(y)
    for (i=0; i<N; i++) {     
      y[i] = dotproduct(i,x);
    // find largest eigenvalue of y
    eig = 0;    
    for (i=0; i<N; i++) {
      eig = eig + y[i]*y[i];
    eig = sqrt(eig);
    printf("The largest eigenvalue after %2d iteration is %16.15e\n",k+1, eig);

    // normalize
    for (i=0; i<N; i++) {
      x[i] = y[i]/eig;

Julia example

A regularly-updated copy of Julia is available on Darwin under /home/bezanson/julia. One of Julia's charming features is that it can be run by anyone, from anywhere, without any install or setup. Type /home/bezanson/julia/julia to run it. There is also a world-writable directory /home/bezanson/julia/data in case that is useful.

You can add processors to Julia's pool from the prompt as follows:

julia> @time @parallel (+) for i=1:10000000; randn(); end
elapsed time: 1.0429379940032959 seconds

julia> @time @parallel (+) for i=1:10000000; randn(); end
elapsed time: 0.58234810829162598 seconds

julia> addprocs_sge(2)

julia> @time @parallel (+) for i=1:10000000; randn(); end
elapsed time: 3.28881287574768066 seconds

julia> @time @parallel (+) for i=1:10000000; randn(); end
elapsed time: 0.31914687156677246 seconds

Here we summed 10000000 normal-random numbers using a parallel for loop. addprocs_sge requests nodes from the Sun Grid Engine batch queue. Julia can also start processes via SSH, provided you have passwordless access to the nodes in question, using addprocs_ssh with a cell array of machine names. However, one should not do this on a shared cluster managed by a batch queue.

Note one minor annoyance: due to Julia's Just-In-Time (JIT) compilation model, things take longer the first time you do them. The extra delay is typically about 1-3 seconds. As the code cache warms up over a session, these delays become shorter and rarer.

(Optional!) Feel free to check out and build your own copy of Julia from github. We recommend this in order to stay up to date and make it easy to send patches. If you'd like to get a little deeper into Julia and possibly even contribute changes, fixes, example code, etc. you should make your own github account (if you don't have one already).

Emacs setup

An emacs mode for Julia is available. Copy /home/bezanson/.emacs to your home directory, or add the following to your .emacs file:

(require 'julia-mode "/home/bezanson/julia/contrib/julia-mode.el")

Julia MPI

There are MPI bindings available for Julia. See /home/bezanson/julia-mpi/examples for examples. Use a command like the following to run a Julia MPI program:

mpirun -np # /home/bezanson/.julia/mpi/juliampi sourcefile.jl

You will need to add /home/bezanson/bin to your path, or make a link to /home/bezanson/julia/julia in some directory in your path, such as $HOME/bin.

The MPI bindings are maintained by Lucas Wilcox in
a separate github repository.