What is Singularity?

From the Singularity Homepage:

Singularity enables users to have full control of their environment. Singularity containers can be used to package entire scientific workflows, software and libraries, and even data. This means that you don’t have to ask your cluster admin to install anything for you – you can put it in a Singularity container and run. Did you already invest in Docker? The Singularity software can import your Docker images without having Docker installed or being a superuser. Need to share your code? Put it in a Singularity container and your collaborator won’t have to go through the pain of installing missing dependencies. Do you need to run a different operating system entirely? You can “swap out” the operating system on your host for a different one within a Singularity container. As the user, you are in control of the extent to which your container interacts with its host. There can be seamless integration, or little to no communication at all. What does your workflow look like?

Why use Singularity on MARCC?

There are several reasons to consider using containers on MARCC:
Consider that most of MARCC hardware is using CentOS 6

  1. Software development was performed on Ubuntu or RHEL7 (or OS that is not CentOS 6)
  2. Working with proprietary licensed Linux software that has only support in other Linux distributions
  3. Need flexibility to work with software immediately on other HPC hardware that runs with Singularity
  4. Software metadata files are unmanageable (e.g. installing software with Python, R, conda) – the use of a Singularity container allows for the use of a single compressed image file.
  5. You want to run software quickly because software is rapidly evolving (e.g. TensorFlow PyTorch)

We are actively supporting the use of Singularity to support a variety of software.  Sometimes containers are packaged inside of Lmod modules, so that users don’t have to actively worry about the use of Singularity commands.  However, it is very quick to learn how to use Singularity.

What are some examples of running Singularity?

PyTorch Example

TensorFlow Example

Where can I find MARCC’s Singularity software collections list?

We are using Singularity Hub, which allows for open source development of Singularity recipes using Github.  Keep in mind that images can be used across HPC centers or any Linux installation of Singularity.

MARCC HPC Singularity Hub Collections

What about Docker containers?

Yes you can import docker containers easily.

The software I want is not on MARCC’s collection, can I build my own on MARCC?

Yes, but you need a computer with root privileges (MARCC cannot provide this from a security viewpoint).

From Singularity’s Quick Start:

This guide is intended for running Singularity on a computer where you have root (administrative) privileges

Please refer to the above guide to get started.

MARCC has some Singularity software that I currently use but it doesn’t have all the packages I need.

There are some instances in which we can just add that software to the base image if it’s common open source software that many users will need.  In any case, email us at marcc-help@marcc.jhu.edu