What is Singularity?

From the Singularity Homepage:

Singularity enables users to have full control of their environment. Singularity containers can be used to package entire scientific workflows, software and libraries, and even data. This means that you don’t have to ask your cluster admin to install anything for you – you can put it in a Singularity container and run. Did you already invest in Docker? The Singularity software can import your Docker images without having Docker installed or being a superuser. Need to share your code? Put it in a Singularity container and your collaborator won’t have to go through the pain of installing missing dependencies. Do you need to run a different operating system entirely? You can “swap out” the operating system on your host for a different one within a Singularity container. As the user, you are in control of the extent to which your container interacts with its host. There can be seamless integration, or little to no communication at all. What does your workflow look like?

Why use Singularity on MARCC?

There are several reasons to consider using containers on MARCC:
Consider that most of MARCC hardware is using CentOS 6

  1. Software development performed on Ubuntu or RHEL7 (or OS that is not same as the MARCC Blue Crab cluster)
  2. Working with proprietary licensed Linux software that has only support in other Linux distributions
  3. Need the flexibility to work with software immediately on other HPC hardware that runs with Singularity
  4. Software metadata files are unmanageable (e.g., installing software with Python, R, conda) – a Singularity container allows for the use of a single compressed image file.
  5. You want to run software quickly because the software versions are rapidly evolving

We are actively supporting the use of Singularity to support a variety of software.  Sometimes containers are packaged inside of Lmod modules so that users don’t have to worry about the use of Singularity commands.

What are some examples of running Singularity?

PyTorch Example

TensorFlow Example

Where can I find MARCC’s Singularity software collections list?

We are using Singularity Hub, which allows for open source development of Singularity recipes using Github.  Keep in mind that images are available for easy use in multiple HPC centers that have Singularity installed.

MARCC HPC Singularity Hub Collections

What about Docker containers?

Yes, you can import docker containers easily.

The software I want is not on MARCC’s collection, can I build my own on MARCC?

Yes, you can build personal Singularity images, but you need a computer with root privileges (MARCC cannot provide this from a security viewpoint) – this means that you need to work on a machine with root privileges on Linux or a Linux VM.

From Singularity’s Quick Start:

This guide is intended for running Singularity on a computer where you have root (administrative) privileges

Please refer to the above guide to get started.

MARCC has some Singularity software that I currently use but it doesn’t have all the packages I need.

There are some instances in which we can add that software to the base image if it’s open source software that many users will need.  In any case, email us at marcc-help@marcc.jhu.edu

Troubleshooting

There may be limits to supporting Ubuntu >= 17 on the existing Linux kernel 2.6.32-573.22.1.el6.x86_64

Not working

singularity exec docker://ubuntu:latest echo "hello"

Works

singularity exec docker://ubuntu:16.04 echo "hello"
singularity exec docker://centos:7 echo "hello"