Apptainer/Singularity - Container Technology for HPC and Scientific Computing

21. Apptainer/Singularity - Container Technology for HPC and Scientific Computing#

21.1. What is Apptainer/Singularity?#

Apptainer (formerly Singularity) is a container platform designed specifically for High-Performance Computing (HPC), scientific computing, and environments where security and performance are critical. Unlike Docker, it’s designed to run on shared systems without requiring root privileges.

21.1.1. Key Characteristics:#

No daemon required - Containers run as regular processes
Single file containers - Images are stored as single files (.sif format)
User-centric security - Containers run as the calling user
HPC-optimized - Designed for batch schedulers and shared systems
Reproducible science - Ensures consistent environments across systems

21.1.2. Primary Use Cases:#

High-Performance Computing clusters
Scientific research and reproducibility
Shared computing environments
Environments requiring strict security
GPU computing workloads

21.2. History and Naming#

21.2.1. Timeline:#

2015: Singularity created by Gregory Kurtzer at Lawrence Berkeley National Laboratory
2021: Project forked due to licensing disputes
Singularity CE: Community Edition (open source)
SingularityPRO: Commercial version by Sylabs
2021: Linux Foundation creates Apptainer project
2022: Apptainer becomes the primary open-source version

21.2.2. Current Status:#

Apptainer: Linux Foundation project, primary open-source version
Singularity CE: Community-maintained version
SingularityPRO: Commercial enterprise version

Note: This tutorial covers Apptainer, but most commands work identically with Singularity CE.

21.3. Key Differences from Docker#

Feature	Docker	Apptainer/Singularity
Target Environment	General purpose, cloud	HPC, scientific computing
Daemon	Required	None
Root Access	Required for daemon	Not required
Image Format	Layered (OCI)	Single file (.sif)
User Inside Container	Can be different	Same as host user
File System	Isolated by default	Integrates with host
Network	Isolated by default	Uses host network
Performance	Good	Optimized for HPC
Scheduler Integration	Limited	Excellent (Slurm, PBS, etc.)
GPU Support	Good	Excellent
MPI Support	Complex	Native

21.4. Installation#

21.4.1. On Ubuntu/Debian:#

# Install dependencies
sudo apt update
sudo apt install -y software-properties-common

# Add Apptainer repository
sudo add-apt-repository -y ppa:apptainer/ppa
sudo apt update

# Install Apptainer
sudo apt install -y apptainer

# Verify installation
apptainer --version

21.4.2. On CentOS/RHEL/Rocky Linux:#

# Install EPEL repository
sudo dnf install -y epel-release

# Install Apptainer
sudo dnf install -y apptainer

# Or using RPM directly
sudo dnf install -y https://github.com/apptainer/apptainer/releases/download/v1.2.5/apptainer-1.2.5-1.x86_64.rpm

21.4.3. From Source (Advanced):#

# Install dependencies
sudo apt install -y build-essential libssl-dev uuid-dev libgpgme-dev \
    squashfs-tools libseccomp-dev wget pkg-config git cryptsetup-bin

# Install Go (if not already installed)
wget https://go.dev/dl/go1.21.5.linux-amd64.tar.gz
sudo tar -C /usr/local -xzf go1.21.5.linux-amd64.tar.gz
export PATH=$PATH:/usr/local/go/bin

# Clone and build Apptainer
git clone https://github.com/apptainer/apptainer.git
cd apptainer
./mconfig
make -C builddir
sudo make -C builddir install

21.4.4. Verification:#

# Check version
apptainer version

# Test basic functionality
apptainer run library://alpine

21.5. Basic Concepts#

21.5.1. Container Images (.sif files):#

SIF: Singularity Image Format - single, immutable file
Portable: Can be copied, shared, and executed anywhere
Signed: Can be cryptographically signed for verification
Read-only: Images are immutable during execution

21.5.2. Execution Modes:#

apptainer run: Execute default runscript
apptainer exec: Execute specific command
apptainer shell: Start interactive shell
apptainer instance: Start persistent service

21.5.3. Image Sources:#

Library: Apptainer/Singularity library (library://)
Docker Hub: Docker images (docker://)
OCI Registry: Any OCI-compliant registry (oci://)
Shub: Singularity Hub (shub://) - deprecated
Local Files: .sif files or sandbox directories

21.6. Working with Images#

21.6.1. Pulling Images from Registries:#

# From Apptainer Library
apptainer pull ubuntu.sif library://ubuntu:22.04

# From Docker Hub
apptainer pull ubuntu-docker.sif docker://ubuntu:22.04

# From a specific registry
apptainer pull myapp.sif oci://quay.io/myorg/myapp:latest

# List downloaded images
ls -la *.sif

21.6.2. Inspecting Images:#

# Show image metadata
apptainer inspect ubuntu.sif

# Show definition file (recipe)
apptainer inspect --deffile ubuntu.sif

# Show runscript
apptainer inspect --runscript ubuntu.sif

# Show labels
apptainer inspect --labels ubuntu.sif

21.6.3. Image Information:#

# Get detailed image information
apptainer sif list ubuntu.sif

# Verify image signature (if signed)
apptainer verify ubuntu.sif

21.7. Running Containers#

21.7.1. Basic Execution:#

# Run default command (runscript)
apptainer run ubuntu.sif

# Execute specific command
apptainer exec ubuntu.sif cat /etc/os-release

# Start interactive shell
apptainer shell ubuntu.sif

# Run with different shell
apptainer shell --shell /bin/bash ubuntu.sif

21.7.2. Environment and Variables:#

# Pass environment variables
apptainer exec --env MY_VAR=value ubuntu.sif env | grep MY_VAR

# Clean environment (only essential variables)
apptainer exec --cleanenv ubuntu.sif env

# Use custom environment file
echo "export MY_VAR=hello" > myenv.sh
apptainer exec --env-file myenv.sh ubuntu.sif echo $MY_VAR

21.7.3. Binding Directories:#

# Bind current directory to /mnt inside container
apptainer exec --bind $(pwd):/mnt ubuntu.sif ls /mnt

# Multiple bind mounts
apptainer exec --bind /data:/data,/scratch:/scratch ubuntu.sif ls /

# Bind with different destination
apptainer exec --bind /host/path:/container/path ubuntu.sif ls /container/path

# Read-only bind
apptainer exec --bind /data:/data:ro ubuntu.sif touch /data/test  # This will fail

21.7.4. Working Directories:#

# Change working directory inside container
apptainer exec --pwd /tmp ubuntu.sif pwd

# Use specific working directory
apptainer exec --workdir /scratch ubuntu.sif pwd

21.8. Building Custom Images#

21.8.1. Definition Files (Recipes):#

Apptainer uses definition files (similar to Dockerfiles) to build images:

# Create a definition file
cat > ubuntu-dev.def << 'EOF'
Bootstrap: docker
From: ubuntu:22.04

%post
    # Update and install packages
    apt-get update && apt-get install -y \
        build-essential \
        cmake \
        git \
        vim \
        libeigen3-dev \
        pkg-config
    
    # Clean up
    apt-get clean
    rm -rf /var/lib/apt/lists/*

%environment
    export LC_ALL=C
    export PATH=/usr/local/bin:$PATH

%runscript
    echo "Ubuntu development container"
    echo "Available tools: gcc, cmake, git, vim"
    /bin/bash

%labels
    Author YourName
    Version v1.0
    Description Ubuntu development environment

%help
    This container provides a Ubuntu 22.04 development environment
    with build tools and Eigen library installed.
    
    Usage:
        apptainer run ubuntu-dev.sif          # Start interactive session
        apptainer exec ubuntu-dev.sif gcc --version  # Run specific command
EOF

21.8.2. Building Images:#

# Build from definition file (requires root/sudo)
sudo apptainer build ubuntu-dev.sif ubuntu-dev.def

# Build in sandbox mode (writable directory)
sudo apptainer build --sandbox ubuntu-dev-sandbox/ ubuntu-dev.def

# Build from Docker image directly
sudo apptainer build myimage.sif docker://ubuntu:22.04

# Remote build (if you don't have root access)
apptainer build --remote ubuntu-dev.sif ubuntu-dev.def

21.8.3. Sandbox Mode:#

# Create writable sandbox
sudo apptainer build --sandbox myapp-sandbox/ docker://ubuntu:22.04

# Shell into sandbox with write access
sudo apptainer shell --writable myapp-sandbox/

# Make changes inside sandbox, then convert to .sif
sudo apptainer build myapp.sif myapp-sandbox/

21.9. File System Integration#

21.9.1. Default Mounts:#

Apptainer automatically mounts several directories:

# Check what's mounted by default
apptainer exec ubuntu.sif mount | grep bind

# Common default mounts:
# - $HOME (your home directory)
# - /tmp
# - /proc
# - /sys
# - /dev

21.9.2. Custom Bind Mounts:#

# Single bind mount
apptainer exec --bind /data ubuntu.sif ls /data

# Multiple bind mounts
apptainer exec --bind /data,/project,/scratch ubuntu.sif df -h

# Bind with custom mount point
apptainer exec --bind /external/data:/internal/data ubuntu.sif ls /internal/data

# Bind multiple paths to same destination
apptainer exec --bind /path1:/shared,/path2:/shared ubuntu.sif ls /shared

21.9.3. Configuration Files:#

# System configuration
cat /etc/apptainer/apptainer.conf

# User configuration
mkdir -p ~/.apptainer
cat > ~/.apptainer/apptainer.conf << 'EOF'
bind path = /data
bind path = /scratch
mount home = yes
EOF

21.10. HPC-Specific Features#

21.10.1. Batch Scheduler Integration:#

21.10.1.1. SLURM Example:#

# Create SLURM job script
cat > slurm_job.sh << 'EOF'
#!/bin/bash
#SBATCH --job-name=apptainer_job
#SBATCH --ntasks=4
#SBATCH --time=01:00:00
#SBATCH --partition=compute

# Load Apptainer module (if needed)
module load apptainer

# Run application in container
apptainer exec --bind /scratch:/scratch myapp.sif ./my_parallel_app
EOF

# Submit job
sbatch slurm_job.sh

21.10.1.2. PBS/Torque Example:#

# Create PBS job script
cat > pbs_job.sh << 'EOF'
#!/bin/bash
#PBS -N apptainer_job
#PBS -l nodes=1:ppn=4
#PBS -l walltime=01:00:00
#PBS -q batch

cd $PBS_O_WORKDIR

# Run container
apptainer exec --bind /work:/work myapp.sif ./my_application
EOF

# Submit job
qsub pbs_job.sh

21.10.2. MPI Support:#

# MPI application in container
mpirun -np 4 apptainer exec mpi-app.sif /usr/bin/my_mpi_app

# With host MPI
mpirun -np 4 apptainer exec --bind /opt/mpi:/opt/mpi mpi-app.sif ./app

21.10.3. GPU Support:#

# Enable NVIDIA GPU support
apptainer exec --nv gpu-app.sif nvidia-smi

# Enable ROCm (AMD GPU) support
apptainer exec --rocm gpu-app.sif rocm-smi

# Check GPU availability inside container
apptainer exec --nv ubuntu.sif nvidia-smi

21.11. Practical Examples#

21.11.1. Example 1: Scientific Python Environment#

Create a definition file for scientific computing:

cat > scientific-python.def << 'EOF'
Bootstrap: docker
From: python:3.9-slim

%post
    # Install system dependencies
    apt-get update && apt-get install -y \
        build-essential \
        gfortran \
        libopenblas-dev \
        liblapack-dev \
        pkg-config
    
    # Install Python packages
    pip install --no-cache-dir \
        numpy \
        scipy \
        matplotlib \
        pandas \
        scikit-learn \
        jupyter \
        seaborn
    
    # Clean up
    apt-get clean
    rm -rf /var/lib/apt/lists/*

%environment
    export PYTHONPATH=/usr/local/lib/python3.9/site-packages:$PYTHONPATH

%runscript
    echo "Scientific Python Environment"
    echo "Python version: $(python --version)"
    echo "Available packages: numpy, scipy, matplotlib, pandas, scikit-learn"
    python3 "$@"

%labels
    Author Scientific Computing Team
    Description Python environment for scientific computing
    Version 1.0
EOF

# Build the image
sudo apptainer build scientific-python.sif scientific-python.def

21.11.2. Example 2: C++ Development with Eigen#

cat > cpp-eigen.def << 'EOF'
Bootstrap: docker
From: ubuntu:22.04

%post
    apt-get update && apt-get install -y \
        build-essential \
        cmake \
        libeigen3-dev \
        pkg-config \
        git \
        gdb \
        valgrind
    
    apt-get clean
    rm -rf /var/lib/apt/lists/*

%environment
    export CXX=g++
    export CC=gcc

%runscript
    echo "C++ Development Environment with Eigen"
    echo "Compiler: $(g++ --version | head -n1)"
    echo "Eigen version: $(pkg-config --modversion eigen3)"
    /bin/bash

%labels
    Author Development Team
    Description C++ development with Eigen library
EOF

sudo apptainer build cpp-eigen.sif cpp-eigen.def

21.11.2.1. Use the C++ environment:#

# Create a test program
cat > test_eigen.cpp << 'EOF'
#include <iostream>
#include <Eigen/Dense>

int main() {
    Eigen::Matrix3d m;
    m << 1, 2, 3,
         4, 5, 6,
         7, 8, 9;
    
    std::cout << "Matrix m:" << std::endl << m << std::endl;
    std::cout << "Determinant: " << m.determinant() << std::endl;
    
    return 0;
}
EOF

# Compile and run inside container
apptainer exec --bind $(pwd):/work --pwd /work cpp-eigen.sif \
    g++ -I/usr/include/eigen3 test_eigen.cpp -o test_eigen

apptainer exec --bind $(pwd):/work --pwd /work cpp-eigen.sif ./test_eigen

21.11.3. Example 3: Machine Learning Environment#

cat > ml-pytorch.def << 'EOF'
Bootstrap: docker
From: pytorch/pytorch:latest

%post
    # Install additional packages
    conda install -y \
        scikit-learn \
        matplotlib \
        seaborn \
        jupyter \
        tensorboard
    
    pip install --no-cache-dir \
        transformers \
        datasets \
        wandb
    
    # Clean conda cache
    conda clean -ya

%environment
    export PYTHONPATH=/opt/conda/lib/python3.8/site-packages:$PYTHONPATH

%runscript
    echo "PyTorch Machine Learning Environment"
    echo "Python: $(python --version)"
    echo "PyTorch: $(python -c 'import torch; print(torch.__version__)')"
    python "$@"
EOF

sudo apptainer build ml-pytorch.sif ml-pytorch.def

21.12. Scientific Computing Use Cases#

21.12.1. Bioinformatics Pipeline:#

cat > bioinformatics.def << 'EOF'
Bootstrap: docker
From: ubuntu:22.04

%post
    apt-get update && apt-get install -y \
        wget \
        bzip2 \
        ca-certificates \
        curl \
        git
    
    # Install Miniconda
    wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
    bash Miniconda3-latest-Linux-x86_64.sh -b -p /opt/conda
    rm Miniconda3-latest-Linux-x86_64.sh
    
    # Install bioinformatics tools
    /opt/conda/bin/conda install -c bioconda \
        bwa \
        samtools \
        bcftools \
        bedtools \
        fastqc
    
    /opt/conda/bin/conda clean -ya

%environment
    export PATH=/opt/conda/bin:$PATH

%runscript
    echo "Bioinformatics Tools Container"
    echo "Available tools: bwa, samtools, bcftools, bedtools, fastqc"
    /bin/bash
EOF

21.12.2. Computational Chemistry:#

cat > chemistry.def << 'EOF'
Bootstrap: docker
From: ubuntu:22.04

%post
    apt-get update && apt-get install -y \
        build-essential \
        gfortran \
        python3 \
        python3-pip \
        libopenmpi-dev \
        libscalapack-openmpi-dev
    
    # Install chemistry software
    pip3 install \
        ase \
        pymatgen \
        rdkit-pypi \
        cclib
    
    apt-get clean

%environment
    export PYTHONPATH=/usr/local/lib/python3.10/site-packages:$PYTHONPATH

%runscript
    echo "Computational Chemistry Environment"
    python3 "$@"
EOF

21.13. Performance Considerations#

21.13.1. Optimizing Container Performance:#

21.13.1.1. 1. Image Size Optimization:#

# Multi-stage build for smaller images
cat > optimized.def << 'EOF'
Bootstrap: docker
From: ubuntu:22.04 as builder

%post
    # Build stage
    apt-get update && apt-get install -y \
        build-essential \
        cmake \
        git
    
    # Build application
    git clone https://github.com/example/app.git /tmp/app
    cd /tmp/app && mkdir build && cd build
    cmake .. && make

Bootstrap: docker
From: ubuntu:22.04

%files from builder
    /tmp/app/build/myapp /usr/local/bin/myapp

%post
    # Only install runtime dependencies
    apt-get update && apt-get install -y \
        libstdc++6
    apt-get clean
    rm -rf /var/lib/apt/lists/*

%runscript
    /usr/local/bin/myapp "$@"
EOF

21.13.1.2. 2. Parallel Processing:#

# Use all available cores during build
%post
    make -j$(nproc)

21.13.1.3. 3. Memory Optimization:#

# Limit memory usage during build
apptainer build --tmpdir /scratch myapp.sif myapp.def

21.13.2. Storage Considerations:#

# Check image size
ls -lh *.sif

# Compress images (if filesystem supports)
apptainer sif setprim myapp.sif 1  # Set primary partition

21.14. Security Model#

21.14.1. User Namespace:#

# Container runs as calling user
whoami                    # host user
apptainer exec ubuntu.sif whoami  # same user inside container

# Check user mapping
apptainer exec ubuntu.sif id

21.14.2. Security Features:#

# Run with security options
apptainer exec --security allow-setuid myapp.sif command

# Disable network access
apptainer exec --net myapp.sif command

# Run without privileges
apptainer exec --no-privs myapp.sif command

21.14.3. Image Signing and Verification:#

# Sign image (requires key)
apptainer sign myapp.sif

# Verify signed image
apptainer verify myapp.sif

# Check signature details
apptainer sif list myapp.sif

21.15. Best Practices#

21.15.1. 1. Image Management:#

# Use descriptive names
apptainer build myproject-v1.2.3.sif myproject.def

# Include version information in labels
%labels
    Version 1.2.3
    BuildDate $(date)
    GitCommit $(git rev-parse HEAD)

21.15.2. 2. Reproducibility:#

# Pin base image versions
Bootstrap: docker
From: ubuntu:22.04  # Specific version, not 'latest'

# Document dependencies
%help
    This container includes:
    - GCC 11.2.0
    - CMake 3.22.1
    - Eigen 3.4.0

21.15.3. 3. Environment Variables:#

%environment
    # Set locale
    export LC_ALL=C.UTF-8
    export LANG=C.UTF-8
    
    # Set paths
    export PATH=/usr/local/bin:$PATH
    export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH

21.15.4. 4. File Permissions:#

# Ensure proper permissions in %post
%post
    chmod +x /usr/local/bin/myapp
    chown root:root /usr/local/bin/myapp

21.15.5. 5. Testing:#

# Include tests in definition
%test
    # Test basic functionality
    /usr/local/bin/myapp --version
    
    # Test dependencies
    python -c "import numpy; print('NumPy version:', numpy.__version__)"

21.16. Troubleshooting#

21.16.1. Common Issues and Solutions:#

21.16.1.1. Issue 1: Permission Denied#

# Problem: Cannot write to mounted directory
# Solution: Check directory permissions on host
ls -la /path/to/mount
# Fix permissions if needed
chmod 755 /path/to/mount

21.16.1.2. Issue 2: Library Not Found#

# Problem: Shared library not found
# Solution: Check LD_LIBRARY_PATH or use ldconfig
apptainer exec myapp.sif ldd /usr/local/bin/myapp
apptainer exec myapp.sif ldconfig -p | grep libname

21.16.1.3. Issue 3: Build Fails#

# Problem: Build fails due to network issues
# Solution: Use --fix-perms or build with --tmpdir
sudo apptainer build --fix-perms --tmpdir /tmp myapp.sif myapp.def

21.16.1.4. Issue 4: GPU Not Accessible#

# Check if NVIDIA drivers are available
nvidia-smi

# Test GPU access in container
apptainer exec --nv ubuntu.sif nvidia-smi

# Check CUDA installation
apptainer exec --nv cuda-app.sif nvcc --version

21.16.2. Debugging:#

# Enable verbose output
apptainer -v exec myapp.sif command

# Enable debug mode
apptainer -d exec myapp.sif command

# Check container environment
apptainer exec myapp.sif env | sort

# Inspect running processes
apptainer exec myapp.sif ps aux

21.17. Exercises#

21.17.1. Exercise 1: Basic Image Operations#

Pull an Ubuntu image from the library
Run a simple command inside the container
Start an interactive shell session
Inspect the image metadata

21.17.2. Exercise 2: Custom Environment#

Create a definition file for a Python data science environment
Include pandas, numpy, and matplotlib
Build the image
Test the installation by importing the libraries

21.17.3. Exercise 3: File System Integration#

Create a local directory with some data files
Run a container with the directory bound to /data
Process the files inside the container
Verify results are available on the host

21.17.4. Exercise 4: HPC Simulation#

Create a simple MPI “Hello World” program
Build a container with MPI support
Run the program with multiple processes
Test with a job scheduler (if available)

21.17.5. Exercise 5: GPU Computing#

Pull a CUDA-enabled image
Write a simple CUDA program or use existing GPU software
Run with GPU support enabled
Verify GPU utilization

21.17.6. Exercise Solutions:#

21.17.6.1. Exercise 1 Solution:#

# Pull Ubuntu image
apptainer pull ubuntu.sif library://ubuntu:22.04

# Run simple command
apptainer exec ubuntu.sif cat /etc/os-release

# Interactive shell
apptainer shell ubuntu.sif

# Inspect metadata
apptainer inspect ubuntu.sif

21.17.6.2. Exercise 2 Solution:#

cat > datascience.def << 'EOF'
Bootstrap: docker
From: python:3.9-slim

%post
    pip install pandas numpy matplotlib jupyter

%runscript
    python3 "$@"

%test
    python3 -c "import pandas, numpy, matplotlib; print('All packages imported successfully')"
EOF

sudo apptainer build datascience.sif datascience.def
apptainer exec datascience.sif python3 -c "import pandas; print(pandas.__version__)"

21.17.6.3. Exercise 3 Solution:#

# Create test data
mkdir data
echo "sample,value" > data/test.csv
echo "A,1" >> data/test.csv
echo "B,2" >> data/test.csv

# Process in container
apptainer exec --bind data:/data datascience.sif python3 -c "
import pandas as pd
df = pd.read_csv('/data/test.csv')
df['doubled'] = df['value'] * 2
df.to_csv('/data/result.csv', index=False)
print('Processing complete')
"

# Check results
cat data/result.csv

21.18. Conclusion#

Apptainer/Singularity provides a powerful containerization solution specifically designed for HPC and scientific computing environments. Key advantages include:

21.18.1. Strengths:#

Security: Runs without root privileges, user-centric model
Performance: Optimized for HPC workloads
Integration: Excellent support for schedulers, MPI, and GPUs
Reproducibility: Single-file images ensure consistent environments
Portability: Works across different HPC systems

21.18.2. When to Choose Apptainer/Singularity:#

High-Performance Computing environments
Scientific computing workflows
Shared computing resources
GPU-accelerated applications
Environments requiring strict security
Reproducible research

21.18.3. Comparison Summary:#

Docker: Best for general development, microservices, CI/CD
Podman: Good Docker alternative with better security
Apptainer/Singularity: Ideal for HPC, scientific computing, shared systems

The choice depends on your specific requirements, but for scientific computing and HPC environments, Apptainer/Singularity is often the preferred solution.