Skip to content

Containers

Containers are frameworks to isolate and collect the dependencies of software codes. Containers are typically used to enhance portability – especilly of difficult to compile codes, overcome compatibility limitations, and preserve codes in a specific (runnable) state. Put another way, a container is a way to run one operating system embedded in a different “host” operating system or to easily move a complex, compiled code from one compute platform to another. Properly deployed, the container layer can have very little impact on the user experience, but it is good practice to understand how they work – in order to best capitalize on their benefits.

Apptainer – formerly known as Singularity, is the pricipal containerization system on Sherlock. PodMan will also work, but will likely require installation by the user. Docker – arguably the best known and most popular containerization system, is not well suited to shared HPC environments. For more on that, and other related topics, see the Sherlock documentation:

Additionally, Apptainer’s documentation is excellent:

Given the quality of available external documentation, here we will touch on a few simple examples and interesting tricks and test cases. Note also that this tutorial comes with no guarantees, and minor ambituities or discrepancies are left for the reader to resolve, as an exercise.

A Simple Container

Simple containers can be created, for testing, training, or developmeent purposes by downloading templates from dockerhub, singularity hub, or some other location. For example, to create a simple Ubuntu base container,

$ apptainer build ubuntu.sif docker://ubuntu:latest
INFO:    Starting build...
INFO:    Fetching OCI image...
28.3MiB / 28.3MiB [=================================================================================================================================================================] 100 % 0.0 b/s 0s
INFO:    Extracting OCI image...
INFO:    Inserting Apptainer configuration...
INFO:    Creating SIF file...
[===========================================================================================================================================================================================] 100 % 0s
INFO:    Build complete: ubuntu.sif

Breaking down that command a bit:

  • apptainer: The base commmand to invokle the apptainer container system
  • build: Subcommand to build a container
  • ubuntu.sif: the name of the container to build. This can be anything *.sif is the conventional suffix for “Singularity Image File.”
  • docker://ubuntu:latest: The source of the base image

The astute reader will notice that – while this trick is encouraging, the resulting container lacks substance, having not even the simplest applications installed:

$ apptainer shell ubuntu.sif 
Apptainer> vim monkey.txt
bash: vim: command not found
Apptainer> vi 
bash: vi: command not found
Apptainer> nano
bash: nano: command not found

Simple operations can be tested by running a container from a remote image, without saving a *.sif image file. For example,

apptainer shell -f --writable-tmpfs --fakeroot docker://debian

Will launch a shell app on a container built from the docker://ubuntu:latest image. Note the --writable-tmpfs --fakeroot options will allow us a bit of runtime flexibility with the container. We can then install a few small applications into the temporarly filesystem space

Apptainer> apt update
Apptainer> apt -y install vim nano 
Apptainer> which vim
/usr/bin/vim
Apptainer> which nano
/usr/bin/nano

The minimum requirements for X11 forwarding can be evaluated via,

apptainer shell -f --writable-tmpfs --fakeroot docker://debian
Apptainer> apt update
Apptainer> apt -y install x11-apps
Apptainer> xeyes

image

Definition files

More sophisticated (and useful) containers can be pulled from Docker Hub hub.docker.com and various other sources. Often, a container is built from a definition file. Definition files include several sections, including for example, %files, %environment, and %post where – as their names suggest, files are transferred from the host to the container, environment variables are set, or shell comannds are run after (“post”) the base container is instantiated. For example the popular lolcow toy container,

Bootstrap: docker
From: ubuntu:latest
Stage: build

%setup
    # NOTE: This will fail, because in %setup, it refers to the host root filesystem.
    # touch /file1
    #
    # maybe we meant this:
    #touch $(pwd)/file1
    echo "Data from %setup into host machine, ./file1">$(pwd)/file1
    #
    # do this to create file2 on the Apptainer root filesystem.
    #touch ${APPTAINER_ROOTFS}/file2
    echo "Data from %setup into APPTAINDER_ROOTFS/file2">${APPTAINER_ROOTFS}/file2
%files
    file1
    file1 /opt

%environment
    export LISTEN_PORT=12345
    export LC_ALL=C
    #
    # set up cowsay path(s):
    export PATH=/usr/games:${PATH}
    export COWSAYS_DIR=/usr/share/cowsay/cows
    export COWS_PATH=/usr/share/cowsay/cows

%post
    apt-get update
    #apt-get install -y netcat
    apt-get -y install netcat-traditional
    apt-get -y install cowsay lolcat
    NOW=`date`
    echo "export NOW=\"${NOW}\"" >> $APPTAINER_ENVIRONMENT
    #
    # and if we want to do x11 things, we would add x11-apps here:
    apt-get -y install x11-apps

%runscript
    # be careful with this section! It can be very picky -- like ignoring statements that follow a blank line or an inter-statement coment.
    #   for complex run-scripts, consider writing a runscript.sh; then just execute that script here.
    echo "Container was created $NOW"
    echo "Arguments received: $*"
    echo "$@"
    echo "Cowthink: " $(which cowthink)
    echo $@ | cowthink -f $COWS_PATH/turkey.cow | lolcat

    #echo "Cowsay: " $(date | cowsay | lolcat)
    #echo "Cowthink: " $( $@ | cowthink -f /usr/share/cowsay/cows/turkey.cow | lolcat)

%startscript
    nc -lp $LISTEN_PORT

%test
    grep -q NAME=\"Ubuntu\" /etc/os-release
    if [ $? -eq 0 ]; then
        echo "Container base is Ubuntu as expected."
    else
        echo "Container base is not Ubuntu."
        exit 1
    fi

%labels
    Author alice
    Version v0.0.1

%help
    This is a demo container used to illustrate a def file that uses all
    supported sections.

Build the container,

apptainer build --force lolcow.sif lolcow.def

then, run a test:

$ apptainer run lolcow.sif Gobble! Gobble!
Container was created Thu Aug 28 18:52:39 UTC 2025
Arguments received: Gobble! Gobble!
Gobble! Gobble!
Cowthink:  /usr/games/cowthink
 _________________
( Gobble! Gobble! )
 -----------------
  o                                  ,+*^^*+___+++_
   o                           ,*^^^^              )
    o                       _+*                     ^**+_
     o                    +^       _ _++*+_+++_,         )
              _+^^*+_    (     ,+*^ ^          \+_        )
             {       )  (    ,(    ,_+--+--,      ^)      ^\
            { (@)    } f   ,(  ,+-^ __*_*_  ^^\_   ^\       )
           {:;-/    (_+*-+^^^^^+*+*<_ _++_)_    )    )      /
          ( /  (    (        ,___    ^*+_+* )   <    <      \
           U _/     )    *--<  ) ^\-----++__)   )    )       )
            (      )  _(^)^^))  )  )\^^^^^))^*+/    /       /
          (      /  (_))_^)) )  )  ))^^^^^))^^^)__/     +^^
         (     ,/    (^))^))  )  ) ))^^^^^^^))^^)       _)
          *+__+*       (_))^)  ) ) ))^^^^^^))^^^^^)____*^
          \             \_)^)_)) ))^^^^^^^^^^))^^^^)
           (_             ^\__^^^^^^^^^^^^))^^^^^^^)
             ^\___            ^\__^^^^^^))^^^^^^^^)\\
                  ^^^^^\uuu/^^\uuu/^^^^\^\^\^\^\^\^\^\
                     ___) >____) >___   ^\_\_\_\_\_\_\)
                    ^^^//\\_^^//\\_^       ^(\_\_\_\)
                      ^^^ ^^ ^^^ ^

Writable containers

writable tempfs

Generally speaking, containers should be built start to finish from definition scripts. There are times, however when:

  1. This is not possible
  2. It’s a long script, and it requrie some debugging
  3. You just want to get something done, even if it is sloppy.

In thesse cases, it might make sense to build a writable --sandbox condainer. As it happens, age (of the *.def file above) and evolution (newer versions of ubuntu) provide an excellent example of this. If you try to build this container as it is originally written, you are likely to encounter an error installing the netcat application. In order to determine the correct way to install netcat, we might get away with launching a shell with --writable-tmpfs, as we did with the x11 example above,

apptainer shell --fakeroot --writable-tmpfs lolcow.sif
...
Apptainer> apt-get install netcat
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Package netcat is a virtual package provided by:
  netcat-traditional 1.10-48
  netcat-openbsd 1.226-1ubuntu2
You should explicitly select one to install.

E: Package 'netcat' has no installation candidate

This suggests that there are two netcat candidates, so then try

Apptainer> apt-get install netcat-traditional

or

Apptainer> apt-get install netcat-openbsd

Sandbox

In some cases, it may be necessary to build and (hopefully…) temporarily maintain a writable container. This can be done using the --sandbox option, eg.

$ apptainer build --sandbox lolcow_sbx lolcow.def

This container can now be opened and modified,

$ apptainer shell --writable --fakeroot lolcow_sbx

Ideally, the necessary modifications are tracked and recorded in the *.def file. Alternatively, the sandbox container can be converted to a *.sif,

apptainer build lolcow_from_sbx.sif lolcow_sbx

Binding the local filesystem

The --bind option, which “binds” host paths to container paths, is one of the most powerful features of cuntainer building. The --bind option can be used to

  • Attach source code paths at build time
  • Attach large data sets at runtime
  • Allow large writable spaces at runtime
  • Override configuration files inside a container

In Linux parlance, --bind is a way to mount a host filesystem inside a container. --bind is executed at runtime, as an option bassed to the apptainer build, shell, exec, or run subcommands.

We have built some examples into our lolcow.sif conatiner. Note that th %setup phase creates a file called file1 on the host system:

$ cat file1
Data from %setup into host machine, ./file1

And we also copied this file to /opt in the %files section,

$ apptainer shell lolcow.sif 
Apptainer> cat /opt/file1
Data from %setup into host machine, ./file1

and similarly, we created /file2 on the root level of the Apptainer filesystem,

$ apptainer shell lolcow.sif 
Apptainer> cat /file2 
Data from %setup into APPTAINDER_ROOTFS/file2

We can overlay those files by using --bind, for example:

$ echo "Override data from host">override_data.dat
$ apptainer shell --bind $(pwd)/override_data.dat:/opt/file1  lolcow.sif
Apptainer> cat /opt/file1
Override data from host

Example applications include:

Source Code

apptainer build --bind $(pwd)/my_app/src:/SRC container.sif container.def

This will map the host path ./my_app/src to the path /SRC inside the container. In this fashion, the build script inside the container can be kept relatively simple.

Data sets and Writable space

For cases where complex codes or pipelines are containerized, --bind can be employed to map the runtime code to the container. For example,

apptainer exec --bind $(pwd)/my_input_data_path/my_data.hdf5:/INPUT_DATA/data.hdf5 --bind $(pwd)/my_output_data_path:/OUTPUT_DATA container.sif python3 do_science.py

Note that this example implies that a large data set file ./my_input_data_path/my_data.hdf5 can be processed by the container’s do_science.py Python module, then the output is written to the host filesystem path, ./my_output_data_path/. This same approach can be employed – with a bit of trickery, to run interactive software installations, pipelines, or similar workflows.

Override configuration files

In The previous examples, --bind mapped a directory from the host to the container, as a way to efficiently port data into our out of the container. --bind can also be applied to override an existing directory or file. For example, if license configuration has changed, rather than rebuilding a container to update that license information, we can use --bind to replace the container’s internal license file with a local file – something like:

apptainer run --bind $(pwd)/new_license_config.lic:/SW/license/license.lic container.sif

Note that this approach can be useful for cases where licensing information changes for a difficult to build compiler or to move a container to a new compute platform – eg, porting heavy compute loads to Natioal Labs HPC.