Skip to content

CMG on Sherlock

Computer Modeling Group Ltd (CMG) (cmgl.ca, also see Wikipedia produces a suite of proprietary physics based simulators popular in a variety of Earth and other physical sciences. Simulators include:

  • GEM
  • IMEX
  • STARS

In this document, we compile some basic “how to” cases, descriptions of run-time parameters, discussions of environment variables, and general run-time syntax.

Versions up to 2025.x

Multiple versions of CMG are installed as part of the SERC software stack on Sherlock. Versions <2025.x are installed directly on Sherlock; to enable the SW stack, execute the following setup scritp – either at runtime or as part of your .bashrc:

module use /home/groups/sh_s-dss/share/sdss/modules/modulefiles

then load the CMG module with some LMOD consistent variant of:

module load CMG/

Versions 2026.x

CMG versions >2026.x are available on Sherlock as Apptainer containers. The precise workflow for these containers is a work in progress, and the utility of modules when using containers is limited, so for the time being the container files are stored alongside the lower version software directories, eg.

/home/groups/sh_s-dss/share/sdss/software/no_arch/CMG/cmg_202611.sif
/home/groups/sh_s-dss/share/sdss/software/no_arch/CMG/cmg_202620.sif

CMG programs can be run interactively using the apptainer shell command, eg.

apptainer shell /home/groups/sh_s-dss/share/sdss/software/no_arch/CMG/cmg_202611.sif

or exec,

apptainer exec /home/groups/sh_s-dss/share/sdss/software/no_arch/CMG/cmg_202611.sif stars

Modules for versions 2026.x

Modules for the CMG@2026 containers provide environment variables and alias definitions that can simplify the syntax. For example,

$ module load CMG/2026.20
[myoder96@sh04-09n30 /scratch/users/myoder96/Downloads/CMG/apptainer] (job 45886418) $ echo $CMG_SIF 
/home/groups/sh_s-dss/share/sdss/software/no_arch/CMG/cmg_202620.sif
[myoder96@sh04-09n30 /scratch/users/myoder96/Downloads/CMG/apptainer] (job 45886418) $ # use an alias to launch a shell
[myoder96@sh04-09n30 /scratch/users/myoder96/Downloads/CMG/apptainer] (job 45886418) $ CMG_shell 
Apptainer> which stars
/opt/CMG/bin/stars
Apptainer> ls -lh /opt/CMG/bin/stars
lrwxrwxrwx 1 myoder96 bprogers 49 Sep 29 09:53 /opt/CMG/bin/stars -> /opt/CMG/stars/2026.20/linux_x64/exe/st202620.exe
Apptainer> exit
exit

Note that shortcut links to the executables, eg. stars, stars_mpi, mx have been added to the container’s /opt/CMG/bin – which will simplify general syntax and CMG upgrades. Similarly, an exec alias is provided,

$ CMG_exec stars
 ********************************************************************************
 *                                                                              *
 *                                STARS  2026.20                                *
 *               Advanced Process and Thermal Reservoir Simulator               *
 *                      Quarterly Release 2 for Linux x64                       *
 *                            2026-Jun-15   08:17:18                            *
 *                                                                              *
 *                          (c) Copyright 1977 - 2026                           *
 *                Computer Modelling Group Ltd., Calgary, Canada                *
 *                             All Rights Reserved                              *
 *                                                                              *
 ********************************************************************************

Example jobs

Examples and test jobs are included with each release of CMG; they can be found in the CMG directory structure like, ${CMG_ROOT}/${CMG_program}/${CMG_VER}/tpl/{job_type}/{sample jobs}. For example, for version @2023.10, installed natively on Sherlock,k

$ ls -lh /home/groups/sh_s-dss/share/sdss/software/x86_64_arch/CMG/2023.101/gem/2023.10/tpl/geo/
total 528K
-rwxrwxr-x 1 myoder96 jfreshwa  359 Mar  6  2000 gmgeo000.doc
-rwxrwxr-x 1 myoder96 jfreshwa  44K Feb 17  2011 gmgeo001.dat
-rwxrwxr-x 1 myoder96 jfreshwa  43K Feb 17  2011 gmgeo002.dat
-rwxrwxr-x 1 myoder96 jfreshwa 474K Feb 28  2023 gmgeo003.dat
-rwxrwxr-x 1 myoder96 jfreshwa  43K Feb 17  2011 gmgeo004.dat

For CMG@2026.x, example jobs are stored in the container in directories like, /opt/CMG/{CMG_program}/2026.20/tpl/{job_type}/{sample jobs}, eg.

/opt/CMG/gem/2026.20/tpl/geo/gmgeo002.dat

Note that future versions of containerized CMG might move to the host filesystem, to reduce the size of the container.

To run sample jobs – containerized or otherwise, the sample input data can be copied from the container (or shared, read-only filesystem), to the local file space or accessed from inside the container. For example, having copied the sample or prepared a local job in advance,

$ module load CMG/2026.20
$ CMG_shell
Apptainer> cp -r /opt/CMG/gem/2026.20/tpl/geo ./
Apptainer> ls -lh geo
Apptainer> exit

Now, from the host system:

$ ls -lh geo
total 612K
-rw-r--r-- 1 myoder96 bprogers  359 Sep 29 13:32 gmgeo000.doc
-rw-r--r-- 1 myoder96 bprogers  44K Sep 29 13:32 gmgeo001.dat
-rw-r--r-- 1 myoder96 bprogers  43K Sep 29 13:32 gmgeo002.dat
-rw-r--r-- 1 myoder96 bprogers 474K Sep 29 13:32 gmgeo003.dat
-rw-r--r-- 1 myoder96 bprogers  43K Sep 29 13:32 gmgeo004.dat

And we can run the test job against the local copy of the input data, by either directly referencing the container or using an alias shorthand (from the LMOD module):

$ apptainer exec /home/groups/sh_s-dss/share/sdss/software/no_arch/CMG/cmg_202620.sif gm -f geo/gmgeo001.dat -o gm001_local.out
$ CMG_exec gm -f geo/gmgeo001.dat -o gm01.out

or we can run the example from the container file space:

CMG_exec gm -f /opt/CMG/gem/2026.20/tpl/geo/gmgeo001.dat -o gm01.out

Executables and directory structure

The directory structure for CMG models is somewhat non-standard in HPC environments, but logical enough. For linux, the executables (binaries) are located in folders like,

${CMG_PATH}/{simulator}/{version}/linux_x64/exe/

So for the 2023 version of GEM,

${CMG_PATH}/gem/2023.10/linux_x64/exe/

These paths are configured in the module definition file:

[myoder96@sh03-09n72 ] (job 31521159) $ module show CMG
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
   /home/groups/sh_s-dss/share/sdss/modules/modulefiles/CMG/2023.101.lua:
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
whatis("CMG - gem, imex, stars. br, launcher, rlmsecure should also be supported")
pushenv("CMG_HOME","/home/groups/sh_s-dss/share/sdss/software/no_arch/CMG/2023.101")
pushenv("CMG_LIC_HOST","27053@srcc-license-srcf.stanford.edu:27053")
pushenv("CMG_FAILOVER_HOSTS","27053@srcc-license-srcf.stanford.edu:27053")
pushenv("CMG_VER","2023.10")
prepend_path("PATH","/home/groups/sh_s-dss/share/sdss/software/no_arch/CMG/2023.101")
prepend_path("PATH","/home/groups/sh_s-dss/share/sdss/software/no_arch/CMG/2023.101/dicts")
prepend_path("PATH","/home/groups/sh_s-dss/share/sdss/software/no_arch/CMG/2023.101/gem/2023.10/linux_x64/exe")
prepend_path("PATH","/home/groups/sh_s-dss/share/sdss/software/no_arch/CMG/2023.101/imex/2023.10/linux_x64/exe")
prepend_path("PATH","/home/groups/sh_s-dss/share/sdss/software/no_arch/CMG/2023.101/stars/2023.10/linux_x64/exe")
prepend_path("PATH","/home/groups/sh_s-dss/share/sdss/software/no_arch/CMG/2023.101/br/2023.10/linux_x64/exe")
prepend_path("PATH","/home/groups/sh_s-dss/share/sdss/software/no_arch/CMG/2023.101/launcher/2023.10/linux_x64/exe")
prepend_path("PATH","/home/groups/sh_s-dss/share/sdss/software/no_arch/CMG/2023.101/rlmsecure")
prepend_path("PATH","/home/groups/sh_s-dss/share/sdss/software/no_arch/CMG/2023.101/rlmsecure/linux_x64")
prepend_path("PATH","/home/groups/sh_s-dss/share/sdss/software/no_arch/CMG/2023.101/CEIPServer/linux_x64/exe")
prepend_path("PATH","/home/groups/sh_s-dss/share/sdss/software/no_arch/CMG/2023.101/sdss_bin")
prepend_path("LD_LIBRARY_PATH","/home/groups/sh_s-dss/share/sdss/software/no_arch/CMG/2023.101/imex/2023.10/linux_x64/lib")
prepend_path("LD_LIBRARY_PATH","/home/groups/sh_s-dss/share/sdss/software/no_arch/CMG/2023.101/stars/2023.10/linux_x64/lib")
prepend_path("LD_LIBRARY_PATH","/home/groups/sh_s-dss/share/sdss/software/no_arch/CMG/2023.101/CEIPServer/linux_x64/exe/lib")

Execution signature and runtime parameters:

The necessary paths to executable and linking libraries are defined by the module definition. For versions <@2026.x – where CMG is installed directly onto the host system (Sherlock), GEM can be run interactively using the full path, for example

${CMG_HOME}/gem/2023.10/linux_x64/exe/

or just (because the path is defined),

gm202310.exe

Note that the latter syntax is prefered, since the sysadmin will likely modify the specific installation directory. The same principle applies to containerized versions – the PATH variable in the container is set to find the CMG executables, and softlink shortcuts might also be provided (eg., /opt/CMG/bin/gm --> /opt/CMG/gem/2026.20/linux_x64/exe/gm202620.exe).

GEM will then request an input file and write output using a default naming cvonvention. Alternatively, the input file and output format can be specified – as will be necessary for batch scripts. For example, to define the simulation from a file input.dat, and output to files like my_output.*,

gm201810.exe -f input.dat -o my_output

See above for sample jobs.

Parallel computation

By itself, CMG will not automatically parallelize computation; CMG must be told how to parallelize, with a combination of runtime parameters and environment variables – see below for details. Note that while CMG is capable of supporting MPI (see Appendix A below), current licensing might not support it; parallelization is most likely limited to single node, OMP-like paradigms. When running CMG programs, use the -pnthreads and -prasol parameters to define the number of threads and processors, eg

CMG_exec gm -f /opt/CMG/gem/2026.20/tpl/geo/gmgeo001.dat -o gm01.out --pnthreads ${SLURM_CPUS_PER_TASK} -parasol ${SLURM_CPUS_PER_TASK} -doms

Runtime parameters

  • -f {input_file_path}: full path and file name of input data file (usually ends in .dat)
  • -o {file_path_root}: Output files will be file_path_root.irf, .mrf, .out, .rst, .sr3 .
  • `-pnthreads {n_threads}
  • -parasol {n_cores}: Parallel solutions, (parsol < n_tasks x pnthreds). Note that MPI (ntasks>1) may not be supported by current licensing.
  • -doms
  • -solverg: Something to do with parallelization? It is mentioned as a keyword along with parasol
  • -combinative : A combinative solver, pre-conditioner. Supposed to reduce the required number of solver iterations. Can it be used in combination with parasol?
  • -htuse : Use hyperthreads. This is not recommended. Testing has shown that hyperthreads degrade performance.

Environment varialbes:

These variables affect how parallelization is handled. Different setting might be optimal for various runtime scenarios. Some pertinent issues:

  1. How many cores, processors, nodes
  2. Are multiple CMG jobs running on the same machine
  3. With or with out a job scheduler? Aka, are the jobs competing for resources?

Some environmebnt variables include:

  • KMP_AFFINITY: Controls allocation of job components onto processors
    • KMP_AFFINITY=compact,0 : Allocates load to as few processors as possible; cores can share memory. Recommended for HPC
    • KMP_AFFINITY=compact,1 : Allocates load to as many processors as possible. Increases cross-talk between cores/processors, but might make more efficient use of memory in some cases, or can be optimal for processors that can run on boosted clock speeds.
    • To run multiple simulations on one node without a scheduler, do not set this variable.
  • OMP_SCHEDULE : OpenMP threads parameter (aka, how many OpenMP threads to use)
    • Typically: OMP_SCHEDULE=static,1 gm201810.exe -f TC12_GEM_500Grid_106_C_2800_psi_VM4.dat -o my_output -parasol 4

Appendix A: MPI compatible run script (not compatible with current licensing)

The current Stanford license for CMG does not support MPI, but in the even that might change, the below script – most likely with minor modifications, should run a container in MPI mode. Note the container MPI does not support Infiniband, so some Sherlock defaults need to be overridden. Note also that intel-onaapi-mpi@2021.14 appears to be sufficiently compatible. If more precise compatibility is required, newer versions of Spack include @2021.16. Additionally, setting up LMOD modules based Intel’s oneapi-mpi installer build is, some might suggest, not straight forward, but the approach taken by this script – to define and bind to the $IMP will be straight forward, if the Intel installer is used.

Modifications to this script, to run on Sherlock include:

  • File paths for SIF (version dependent) and IMPI
    • The IMPI definitions provided will work, but see notes above if a different IMPI is required.
  • set FABRIC=tcp
#!/bin/bash
# Run CMG GEM in MPI (distributed-memory) mode from the Apptainer container.
#
# The container ships the MPI build (gm202620_mpi.exe) but not the Intel MPI
# runtime it links against (libmpi.so.12, libmpifort.so.12), so the host's
# Intel MPI 2021.16 (the version GEM was built with) is bound in. Slurm starts
# the ranks via PMI2.
#
# Usage (inside an allocation, or from an sbatch script):
#   ./run_gem_mpi.sh <input.dat> <output_basename> [extra GEM args...]
# Env:
#   FABRIC=tcp|verbs   interconnect for inter-node traffic (default: verbs)
#   PNTHRDS=K          threads per rank (default: $SLURM_CPUS_PER_TASK or 1)
#   PARASOL=P          PARASOL classes (default: ranks*threads, must be >= N*K)
set -euo pipefail

SIF=/home/groups/sh_s-dss/share/sdss/software/x86_64_arch/CMG/cmg_202620.sif
EXE=/opt/CMG/gem/2026.20/linux_x64/exe/gm202620_mpi.exe
IMPI=/home/groups/sh_support/share/spack_SRC/spack/opt/spack/linux-x86_64/intel-oneapi-mpi-2021.16.0-jkwyb6okeknyix2tnq2fwkahqv42lnba/mpi/2021.16

DAT=${1:?usage: $0 <input.dat> <output_basename> [extra args]}
OUT=${2:?usage: $0 <input.dat> <output_basename> [extra args]}
shift 2

NTASKS=${SLURM_NTASKS:?run inside a Slurm allocation}
PNTHRDS=${PNTHRDS:-${SLURM_CPUS_PER_TASK:-1}}
PARASOL=${PARASOL:-$(( NTASKS * PNTHRDS ))}
FABRIC=${FABRIC:-verbs}

BINDS="$IMPI,/usr/lib64/libpmi2.so.0.0.0:/host/lib/libpmi2.so"
LDPATH="$IMPI/lib:$IMPI/opt/mpi/libfabric/lib"

export APPTAINERENV_I_MPI_PMI_LIBRARY=/host/lib/libpmi2.so
export APPTAINERENV_I_MPI_FABRICS=shm:ofi
export APPTAINERENV_FI_PROVIDER_PATH=$IMPI/opt/mpi/libfabric/lib/prov
export APPTAINERENV_OMP_NUM_THREADS=$PNTHRDS

case $FABRIC in
  tcp)
    export APPTAINERENV_FI_PROVIDER=tcp ;;
  verbs)
    # The container has no InfiniBand userspace; bind the host's.
    for f in /usr/lib64/libibverbs.so.1* /usr/lib64/librdmacm.so.1* /usr/lib64/libnl-3.so.200* /usr/lib64/libnl-route-3.so.200*; do
      BINDS="$BINDS,$f:/host/lib/$(basename "$f")"
    done
    BINDS="$BINDS,/usr/lib64/libibverbs:/usr/lib64/libibverbs,/etc/libibverbs.d,/dev/infiniband"
    LDPATH="$LDPATH:/host/lib"
    export APPTAINERENV_FI_PROVIDER="verbs;ofi_rxm" ;;
  *) echo "unknown FABRIC=$FABRIC" >&2; exit 1 ;;
esac
export APPTAINERENV_LD_LIBRARY_PATH=$LDPATH

echo "GEM MPI: ranks=$NTASKS nodes=${SLURM_NNODES:-?} pnthrds=$PNTHRDS parasol=$PARASOL fabric=$FABRIC"
srun --mpi=pmi2 --ntasks="$NTASKS" \
  apptainer exec --bind "$BINDS" "$SIF" \
  "$EXE" -f "$DAT" -o "$OUT" -doms -parasol "$PARASOL" -pnthrds "$PNTHRDS" "$@"