Skip to content

Running kGWASflow on Sherlock

(NOTE: This document is a work in progress, aided heavily by AI. Please excuse any errors, confusion, or… commentary, until it can be properly reviewed and edited.)

Overview

This guide covers the basics of running kgWASflow jobs on Sherlock using the py-kgwasflow module, setting up and running a job, and tuning parallelization.

Native conda/mamba installs of kGWASflow on Sherlock might be complicated by a requirement for GLIBC@ >=2.28, and a handful of other version matching requirements. At the time of writing (circa, Aug. 2026), Sherlock HPC is running the CENTOS-7 operating system, which is based on GLIBC@2.17, and one or more kGWASflow modules might conflict with syntax related to the Python SetupTools module.

GLIBC version conflicts can often be worked around by compiling dependencies from source, sometimes by not compiling from source (it can be difficult to understand which approach to take, and when), or by selecting older versions of some packages. GLIBC conflicts can arise with various conda installers; a successful workaround is often to simply download an older installer, which will still install the same conda as the newer installer package.

This document will cover some of these work-arounds, at lest summarily. Alternatively, A centrally built module is provided which sidesteps these issues.

Parallel jobs and memory management

Parallelization in kGWASflow is uneven, depending on the parallel capabilities of each module in the pipeline. The general idea is (seems to be…) that:

  1. The parent job is run with an awareness of a total thread count via the -t flag, eg. kgwasflow -t $OMP_NUM_THREADS
  2. Parallelization instructions or thread-counts (n_threads) can be passed to individual sub-tasks (pipeline steps)
  3. The parent job will then attempt to run multiple instances of steps where the local n_threads is less than the total -t threads.

The choice, then – of how each step of the job is parallelized affects compute speed performance and memory consumption. Typically – but not always, increasing the parallelization of a given step (up to $OMP_NUM_THREADS) will have diminishing returns, as per parallelization overhead and considering the general geometry of the problem being calculated (eg., realistically - how many pieces can a loop be split into?). In contrast, launching multiple instances of a step might (will likely…) require additional memory. Additionally, some steps might be fundamentally serial in a way that each instance can thread-parallelize, but it is not possible (or supported) to run multiple steps simultaneously.

Generally then, it is important to have some understanding of how each step of the workflow can run in parallel, then define the overall parallelization strategy to optimize compute speed performance and memory requirements.

kGWASflow: Operational overview

kGWASflow is a SnakeMake pipeline. Various scripts and codes are stored in the kGWASflow software directory (where the SW is “installed”); the program is then run in a working directory, eg $SCRATCH/my_kgwasflow_project/workdir_experiment_, which we will call $WORKDIR. The workflow will create a bunch of conda environments, scripts, etc. in $WORKDIR. It will then run the various pipeline steps and store results under $WORKDIR/results.

Real workflows will almost certainly require a customized configuration, $CUSTOM_CONFIG/config.yaml. Snakemake/kGWASflow will append this to a default config.yaml file, which is created during the kgwasflow --init phase. As will be covered below, to start a fresh pipeline run, only the $CUSTOM_CONFIG directory – which should contain any required data, configuration files, etc., needs to be copied to $WORKDIR. It is generally recommended then, that a clean copy of this directory be kept in a safe place, then copied to $WORKDIR at runtime. This simplifies the run and re-run protocol to simply rm -rf deleting $WORKDIR and starting a workflow from the beginning. Of course, more advanced operational modes or restarting an interrupted run might be handled differently (ie, not deleting $WORKDIR and picking up from a partially completed pipeline).

Another important consideration, for maintainability and portability, is the use of absolute or relative paths in the various configuration files. In summary, a handful of files point to data and possibly other supporting configuration files. Especially for data, it might be desirable to define these as absolute paths, to avoid the time and cost to copy those data for each run. On the other hand:

  1. This restricts the portability of the job, presumably to group members on a given compute platform, which might be fine.
  2. It either pushes active compute to non-performant permanent storage (ie, reading directly from $OAK), requires the (mis-)use of $SCRATCH as non-ephemeral (even if short-term) storage, or requires a more complicated workflow to mitigate those two circumstances.

Therefore, it is generally recommended that a data copy, from a permanent repo to $SCRATCH, be integrated into the workflow. This also facilitates running multiple jobs simultaneously, since each job runs in its own separate $WORKDIR, and generally simplifies the workflow and contributes to lower likelihoods of making mistakes.

1. Sherlock Installations

1.b Loading the module

module use /home/groups/sh_support/share/modules/modulefiles
module load py-kgwasflow/1.3.0

If the module isn’t found, refresh the module cache first:

module --ignore-cache load py-kgwasflow/1.3.0

All of the dependencies for kgwasflow should load with the module – eg., you do not need to load gcc/ separately for the module to run correctly, even though it was built against gcc/12.4.0.

1.b Apptainer

An Apptainer/Singularity container build is also available as a fallback:

/home/groups/sh_support/share/software/no_arch/py-kgwasflow/1.3.0/apptainer/kgwasflow_basic.sif

It can be run like,

apptainer exec --writable-tmpfs \
  /home/groups/sh_support/share/software/no_arch/py-kgwasflow/1.3.0/apptainer/kgwasflow_basic.sif \
  kgwasflow init

The Container implementation should be considered if you expect to port your job to alternate platforms – eg., as part of a multi-institutional collaboration or porting the job to public HPC platforms (eg. NSF ACCESS or NASA compute). On Sherlock however, the module load implementation is the recommended path.

2. Setting up a working directory

kGWASflow (via Snakemake) writes a large number of intermediate files and per-rule conda environments into whatever directory you run it from. Keep your source configuration/data separate from the disposable working directory so a broken run can be wiped and restarted cleanly:

  • srcdir — a stable location (e.g. $GROUP_HOME or $OAK) holding your config directory (config.yaml, samples.tsv, phenotypes.tsv, and any reference/genotype/phenotype data).
  • workdir — a scratch location (e.g. $SCRATCH) where you copy the config directory and actually run the pipeline.
WORKDIR=$SCRATCH/my_kgwas_experiment/workdir_1
mkdir -p $WORKDIR
#
cp -r $SRCDIR/config_myrun $WORKDIR/
cd $WORKDIR
kgwasflow init
  • Relative Paths Recommended: paths inside your config files (samples:, phenotypes:, fasta:, etc.) should typically be relative to the working directory, not absolute paths. Absolute paths baked in from someone else’s run directory will break for you. Fix them in bulk with sed, e.g.:
sed -i 's|/scratch/groups/mylab/old/path/config_myrun/|config_myrun/|g' \
  config_myrun/samples.tsv config_myrun/phenotypes.tsv config_myrun/config.yaml

To reset a broken run, removing logs, results, scripts, and .snakemake is generally sufficient and lets you restart cleanly:

rm -r logs results scripts .snakemake

3. Running a job

A minimal SLURM batch script:

#!/bin/bash
#SBATCH --job-name=run_kgwas
#SBATCH --time=10:00:00
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=16
#SBATCH --partition=normal
#SBATCH --mem-per-cpu=8g

module purge
module use /home/groups/sh_support/share/modules/modulefiles
module load py-kgwasflow/1.3.0

WORKDIR=$(pwd)/workdir
CONFIG_SRC=/path/to/your/config_myrun

mkdir -p ${WORKDIR}
cp -r ${CONFIG_SRC} ${WORKDIR}/
cd ${WORKDIR}

kgwasflow init

# --unlock is harmless on a fresh run and necessary if a previous run
# was interrupted / left a lockfile behind.
kgwasflow run -c config_myrun/config.yaml --unlock
kgwasflow run -t ${SLURM_CPUS_ON_NODE} -c config_myrun/config.yaml --conda-frontend mamba

Notes:

  • kgwasflow is a Snakemake wrapper. -t <N> sets Snakemake’s total core budget (--cores), not necessarily the thread count of every individual step.
  • --conda-frontend mamba is strongly recommended — it builds the per-rule conda environments dramatically faster than the default conda frontend (roughly an 8x speedup on environment setup in testing).
  • This is a single-task job (one node, threaded/multiprocessing style), not an MPI job — request resources with --ntasks=1 --cpus-per-task=N, not --ntasks=N --cpus-per-task=1.
  • Typical full runs land around 90–130 GB peak memory and a few hours of wall time, though this scales with dataset size and thread count (see below).

--latency-wait

On network filesystems, there can be a short delay between a step finishing and its output file becoming visible. If you see something like:

MissingOutputException ... Missing files after 5 seconds. This might be due
to filesystem latency. If that is the case, consider to increase the wait
time with --latency-wait:

add --latency-wait 60 (or higher) to the kgwasflow run invocation. Note that this error can also be a red herring: if a step legitimately failed to produce output (e.g. insufficient data, a real crash), increasing --latency-wait won’t help — check the step’s log first.

All of that said, if a job step is looking for files that appear to have not been created, it is more likely that the previous step did not run correctly and may have exited without throwing an error, or the error message is buried in an obscure log somewhere.

4. Parallelization configurations

As discussed above, parallelization in kGWASflow – and most SnakeMake workflows in general, is uneven. Generally, and particularly for kGWASflow, there are two “dials”: how many total cores Snakemake has to work with, and how many threads each individual rule is allowed to use. For example,

kgwasflow run -t ${SLURM_CPUS_ON_NODE} \
  -c config_myrun/config.yaml \
  --snake-default \
  --conda-frontend mamba
  --set-threads cutadapt=${SLURM_CPUS_ON_NODE} kmc_count=${SLURM_CPUS_ON_NODE}  \

or specifying to run merge_kmers jobs with the maximum available threads,

kgwasflow run -t ${SLURM_CPUS_ON_NODE} -c config_xvar_kgwas_red/config.yaml  --conda-frontend mamba --set-threads cutadapt=${SLURM_CPUS_ON_NODE} kmc_count=${SLURM_CPUS_ON_NODE} kmers_gwas=$SLURM_CPUS_ON_NODE merge_kmers=$SLURM_CPUS_ON_NODE
  • -t <N> — total core budget passed to Snakemake (--cores).
  • --set-threads rule=N [rule2=N2 ...] — overrides the thread count for specific rules at runtime, rather than editing the bundled workflow/config/config.yaml. Useful rules to tune include cutadapt and kmc_count (k-mer counting), which are the more parallelizable steps.
  • --snake-default — use kGWASflow’s bundled default config layered under your -c config, rather than only your custom config. Some runs failed with confusing errors (e.g. a “FLAG 00 should be equal to zero” k-mer canonicalization error) until this flag was added; it’s worth including by default.
  • -n / --dry-run — shows a preview table of each rule with its min/max thread counts, useful for sanity-checking before submitting.

Memory typically scales with thread/task count, not just total request. Typically, kGWASflow will attempt to use all of the threads it is told it can use, as per the -t flag, by launching multiple instances of a given step. For example,

kgwasflow run -t ${SLURM_CPUS_ON_NODE} ... --set-threads cutadapt=${SLURM_CPUS_ON_NODE}

Will attempt to run one instance, of the cutadapt step, at a time, using all available threads for that instance. Nominally this approach should require the least amount of memory – because it launches only one instance of cutadapt, but might suffer from parallelization fatigue (poor parallelization performance scaling) for high thread-counts.

Conversely,

kgwasflow run -t ${SLURM_CPUS_ON_NODE} ... --set-threads cutadapt=1

will (probably…) attempt to run ${SLURM_CPUS_ON_NODE} simultaneous instances of cutadapt. In principle, this approach is likely to be faster than the previous example, because each instance – by itself, is serial and so there is no parallel scaling penalty, within each process. On the other hand, this approach makes ${SLURM_CPUS_ON_NODE} separate copies of the step, which can lead to Out of Memory (OoM) job failures.

Optimal job performance, then requires some understanding of how each step parallelizes and how to manage that parallelization. Some practical observations, comments, and Sherlock specifications:

  • Sherlock 3 and 4 (SH3, SH4) machines typically have either 8GB/CPU (all SH3 and SH4 CBASE); the higher cpu-count machines (SH4_CPERF, SH4_CSCALE) have 6GB/CPU, but higher CPU counts.
  • Generally then, it is good practice to design jobs to run on 6-8 GB/CPU
  • To reserve all the memory on a machine, use --mem=0, but be sure the rest of your request is consistent with using the whole machine, and specify an appropriate machine – for example, --cpus-per-task=32 --mem=0 --constraint="[CLASS:SH3_CBASE|CLASS:SH3_CBASE.1]"
  • If a job terminates with OoM:
    • Request more memory?
    • But as discussed, if you are requesting much more than 8GB/CPU, it is probably worth a more sophisticated approach
    • You might limit memory use by reducing the threads, ` -t {n < $OMP_NUM_THREADS}`, which will reduce the memory requirement for multi-instance parallelization, but you’ll be leaving compute capacity on the table.
    • Ideally, increase OMP parallelization for steps that exceed the memory allocation, eg. run kGWASflow with --set-threads cutadapt=4 merge_kmers=4, or more extremely, -t $OMP_NUM_THREADS --set-threads cutadapt=$OMP_NUM_THREADS merge_kmers=$OMP_NUM_THREADS
  • You can monitor CPU performance and memory use with SLURM accounting (sacct) or seff. Typically, expect CPU efficiency for a full workflow to be relatively low, due to environment builds and other IO-bound steps, but it is often possible to observe changes in this metric for different parallelization configurations. Memory use efficiency should be more apparent.

When structuring a SLURM resource request, to accommodate kGWASflow/SnakeMake requirements, consider not only how much memory and how many CPUs you request, but also how you request them. For example:

# Single-node, moderate parallelism
sbatch --partition=normal --ntasks=1 --cpus-per-task=16 --mem-per-cpu=8g my_script.sh

# Whole node
sbatch --partition=normal --ntasks=1 --cpus-per-task=32 --mem=0 --exclusive my_script.sh

# Bigger memory partition
sbatch --partition=normal,bigmem --ntasks=1 --cpus-per-task=16 --mem-per-cpu=16g my_script.sh

5. Some common errors

  • NOT_ENOUGH_DATA / missing pass_threshold_5per: this is not a crash — the k-mers GWAS step checks your --min_data_points threshold against the number of overlapping samples between your phenotype file and the k-mers table/kinship matrix, and cleanly exits early if too few samples qualify. If you see a NOT_ENOUGH_DATA file under results/kmers_gwas/<pheno>/, check your actual sample count against min_data_points in config.yaml (default is 30) — lower the threshold or add more qualifying samples.
  • multiqc / pkg_resources errors: newer setuptools (>=81) dropped the deprecated pkg_resources module that older multiqc/Snakemake wrapper environments import. Fix by pinning setuptools<81 in the affected conda environment, or upgrading multiqc to >=1.22 in the wrapper’s environment.yaml.
  • Permission errors writing to the module’s environment files: if you hit a PermissionError on a file under the shared module install path, that’s a bug in our install permissions, not something on the user side — report it so it can be fixed centrally.
  • Always run kgwasflow run ... --unlock before a restart if a previous run was interrupted.