Skip to content

Running kGWASflow on Sherlock

(NOTE: This document is a work in progress, aided heavily by AI. Please excuse any errors, confusion, or… commentary, until it can be properly reviewed and edited.)

This guide covers the basics of loading the py-kgwasflow module, setting up and running a job, and tuning parallelization. It’s based on lessons learned supporting kGWASflow (https://github.com/akcorut/kGWASflow) runs on Sherlock (SRCC ticket #97332).

Native conda/mamba installs of kGWASflow are not recommended on Sherlock: the Miniconda/Anaconda installers require GLIBC >=2.28, but Sherlock’s login and compute nodes currently run GLIBC 2.17, so the installer fails outright. A centrally built module is provided instead, which sidesteps this problem.

1. Loading the module

module use /home/groups/sh_support/share/modules/modulefiles
module load py-kgwasflow/1.3.0

If the module isn’t found, refresh the module cache first:

module --ignore-cache load py-kgwasflow/1.3.0

You do not need to load gcc/ separately for the module to run correctly, even though it was built against gcc/12.4.0.

An Apptainer/Singularity container build is also available as a fallback:

/home/groups/sh_support/share/software/no_arch/py-kgwasflow/1.3.0/apptainer/kgwasflow_basic.sif

used like:

apptainer exec --writable-tmpfs \
  /home/groups/sh_support/share/software/no_arch/py-kgwasflow/1.3.0/apptainer/kgwasflow_basic.sif \
  kgwasflow init

The module load is the recommended path; the container is a secondary option if you run into module-specific issues.

2. Setting up a working directory

kGWASflow (via Snakemake) writes a large number of intermediate files and per-rule conda environments into whatever directory you run it from. Keep your source configuration/data separate from the disposable working directory so a broken run can be wiped and restarted cleanly:

  • srcdir — a stable location (e.g. $GROUP_HOME or $OAK) holding your config directory (config.yaml, samples.tsv, phenotypes.tsv, and any reference/genotype/phenotype data).
  • workdir — a scratch location (e.g. $SCRATCH) where you copy the config directory and actually run the pipeline.
mkdir -p $WORKDIR
cp -r $SRCDIR/config_myrun $WORKDIR/
cd $WORKDIR
kgwasflow init

Important: paths inside your config files (samples:, phenotypes:, fasta:, etc.) should typically be relative to the working directory, not absolute paths. Absolute paths baked in from someone else’s run directory will break for you. Fix them in bulk with sed, e.g.:

sed -i 's|/scratch/groups/mylab/old/path/config_myrun/|config_myrun/|g' \
  config_myrun/samples.tsv config_myrun/phenotypes.tsv config_myrun/config.yaml

To reset a broken run, removing logs, results, scripts, and .snakemake is generally sufficient and lets you restart cleanly:

rm -r logs results scripts .snakemake

3. Running a job

A minimal SLURM batch script:

#!/bin/bash
#SBATCH --job-name=run_kgwas
#SBATCH --time=10:00:00
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=16
#SBATCH --partition=normal
#SBATCH --mem-per-cpu=8g

module purge
module use /home/groups/sh_support/share/modules/modulefiles
module load py-kgwasflow/1.3.0

WORKDIR=$(pwd)/workdir
CONFIG_SRC=/path/to/your/config_myrun

mkdir -p ${WORKDIR}
cp -r ${CONFIG_SRC} ${WORKDIR}/
cd ${WORKDIR}

kgwasflow init

# --unlock is harmless on a fresh run and necessary if a previous run
# was interrupted / left a lockfile behind.
kgwasflow run -c config_myrun/config.yaml --unlock
kgwasflow run -t ${SLURM_CPUS_ON_NODE} -c config_myrun/config.yaml --conda-frontend mamba

Notes:

  • kgwasflow is a Snakemake wrapper. -t <N> sets Snakemake’s total core budget (--cores), not necessarily the thread count of every individual step.
  • --conda-frontend mamba is strongly recommended — it builds the per-rule conda environments dramatically faster than the default conda frontend (roughly an 8x speedup on environment setup in testing).
  • This is a single-task job (one node, threaded/multiprocessing style), not an MPI job — request resources with --ntasks=1 --cpus-per-task=N, not --ntasks=N --cpus-per-task=1.
  • Typical full runs land around 90–130 GB peak memory and a few hours of wall time, though this scales with dataset size and thread count (see below).

--latency-wait

On network filesystems, there can be a short delay between a step finishing and its output file becoming visible. If you see something like:

MissingOutputException ... Missing files after 5 seconds. This might be due
to filesystem latency. If that is the case, consider to increase the wait
time with --latency-wait:

add --latency-wait 60 (or higher) to the kgwasflow run invocation. Note that this error can also be a red herring: if a step legitimately failed to produce output (e.g. insufficient data, a real crash), increasing --latency-wait won’t help — check the step’s log first.

4. Parallelization configurations

There are two dials: how many total cores Snakemake has to work with, and how many threads each individual rule is allowed to use.

kgwasflow run -t ${SLURM_CPUS_ON_NODE} \
  -c config_myrun/config.yaml \
  --snake-default \
  --conda-frontend mamba
  --set-threads cutadapt=${SLURM_CPUS_ON_NODE} kmc_count=${SLURM_CPUS_ON_NODE}  \

or specifying to run merge_kmers jobs with the maximum available threads,

kgwasflow run -t ${SLURM_CPUS_ON_NODE} -c config_xvar_kgwas_red/config.yaml  --conda-frontend mamba --set-threads cutadapt=${SLURM_CPUS_ON_NODE} kmc_count=${SLURM_CPUS_ON_NODE} kmers_gwas=$SLURM_CPUS_ON_NODE merge_kmers=$SLURM_CPUS_ON_NODE
  • -t <N> — total core budget passed to Snakemake (--cores).
  • --set-threads rule=N [rule2=N2 ...] — overrides the thread count for specific rules at runtime, rather than editing the bundled workflow/config/config.yaml. Useful rules to tune include cutadapt and kmc_count (k-mer counting), which are the more parallelizable steps.
  • --snake-default — use kGWASflow’s bundled default config layered under your -c config, rather than only your custom config. Some runs failed with confusing errors (e.g. a “FLAG 00 should be equal to zero” k-mer canonicalization error) until this flag was added; it’s worth including by default.
  • -n / --dry-run — shows a preview table of each rule with its min/max thread counts, useful for sanity-checking before submitting.

Memory scales with thread/task count, not just total request. kGWASflow’s parallelization appears to principally run multiple copies of job steps in a way that increases memory consumption with increased thread count – depending on the configuration. Practical implications:

  • If SLURM grants you more CPUs than you actually pass via -t, that’s fine — but if you pass a large -t (or per-rule --set-threads) without enough memory to back it, you’ll hit OUT_OF_MEMORY kills.
  • As a rule of thumb from observed runs on SH3-class nodes (8 GB/CPU): -t 8 per task with --mem-per-cpu=8g and --cpus-per-task=8 completed reliably with headroom; pushing thread count up to 16–24 without a matching memory increase reliably OOM’d.
  • If you want the whole node’s memory regardless of CPU count requested, use --mem=0 (with --exclusive if you want the whole node reserved).
  • Observed CPU efficiency (via seff) on real runs is often low (13–20%), largely because conda/mamba environment builds and several I/O-bound steps are single-threaded. This is expected — don’t read low CPU efficiency alone as a sign something is misconfigured.

Example resource progression to try, in order, if a run needs more memory:

# Single-node, moderate parallelism
sbatch --partition=normal --ntasks=1 --cpus-per-task=16 --mem-per-cpu=8g my_script.sh

# Whole node
sbatch --partition=normal --ntasks=1 --cpus-per-task=32 --mem=0 --exclusive my_script.sh

# Bigger memory partition
sbatch --partition=normal,bigmem --ntasks=1 --cpus-per-task=16 --mem-per-cpu=16g my_script.sh

5. Common gotchas

  • NOT_ENOUGH_DATA / missing pass_threshold_5per: this is not a crash — the k-mers GWAS step checks your --min_data_points threshold against the number of overlapping samples between your phenotype file and the k-mers table/kinship matrix, and cleanly exits early if too few samples qualify. If you see a NOT_ENOUGH_DATA file under results/kmers_gwas/<pheno>/, check your actual sample count against min_data_points in config.yaml (default is 30) — lower the threshold or add more qualifying samples.
  • multiqc / pkg_resources errors: newer setuptools (>=81) dropped the deprecated pkg_resources module that older multiqc/Snakemake wrapper environments import. Fix by pinning setuptools<81 in the affected conda environment, or upgrading multiqc to >=1.22 in the wrapper’s environment.yaml.
  • Permission errors writing to the module’s environment files: if you hit a PermissionError on a file under the shared module install path, that’s a bug in our install permissions, not something on the user side — report it so it can be fixed centrally.
  • Always run kgwasflow run ... --unlock before a restart if a previous run was interrupted.