EVO2 on Sherlock

(Document in process; good documentation to come ASAP).

Evo2 is a foundational Genomic model, developed and distributred by the Arc Institute. Source code and some docs can be found in thier GitHub repository:

https://github.com/ArcInstitute/evo2

Background and GLIBC

Their documentation suggests an installation process, but the reality of this experience might be challenging, especially on an older operationg system. In particular, installations using pip, conda or other package management tools are likely to encounter incompatibilities with Sherlock’s older glibc libraries.

As of this writing, Stanford’s Sherlock HPC runs the CentOS-7 operating system, which uses an older version of glibc@2.17. Many newer software packages require – to some degree or another, a newer version of this library. As it is integral to the operating system itself, glibc cannot (realistically) be replaced or upgraded. In some cases, newer codes explicitly reference features of the newer glibc@>2.17, and are therefore more or less irrecoverably incompatible with CentOS-7. It is often possible, however, to compile a code against more or less any version of glibc, and so an error reporting glibc incompatibility can be resolved by recompiling the offending code against the local glibc.

Fortunately, Evo2 appears to fall into this latter category, where pip and conda installations may not succeed directly, but the package can be build if one or a few dependencies are compiled from source. An elegant solution to this can be found at: https://github.com/evo-design/evo2-sherlock-env.

We liberally borrwoed from this solution, using gcc@12.4.0 and venv in place of gcc@9.x and conda, but othwerwise following their general formula to compile Evo2 both natively on Sherlock – where it can be loaded as a module, and in an Apptainer container.

Caches and quotas

Evo2 may pull large reference data sets to cache locations, which are – by default, located in your $HOME space, eg. ~/.cahce/huggingface. These datasets will quickly exceed tghe 15GB quota of your $HOME space on Sherlock, and might even push the comfort zone for your $GROUP_HOME, so it is recommended to redirect these to $SCRATCH or $GROUP_SCRATCH.

Sherlock Modules

Note that a GPU is requried, even for testing. Proof of concept tessts can be run successfully using the “sharded” A100 GPUs in Sherlock’s dev partition, but even most tests will quickly throw an, Out of vRAM error. For interactive operations then, consider a request along the lines of:

salloc --partition=serc --cpus-per-task=8 --mem-per-cpu=8g --gpus=1 --constraint="GPU_GEN:AMP"

Of course, substitute --partition and --constraint to reflect available partitions and hardware.

To enable and use Evo2,

module use /home/groups/sh_support/share/modules/modulefiles
module load evo2/
python -m evo2.test.test_evo2_generation --model_name evo2_7b

You can also batch a test job like,

sbatch -p gpu --cpus-per-task=8 --mem-per-cpu=8g --gpus=1 --constraint="GPU_GEN:AMP" --wrap="module use /home/groups/sh_support/share/modules/modulefiles&&module load evo2/&&python -m evo2.test.test_evo2_generation --model_name evo2_7b" --output=evo2_test.out --error="evco2_test.err"

Evo2 Apptainer

Copying evo2.def and evo2.sif files to $SRC_HOME

# Like with the native build, request a >A100 GPU:
salloc -p serc --cpus-per-task=8 --mem-per-cpu=8g --gpus=1 --constraint="GPU_GEN:AMP"
# launch the container; map the cache to a safe place.
module load evo2/
apptainer shell --bind ${SCRATCH}/CACHE:/CACHE --writable-tmpfs --nv $EVO2_DIR/apptainer/evo2.sif

Note that the --bind statement maps the cache to a safe place (large capacity cache). This ass