--- title: "CESM (Community Earth System Model)" canonical: "https://kb.uconn.edu/space/SH/28608921601/CESM%20(Community%20Earth%20System%20Model)" format: markdown --- > Macro (toc) > ⚠️ This guide is Work in Progress and not fully completed yet, stay tuned. # Using CESM on the cluster This is a quick start guide for CESM on Storrs HPC UCONN cluster. You will still need to read the CESM User Guide and work with your fellow research group members to design and run your simulations, but this guide will cover the basics that are specific to running CESM on the Storrs HPC UCONN cluster. ## Setup Conda and install Subversion with HTTPS support. To be able to download the necessary Input data, subversion is needed with HTTPS support. Conda is the best way to install Subversion with HTTPS support as the subversion Conda package comes pre-built with all the necessary libraries. [https://kb.uconn.edu/space/SH/26079723879/Miniconda+Environment+Set+Up](https://kb.uconn.edu/space/SH/26079723879/Miniconda+Environment+Set+Up) Once the intial conda setup is complete after using the above guide, here are the conda commands that can be used to install Subversion. Submit an interactive SLURM srun job for the initial setup for CESM. ``` srun --x11 -N 1 -n 126 -p general --mem=492G --exclude=cn[473-479] --exclusive --pty bash ``` Wait for a node to assign, then: If conda was freshly installed, a user might need to add the conda-forge and bioconda channels to their channel list. To add the conda-forge and bioconda channels, the following command can be used: ``` conda config --add channels conda-forge conda config --add channels bioconda ``` Create a conda environment with any name for subversion and install Python 3.9 within the environment. ``` conda create -n svn_py39 -c conda-forge python=3.9 subversion ``` Activate the newly created svn_py39 conda environment ``` conda activate svn_py39 ``` Install LibXML in this conda environment: ``` conda install -c bioconda perl-xml-libxml ``` Confirm that conda installed Subversion with HTTPS support ``` svn --version ``` You should see the following options at the bottom of the above output that confirms that HTTPS support was enabled for subversion: ``` * ra_svn : Module for accessing a repository using the svn network protocol. - with Cyrus SASL authentication - handles 'svn' scheme * ra_local : Module for accessing a repository on local disk. - handles 'file' scheme * ra_serf : Module for accessing a repository via WebDAV protocol using serf. - using serf 1.3.10 (compiled with 1.3.10) - handles 'http' scheme - handles 'https' scheme ``` Once the output shows up, that confirms subversion with HTTPS was installed in conda successfully. ## Clone the CESM github repository and setup the CESM source files Depending on where you would like to run your CESM jobs, you can clone the CESM github repository in a location that you would like to manage. This can be under a local user’s home directory or a team shared directory. The following will clone the CESM 2.1.5 release and create a directory called CESM-release-cesm2.1.5 ``` git clone -b release-cesm2.1.5 https://github.com/ESCOMP/CESM.git CESM-release-cesm2.1.5 ``` Then one can cd into this new directory: ``` cd CESM-release-cesm2.1.5 ``` Once in the CESM source directory above, checkout all externals: ``` ./manage_externals/checkout_externals ``` > ℹ️ This is a very important step or most of the CESM suite will not be available. To verify that the externals did in fact get checked out, the following command can be used: ``` ./manage_externals/checkout_externals -S ``` ## Load module dependencies for CESM: ``` module load gcc/12.2.0 module load python/3.9.24 module load cmake/3.23.2 module load ucx/1.17.0 module load ucc/1.3.0 module load openmpi/4.1.5-ucx-ucc-gcc12 module load hdf5/1.14.3-openmpi415 module load netcdf-fortran/4.6.1-openmpi415 module load openblas/0.3.26 ``` ## Updating the CESM config .xml files (work in progress, stay tuned) CESM uses XML files for setting up the CESM job environment. These XML files control how a case is created, setup, built, and submitted to the HPC cluster. We have template CESM XML files located here: > ℹ️ /gpfs/sharedfs1/admin/hpc2.0/apps/cesm/2.1.5/CESM-release-cesm2.1.5/cime/config/cesm/machines The ones that are needed are **config_machines.xml**, **config_batch.xml**, and **config_compilers.xml** Copy/rsync each XML file to your local* CESM-release-cesm2.1.5/cime/config/cesm/machines* directory. **Here is an example to copy/rsync the XML files to your CESM source location, if your CESM 2.1.5 source directory resides under your home:** ``` rsync -a --progress /gpfs/sharedfs1/admin/hpc2.0/apps/cesm/2.1.5/CESM-release-cesm2.1.5/cime/config/cesm/machines/config_*.xml /gpfs/homefs1/`whoami`/CESM-release-cesm2.1.5/cime/config/cesm/machines/ ``` Once the files are copied over, cd into your *“CESM-release-cesm2.1.5/cime/config/cesm/machines/”* directory. Perform the following sed command to replace all mentions of **“netidhere”** within the file with your netid like so: ``` sed -ie 's/netidhere/idk12345/g' /gpfs/homefs1/`whoami`/CESM-release-cesm2.1.5/cime/config/cesm/machines/config_machines.xml ``` > ℹ️ Replace idk12345 with your netid in the above sed command. Once the sed is performed, update the **config_machines.xml **file within the **** section at the **bottom** of the file. The paths that would need to be updated are: ``` /gpfs/homefs1/idk12345/YourCESMOutputDirectory /gpfs/homefs1/idk12345/YourCESMOutputDirectory/archive/$CASE ``` The fields that say **idk12345 **should have been replaced with your netid. If the Output CESM directory does not already exist, exit out of the config_machines.xml file and create the output directory: Example: ``` mkdir -pv /gpfs/homefs1/`whoami`/YourCESMOutputDirectory ``` Once your output directory is created, open up the **config_machines.xml** file, go back down to the bottom of the file, and update the two paths above to the exact location where you would like your CESM output directory to be. Also, the previous sed command should have renamed the following three lines with your netid instead of idk12345. ``` /gpfs/homefs1/idk12345/miniconda3/envs/svn_py39/bin/svn /gpfs/homefs1/idk12345/miniconda3/envs/svn_py39/lib/perl5/site_perl /cm/shared/apps/cmake/3.23.2/cmake-3.23.2-linux-x86_64/bin:/gpfs/homefs1/idk12345/miniconda3/envs/svn_py39/bin:/gpfs/sharedfs1/admin/hpc2.0/apps/cesm/2.1.5/cesm_wrappers:/gpfs/sharedfs1/admin/hpc2.0/apps/openmpi/4.1.5-ucx-ucc-gcc12/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin ``` If your svn_py39 conda environment is in a different path outside of your home directory, the mentions of miniconda in the above three lines need to be updated to the exact path where your svn_py39 conda environment is located. > ℹ️ The above 3 lines are very important to tell CESM to use Subversion within your conda environment. Save the file after all the changes have been made. ## Creating cases Within CESM, each simulation is called a “case”. If the XML files are configured correctly, a case can be created by running CESM’s create_newcase script that resides under the following directory: ``` code_base/cime/scripts/create_newcase ``` So, if the CESM code base was cloned from github under a user’s home directory, the format will be (for version 2.1.5): ``` /home/netidhere/CESM-release-cesm2.1.5/cime/scripts ``` The path can change depending on where CESM was cloned from github and where the CESM externals got checked out. Here is an example on how to create a test F1850 case: > 📝 ./create_newcase --case /gpfs/homefs1/netidhere/path/to/where/you/would/like/cases/tobecreated/F1850_test --compset F1850 --res f09_f09_mg16 --mach uconn_slurm --run-unsupported So, here is a full example to create the case under a user’s home under an existing directory called **cesm_cases**: ``` ./create_newcase --case /gpfs/homefs1/netidhere/cesm_cases/F1850_test --compset F1850 --res f09_f09_mg16 --mach uconn_slurm --run-unsupported ``` > ℹ️ Replace the string above with your netid If the XML files are configured correctly, the case should create successfully without any errors. ## Setting up cases CESM allows users to setup and edit their cases with specific options to tailor to their job submissions on how many resources they would like to use, etc. To setup a case, cd into the newly created case directory for the specific case. Keeping with our example, the case directory would be: ``` cd /gpfs/homefs1/netidhere/cesm_cases/F1850_test ``` Replace **“netidhere”** with your netid. Once inside the case directory, setting up the case to allocate/request specific resources is very customizable. It all depends on the needs of the case and how many resources a user would like to request when the case gets submitted to resources on HPC. CESM uses the ./case.setup script to setup a case. ### Setting up the CESM case to submit to different HPC partitions CESM allows users to change variables within a CASE to customize their CESM job submissions to allocated various resources and settings. The following sections will show how to configure a case for the general partition and priority partitions. #### General partition: ``` ./xmlchange PROJECT=PI'sNetidHere ./xmlchange JOB_QUEUE=general ./xmlchange JOB_WALLCLOCK_TIME=12:00:00 ``` > 📝 These must be entered before running the case.setup script. The JOB_WALLCLOCK_TIME declaration can be any value depending on how long you would like the CESM job to run for. If not specified, the CESM case will default to 12 hours for the general job. Then set the QoS: ``` export SLURM_QOS=general ``` The above changes will setup a Submission script with the defaults to look like: ``` #!/bin/bash #SBATCH -N 4 #SBATCH -n 504 #SBATCH --account=PI'sNetidHere #SBATCH -p general #SBATCH -q general #SBATCH --time=12:00:00 ``` To confirm what CESM is assigning by default for the case, the following command can be used: ``` ./xmlquery NTASKS NTHRDS MAX_MPITASKS_PER_NODE MAX_TASKS_PER_NODE ``` To change the values for the number of ntasks and the number of nodes the CESM case will use, the following commands can be used: Here is an example to allocate **2 **nodes: ``` ./xmlchange NTASKS=252 ./xmlchange NTHRDS=1 ``` Each HPC compute node has **126** CPU cores available, changing the value of **NTASKS** will determine how many nodes and cores to allocate to the CESM job. To have CESM save the changes, run the to build the CESM with the changes: ``` ./case.setup ``` If the case.setup command was entered previously, to have CESM take the changes, the following command can be used: ``` ./case.setup --reset ``` To confirm the changes were made, the following command can be used: ``` ./preview_run ``` #### Priority partition: If you have access to HPC priority resources through a Condo model, the priority partition allocation is similar to that of a general job submission from above, however, the only difference is to specify a time limit and change the values for** PROJECT**, **JOB_QUEUE**, and **SLURM_QOS**. The values for **PROJECT,** **JOB_QUEUE, **and** SLURM_QOS** will need to change for your Priority job submission team allocation. ``` ./xmlchange PROJECT=PI'sNetidHere ./xmlchange JOB_QUEUE=priority ./xmlchange JOB_WALLCLOCK_TIME=TimeLimitHere ``` Then set your team’s QoS: ``` export SLURM_QOS=PI'sQoS ``` The above changes will setup a submission script with the defaults to look like: ``` #!/bin/bash #SBATCH -N 4 #SBATCH -n 504 #SBATCH --account=PI'sNetidHere #SBATCH -p priority #SBATCH -q PI'sQoS #SBATCH --time=TimeLimitHere ``` The Time Limit would need to be set if the Priority job needs to run more than 24 hours. To confirm what CESM is assigning by default for the case, the following command can be used: ``` ./xmlquery NTASKS NTHRDS MAX_MPITASKS_PER_NODE MAX_TASKS_PER_NODE ``` To change the values for the number of ntasks and the number of nodes the CESM case will use, the following commands can be used: Here is an example to allocate **2 **nodes: ``` ./xmlchange NTASKS=252 ./xmlchange NTHRDS=1 ``` Each HPC compute node has **126** CPU cores available, changing the value of **NTASKS** will determine how many nodes and cores to allocate to the CESM job. To have CESM save the changes, run the to build the CESM with the changes: ``` ./case.setup ``` If the case.setup command was entered previously, to have CESM take the changes, the following command can be used: ``` ./case.setup --reset ``` To confirm the changes were made, the following command can be used: ``` ./preview_run ``` ## Building cases After cases have ben setup after using the case.setup script, building a case is very easy. The following command is used to build CESM cases: ``` ./case.build ``` If the case(s) build successfully, proceed to the next step on how to submit the case(s) ## Submitting cases to a compute node > ℹ️ After a case gets built from the case.build CESM script, exit out of the SLURM srun job spawned at the beginning of this guide by typing **exit **in the terminal to go back to a login node. > ℹ️ > ℹ️ We do not want to submit a SLURM job from within a SLURM srun job. Back on the login node, the CESM case can be submitted to the SLURM job scheduler on HPC using the following command: ``` ./case.submit ``` If the CESM case was setup correctly, the case.submit CESM script will submit the case using all the settings that were used when the case was setup. If you would like an email from SLURM for a specific type of job status, you can use the following command when submitting your case to a HPC node(s). ``` ./case.submit --mail-type=end --mail-user=first.lastname@uconn.edu ``` We recommend setting the END Mail type as SLURM will email you after job completion or failure instead of when the job begins, ends, or fails if ALL was set. Replace the MAIL_TYPE with the type of mail notification you would like to have. If everything is successful, the case should submit to a compute node or nodes depending on how many NTASKS were set in the previous step. Confirm with the **shist **command. To view more details about a pending/running job, the following command can be used: ``` scontrol show job ``` Replace with the specific job id for the job. ## Resetting cases to use different settings To reset an existing case to use other values, the following steps need to be performed. ``` ./xmlchange PROJECT=PI'sNetidHere ./xmlchange JOB_QUEUE=priority #general , hi-core, or priority ./xmlchange JOB_WALLCLOCK_TIME=TimeLimitHere ``` Then: ``` export SLURM_QOS=PI'sQoS ``` Change the number of NTASK and threads: ``` ./xmlchange NTASKS=252 ./xmlchange NTHRDS=1 ``` Reset and setup the case with: ``` ./case.setup --reset ./case.setup ``` Confirm the changes with the following command: ``` ./preview_run ``` Clean and rebuild the case: ``` ./case.build --clean ./case.build ``` Submit the case: ``` ./case.submit --mail-type=end --mail-user=first.lastname@uconn.edu ``` The mail-type option can be begin, end, fail, or all