--- title: "Compiling Software" canonical: "https://kb.uconn.edu/space/SH/26033587644/Compiling%20Software" format: markdown --- > Macro (toc) # Compiling allows you to access to the latest and greatest software. If you have never compiled software before, the process may seem a little involved at first. This guide will help you understand how it all works. First, we use short code snippets to understand a few concepts. Then we will see more elaborate examples. This page assumes: 1. You are familiar with the command line. 2. You are familiar with at least one programming language. 3. You have never written a C, C++, Fortran, or any program that requires compilation. ## Concepts This section explains various `*PATH` variables and `LDFLAGS` to help you troubleshoot compilation and runtime errors. ## PATH The `PATH` controls where the shell searches for programs to run. On the cluster, we frequently change the `PATH` to run programs we installed ourselves. Let's get an appreciation for how the `PATH` works with a short exercise of creating a program and running it. Inside of your shell on the cluster, try to run the command `hello` ``` # Run our first program hello # -bash: hello: command not found ``` The above message tells us there is no program named `hello` in any of the usual places. So then the question is what are the usual places? The command `which` tells us the locations of the usual places it searches for programs: ``` # Where does the computer search for programs to run? which hello # /usr/bin/which: no hello in (/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/usr/lpp/mmfs/bin:/opt/ibutils/bin:/gpfs/gpfs1/slurm/misc/stubl/stubl-master/bin) ``` You can see a list of different directories separated by a colon. This list of directories is nothing but the `PATH` variable: ``` # The "which" program searches for programs in the directories stored in the variable PATH. echo $PATH # /usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/usr/lpp/mmfs/bin:/opt/ibutils/bin:/gpfs/gpfs1/slurm/misc/stubl/stubl-master/bin ``` ``` # Programs like "cat" and "gcc" are stored in /bin and /usr/bin respectively. which cat # /bin/cat which gcc # /usr/bin/gcc ``` A PATH is a particular type of variable called an "environmental" variable. An environmental variable is stored in the shell, and is therefore available to any program run from the shell, like `which`. Let's create a program called `hello`: ``` # Create the directory to store our programs. mkdir -p ~/.apps/hello # Create the program hello echo '#!/bin/bash' > ~/.apps/hello/hello echo 'echo "Hi, there!"' >> ~/.apps/hello/hello # Give the program executable permission. chmod +x ~/.apps/hello/hello # Run hello. ~/.apps/hello/hello # Hi, there! ``` But we can do better! Add `hello` to the PATH so that we don't have to type the directory `~/.apps/hello/` every time we want to run ``` # Tell the shell to first look in ~/.apps/hello before other places. PATH=$PATH:~/.apps/hello # Run hello. hello # Hi, there! ``` Summary - Learned how the `PATH` variable tells the shell where to find software to run. - Created our own program called `hello`. - Learned how to add our program to the `PATH` so that one does not have to remember where it is to located and simply run it by name. ## Libraries To avoid reinventing the wheel, nearly all programs re-use code from shared "libraries". These names of these shared library files end with the extension `.so` However when computer code re-uses these library programs, we sometimes need to tell the compiler the name of the library to use. We will see how to do this in the next section on `LDFLAGS`. ## LDFLAGS Now we are ready to compile our first program. The program below will inspect the high speed InfiniBand network port present on all the nodes. `LDFLAGS` is a variable that tells the compiler what linker flags to pass. (usually includes required libraries) ``` #include #include #include int main() { int i, num_devices; struct ibv_device **dev_list = ibv_get_device_list(&num_devices); printf("Infiniband Devices:\n"); if (! dev_list) printf("None found"); else { for (i = 0; i < num_devices; ++i) printf("%-16s\n", ibv_get_device_name(dev_list[i])); ibv_free_device_list(dev_list); } return 0; } ``` However the program needs to know the name of the InfiniBand library to use. If we neglect to mention the name of the library and try to compile the file directly, it complains about the missing function references: ``` make ib ``` ``` cc ib.c -o ib /tmp/ccKfTgaM.o: In function `main': ib.c:(.text+0x10): undefined reference to `ibv_get_device_list' ib.c:(.text+0x5a): undefined reference to `ibv_get_device_name' ib.c:(.text+0x7a): undefined reference to `ibv_free_device_list' collect2: ld returned 1 exit status make: *** [ib] Error 1 ``` We need to tell the compiler to use the `ibverbs` library: ``` LDFLAGS="-l ibverbs" make ib # cc -l ibverbs ib.c -o ib ./ib # Infiniband Devices: # mlx4_0 ``` The `-l` flag tells the compiler that the next word following it `ibverbs` is the name of the shared library it should use to create the final program we want, `ib`. For those of you familiar with writing code for compiling programs, you might be surprised to see the use of `make` without any input `Makefile`. The reason we can skip having any input `Makefile` is because of the automatic rules feature of `make`; namely `make` knows how to compile C programs without us explaining the variable name substitutions to use. ## Headers Header files are different than shared libraries, in that they are only needed during compilation time and never again at runtime. Whereas library names end with `.so`, header file names end with `.h` for C code (or `.hpp/.hh/.h` for C++ code). Sometimes header files store information in a manner similar to libraries, in that they stored common useful code instead of simply outlining what is contained in the library; some libraries call themselves "header-only libraries" for this reason which in practice means that those libraries don't have any associated `.so` files. ## CPATH The concept of `CPATH` is similar to `PATH`, except instead of executable programs, it controls where the shell searches for *header files* (also called *headers*) for programs to use. This example below prints the OpenSSL library version using the `OPENSSL_VERSION_TEXT` symbol present in the `opensslv.h` header file: ``` #include #include int main() { printf("%s\n", OPENSSL_VERSION_TEXT); return 0; } ``` Make sure to load one of the GCC modules on HPC when building the above C code first. If the default GCC compiler is called, it will build with the locally installed GCC version on the current node the code is built on. The following example will load the gcc/11.3.0 module version and then invoke gcc to build the basic C code from the above text. This is mentioned in the **RPATH **section further down in this guide. ``` module load gcc/11.3.0 gcc ver.c ``` The system version of OpenSSL is 1.0.1e: ``` make ver # cc ver.c -o ver ./ver # OpenSSL 1.0.1e 11 Feb 2013 ``` To use a more recent version of OpenSSL 1.0.2o (i.e. one that was installed using [spack](https://kb.uconn.edu/space/SH/26074480709/Spack+Package+Manager)), we must tell the compiler where to look by setting the `CPATH` variable: ``` CPATH= make -B ver # cc ver.c -o ver ./ver # OpenSSL 1.0.2o 27 Mar 2018 ``` Using `module` automatically sets the `CPATH` variable for us so that we don't have to worry about it: ``` module purge module load slurm zlib/1.2.12 openssl/1.0.2o ``` ``` make -B ver # cc ver.c -o ver ./ver # OpenSSL 1.0.2o 27 Mar 2018 ``` We can use the "show" command to see how `CPATH` is being set: ``` module show openssl/1.0.2o ``` ``` ------------------------------------------------------------------- /apps2/Modules/3.2.6/modulefiles/openssl/1.0.2o: setenv MOD_APP openssl setenv MOD_VER 1.0.2o module load pre-module prereq zlib/1.2.11 conflict openssl prepend-path PATH /apps2/openssl/1.0.2o/bin prepend-path LD_LIBRARY_PATH /apps2/openssl/1.0.2o/lib prepend-path LIBRARY_PATH /apps2/openssl/1.0.2o/lib prepend-path LD_RUN_PATH /apps2/openssl/1.0.2o/lib prepend-path INCLUDE /apps2/openssl/1.0.2o/include prepend-path CPATH /apps2/openssl/1.0.2o/include prepend-path MANPATH /apps2/openssl/1.0.2o/ssl/man prepend-path PKG_CONFIG_PATH /apps2/openssl/1.0.2o/lib/pkgconfig module load post-module ------------------------------------------------------------------- ``` ## LD_LIBRARY_PATH The concept of `LD_LIBRARY_PATH` is similar to `PATH`, except instead of executable programs, it controls where the shell searches for *libraries* for programs to use. In many cases, we use `LD_LIBRARY_PATH` together with `RPATH`; we will learn more about `RPATH` in the next section. ## RPATH Using `RPATH` tells the compiler to modify the final library or executable program that it creates with a library search path to use at runtime. In other words, it encodes `LD_LIBRARY_PATH` directly into the executable itself so that `LD_LIBRARY_PATH` is no longer needed. You must use `RPATH`s whenever you're trying to take precedence over a system library. Consider how using the `libcurl.so` library located at `/usr/lib/libcurl.so` always takes precedence over `/apps2/libcurl/7.60.0/lib`: ``` #include #include int main() { printf("%s\n", curl_version()); return 0; } ``` ``` LDFLAGS="-l curl" make -B ver # cc -l curl ver.c -o ver ./ver # libcurl/7.19.7 NSS/3.27.1 zlib/1.2.3 libidn/1.18 libssh2/1.4.2 ldd ver | grep curl # libcurl.so.4 => /usr/lib64/libcurl.so.4 (0x0000003995c00000) ``` We can force the program to use the newer `/apps2` version of libcurl using the `-Wl,-rpath, ...` ``` LDFLAGS="-l curl -Wl,-rpath,/apps2/libcurl/7.60.0/lib" make -B ver # cc -l curl -Wl,-rpath,/apps2/libcurl/7.60.0/lib ver.c -o ver ./ver # libcurl/7.60.0 OpenSSL/1.0.2o zlib/1.2.11 ldd ver | grep curl # libcurl.so.4 => /apps2/libcurl/7.60.0/lib/libcurl.so.4 (0x00002b1aeba4a000) readelf -d ver | head -6 # Dynamic section at offset 0x7d0 contains 22 entries: # Tag Type Name/Value # 0x0000000000000001 (NEEDED) Shared library: [libcurl.so.4] # 0x0000000000000001 (NEEDED) Shared library: [libc.so.6] # 0x000000000000000f (RPATH) Library rpath: [/apps2/libcurl/7.60.0/lib] ``` Another example of needing to force precedence over system libraries is when using modern compilers: ``` #include // Requires >= C++17. #include int main() { std::any value = 1; std::cout << value.type().name() << ": " << std::any_cast(value) << std::endl; value = 'A'; std::cout << value.type().name() << ": " << std::any_cast(value) << std::endl; return 0; } ``` Here the program compiles fine, but crashes at runtime because it's trying to use the older system version: ``` module purge module load gcc/11.3.0 ``` ``` CXXFLAGS=-std=c++17 make test ./test # ./test: /usr/lib64/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by ./test) ldd test # ./test: /usr/lib64/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by ./test) # linux-vdso.so.1 => (0x00007ffe97f9f000) # libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00000037ade00000) # libm.so.6 => /lib64/libm.so.6 (0x00000037a9200000) # libgcc_s.so.1 => /apps2/gcc/9.2.0/lib64/libgcc_s.so.1 (0x00002b235a6c3000) # libc.so.6 => /lib64/libc.so.6 (0x00000037a8600000) # /lib64/ld-linux-x86-64.so.2 (0x00000037a8200000) ``` Use the `RPATH` setting so that the executable itself knows to use the location of the newer C++ standard library: ``` CXXFLAGS=-std=c++17 LDFLAGS=-Wl,-rpath,/apps2/gcc/9.2.0/lib64 make test ./test # 1 # A ldd test # ./test: /usr/lib64/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by ./test) # linux-vdso.so.1 => (0x00007ffd535c4000) # libstdc++.so.6 => /apps2/gcc/9.2.0/lib64/libstdc++.so.6 (0x00002b3fe4b99000) # libm.so.6 => /lib64/libm.so.6 (0x00000037a9200000) # libgcc_s.so.1 => /apps2/gcc/9.2.0/lib64/libgcc_s.so.1 (0x00002b3fe4fa2000) # libc.so.6 => /lib64/libc.so.6 (0x00000037a8600000) # /lib64/ld-linux-x86-64.so.2 (0x00000037a8200000) ``` # Compiling a large program ## Don't I need sudo permissions? No, using the administrative program `sudo` is commonly suggested to install software using commands like `sudo make install`, but `sudo` is only needed because the default install locations like `/usr/local` are protected. As long as you choose a different install location where you have write access, such as a location in your home directory, you don't need any special permissions or `sudo`. The setting to change the install location is typically called a "prefix". We will explain how to set the prefix location. ## Tarballs of source code Often source code for GNU/Linux will be provided in a "tarball" file. You can recognize a tarball file by it's file extension; some examples are: ``` .tar.gz .tgz # These 2 are equivalent file extensions .tar.bz2 .tbz2 .tar.xz ``` You can unpack these files in a directory using `tar -xf ${NAME_OF_TARBALL}.tar.gz`. At other times, instead of a tarball, one may need to grab a copy from a version control system like git. In the case of git, one might create the source directory by cloning the source URL. Now that we have our source files in a directory, the next thing we need to do is consider the compiler to use. ## Which compiler should I use? Usually the developer will suggest which compiler(s) are supported in the documentation. If not, using gcc is safest. Our RedHat 8.7 compute nodes use gcc 8.5.0 by default. If your compilation complains about needing a newer version you can load any of the gcc modules. Some of our users report better performance with Intel MPI. One can access them from the intelics or intel oneapi modules, where the version reflects the year. The Intel compiler tends to be more popular among Fortran programmers because it is quicker to implement the latest Fortran standards. AMD provides their own C/C++ and Fortran compilers grouped together in a collection called aocc that are optimized to run on the AMD compute nodes. CPU's are organized by CPU family through AMD.  Our AMD EPYC 7H12 64-Core Processors are part of the 7002 series (or the 7xx2 for compile options PDF)  Our AMD EPYC 7763 CPU 64 Core processor compute nodes belong to the AMD 7003 CPU family (or the 7xx3 in AMD's compiler pdf documentation).  Attached are the PDF documents showing the compile flags needed for our current AMD EPYC 7H12 (7xx2 series) and AMD EPYC 7763 (7xx3 series) 64 Core processor compute nodes on HPC.  Finally, if you are compiling for GPU, you would need to load the nvidia compiler available in the cuda module. ``` # List all compilers: GNU, Intel, Portland Group, and Nvidia $for compiler in gcc intel cuda ; do module avail -l $compiler ; done - Package/Alias -----------------------.- Versions --------.- Last mod. ------- /cm/local/modulefiles: gcc/11.2.0 2022/06/23 16:48:28 /cm/shared/modulefiles: gcc/4.8.2 2023/01/26 08:53:00 gcc/5.4.0 2023/01/26 08:53:25 gcc/5.4.0-alt 2023/01/26 09:55:24 gcc/8.4.0 2023/01/26 08:54:07 gcc/9.5.0 2023/01/26 08:54:24 gcc/10.1.0 2023/01/26 08:51:28 gcc/11.2.0 2023/01/26 08:51:50 gcc/11.3.0 2023/01/26 08:52:12 gcc/12.2.0 2023/01/26 08:52:41 gcc/13.2.0 2023/12/04 13:45:54 - Package/Alias -----------------------.- Versions --------.- Last mod. ------- /cm/shared/modulefiles: intel-tbb-oss/ia32/2020.3 2023/01/26 10:04:38 intel-tbb-oss/ia32/2021.4.0 2023/01/26 10:05:13 intel-tbb-oss/intel64/2020.3 2023/01/26 10:06:09 intel-tbb-oss/intel64/2021.4.0 2023/01/26 10:06:42 intel/oneapi/2022.3 2023/07/31 15:29:03 intelics/2016.3-full 2023/01/26 10:00:45 intelics/2017 2023/01/26 10:01:07 - Package/Alias -----------------------.- Versions --------.- Last mod. ------- /cm/shared/modulefiles: cuda/11.4 2023/01/26 08:35:40 cuda/11.6 2023/07/11 10:22:02 cuda10.2/blas/10.2.89 2023/01/26 08:38:02 cuda10.2/fft/10.2.89 2023/01/26 08:38:28 cuda10.2/toolkit/10.2.89 2023/01/26 08:38:52 cuda11.2/blas/11.2.2 2023/01/26 08:40:19 cuda11.2/fft/11.2.2 2023/01/26 08:40:58 cuda11.2/toolkit/11.2.2 2023/01/26 08:41:25 cuda11.6/blas/11.6.2 2023/01/26 08:42:04 cuda11.6/fft/11.6.2 2023/01/26 08:42:35 cuda11.6/nsight/11.6.2 2023/01/23 09:12:24 cuda11.6/profiler/11.6.2 2023/01/23 09:12:24 cuda11.6/toolkit/11.6.2 2023/01/23 09:12:24 ``` The above module list will change depending on software and hardware changes within HPC. Before compiling programs, you may want to remove any other modules you have loaded so that they do not interfere with your compilation. ``` # Unload all modules module purge ``` # General workflow Follow the documentation in your software source directory. Typically the workflow is: ``` ./configure --prefix=${HOME}/apps make -j $(nproc) make install ``` Good practice is to create a shell script which runs these commands for you, so that a few months from now you remember exactly how you compiled your software and make your work more reproducible for yourself, your lab mates and collaborators. Also, you may want to write the line `set -e` toward the top of your shell script so that the script stops when it encounters errors. Nearly all software that needs compilation will at least ship with a makefile. If you are new with using Makefiles, we recommend the free [Software Carpentry automation and make lesson](https://swcarpentry.github.io/make-novice/). The Software Carpentry course will provide a short introduction to makefiles. The `make` command will search for a file named `Makefile`. If one does not exist you would need to specify a file name using e.g. `make -f ${NAME_OF_MAKEFILE}.mk`. If your software is complex enough to also require other dependencies, it would likely come with a `configure` shell script. It is a good idea to run `./configure --help` to see how to change variables and set PATHs to libraries. You almost always would need to set the `--prefix` option to set the final installation path as you do not have access to the system protected directories of `/bin /lib64 /usr/local` etc. If you obtained your code from version control instead of a traditional release and do not see a `configure` script and your documentation tells you that you need one, you may likely need to also generate the `configure` shell script from `configure.ac` using a program named similar to `bootstrap.sh`, `autogen.sh` or at worst you would need to run `autoreconf` directly. Good resources for understanding how the autotools programs work that process `configure.ac` and `Makefile.am` files are the [basics of autotools](https://devmanual.gentoo.org/general-concepts/autotools/index.html) in the Gentoo Linux development manual, and the Diego Pattenò's comprehensive online book [autotools.io](https://autotools.io/) After compiling the software a module can be created via the following documentation: [https://uconn.atlassian.net/wiki/spaces/SH/pages/26062356547](https://uconn.atlassian.net/wiki/spaces/SH/pages/26062356547) # Compiling with MPI The Message Passing Interface (MPI) is a standardized and portable message-passing standard designed to function on parallel computing architectures. MPI has changed on HPC with the new additions of the AMD EPYC nodes along with Red Hat 8.X The old Infiniband network interfaces and message traffic has changed when calling the MPI standard. OpenMPI versions 5.X+ will no longer support the openib framework and the build for MPI/openmpi has changed on HPC as a result. ## Rebuilding software that was built/using older MPI versions If older software that was previously built using the older MPI versions prior to February 2022 on the original HPC cluster hardware, would need to rebuilt using the newest UCX compatible openmpi module versions available on the current HPC configuration. The versions are the following: openmpi/4.1.4 which calls the GCC compiler built using the UCX framework openmpi/4.1.4-ics which calls and invokes the Intel compiler that needed to be rebuilt with avx2 support and UCX support # Older Libraries Older libraries on newer HPC hardware will no longer be supported through global installs or loadable modules. It is recommend to see if software can be rebuilt with the current libraries available on HPC. If certain software are unable to run with the current libraries available on HPC, there are other options that can help run code on the current HPC hardware. ## SPACK library installs The SPACK package manager helps install older libraries within a package environment locally available under a user’s environment. There is a Knowledge Base article set up explaining the benefits and process of installing and setting up the SPACK package manager on a local user’s HPC account located here: [https://uconn.atlassian.net/wiki/spaces/SH/pages/26074480709](https://uconn.atlassian.net/wiki/spaces/SH/pages/26074480709) ## Apptainer Container option for older libraries Apptainer enables and allows a container option to be run on HPC. The container Image file can be generated with the needed libraries and software that has problem running on the current HPC hardware. We are currently updating a knowledge base article showing the steps for the Apptainer Container solution and will link to the page in this section once available. To be continued # Architecture Specific building options ## Intel oneAPI compiler The intel/oneapi/2022.3 compiler on HPC needed specific hardware support to be able to run on the new AMD EPYC HPC node hardware. - If building software that needs the Intel compiler, the software would need to build with the -march=core-avx2 flag (if supported). # Examples ## FFMPEG [https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftrac.ffmpeg.org%2Fwiki%2FCompilationGuide%2FCentos&data=05%7C01%7Ctechsupport%40uconn.edu%7C922c44881b11466477e208db638f6550%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C638213239769874099%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=G7VVrv1wW2rlzfSjwuf8NG12AfeKvUPh%2Bw1y6Yk0GbU%3D&reserved=0](https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftrac.ffmpeg.org%2Fwiki%2FCompilationGuide%2FCentos&data=05%7C01%7Ctechsupport%40uconn.edu%7C922c44881b11466477e208db638f6550%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C638213239769874099%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=G7VVrv1wW2rlzfSjwuf8NG12AfeKvUPh%2Bw1y6Yk0GbU%3D&reserved=0) Reading the install dependency guide for FFMPEG, there are a lot of options that can be enabled or disabled depending on the build and options looking to run with FFMPEG. ## CAMx 7.20 Reading the comments in the CAMx makefile, we can compile CAMx with the GCC compiler or the Intel compiler. We will use the Intel compiler (to invoke ifort) in this example. (CAMx has a block data issue with newest versions of GCC and building with gfortran will not work): ``` # Create a directory for our CAMx project. mkdir -p ~/src/camx-7.20 cd ~/src/camx-7.20 # Download the CAMx 7.20 source tarball into the directory. camx-7.20]$ wget downloadHTTPspathforCamx7.20 # Unpack the source file. tar -xvpf CAMx_v7.20.src.220430.tgz #cd into the new folder after the tarball unpacks camx-7.20]$ cd src.v7.20/ # Load the Intel compiler. # # List available Intel compiler versions: module avail intel # Get rid of any other modules that might interfere with our compilation. module purge # The latest version at this time is 2022.3. module load slurm intel/oneapi/2022.3 openmpi/4.1.4-ics hdf5/1.13.2-ics netcdf/4.9.0-ics netcdf-fortran/4.6.0-ics # View CAMx Makefile to see options on how to build CAMx. less Makefile # The makefile to compile CAMx supports different types of CPUs, compilers, and library support. # Let's see what the makefile will do before running it. make -n -f Makefile #Determine which options to build CAMx with. If MPI and NETCDF options are needed, the make command options can change. #If MPI and NETCDF are needed, the Makefile directory paths would need to be updated to point to the correct locations where MPI, and NETCDF-FORTRAN are located from the loadable modules on HPC or pre-built locally. #If using a global HPC module, you can view the install path with the following command: module show openmpi/4.1.4-ics # Grab the default path that does not point to /bin or /lib. Once the paths have been copied, the Makefile can be edited: vi Makefile #Paths: MPI_INST = /path/to/openmpi/from/module NCF_INST = /path/to/netcdf/fortran/from/module #Updated Paths: MPI_INST = /gpfs/sharedfs1/admin/hpc2.0/apps/openmpi/4.1.4-ics NCF_INST = /gpfs/sharedfs1/admin/hpc2.0/apps/netcdf-fortran/4.6.0-ics #After the Makefile is saved with the paths to point to MPI and NETCDF, CAMx can now compile. # Compile by running the following make command without the `-n` flag. make COMPILER=ifortomp MPI=openmpi NCF=NCF4_C #Note, if the Makefile complains about -openmp has been deprecated, the string needs to be replaced with -qopenmp along with -extend-source #These edits can happen if the Intel compiler gets updated and older build functions become deprecated. vi Makefile :%s/-openmp/-qopenmp/g :%s/-extend_source/-extend-source/g :wq! #After saving the above change, re-run the make command from the previous step. #Wait for CAMx to build with the settings, once built, CAMx will generate the necessary install executable. #Should be named like the following: CAMx.v7.20.openMPI.NCF4.ifortomp ``` Create a module file for CAMx that so that we can conveniently load CAMx and it's dependencies. The name that you choose for your module file is important as that is what module uses to reference it. We will make our name different by adding the "-mine" suffix to help separate it from the system installed CAMx. ``` mkdir -p ~/mod/camx cd ~/mod/camx vi 7.20-mine ``` ``` #%Module1.0 # Throw an error if any of these modules are loaded. conflict camx # Load the particular Intel compiler module we used for building CAMx, etc. module load intel/oneapi/2022.2 module load openmpi/4.1.4-ics module load hdf5/1.13.2-ics module load netcdf/4.9.0-ics module load netcdf-fortran/4.6.0-ics # Modify the PATH to use our compiled CAMx. Do not use a trailing slash. prepend-path PATH ~/src/camx-7.20/src.v7.20 ``` If you are interested, in learning about module files you can read `man modulefile` Finally, make sure that `module` knows to look in your `~/mod` directory for your module files by setting the `MODULEPATH` environmental variable: ``` vi ~/.bashrc # Add the lines below. ``` ``` 1 # My modules 2 source /etc/profile.d/modules.sh 3 MODULEPATH=${HOME}/mod:${MODULEPATH} ``` Reload your ~/.bashrc file in your current shell: ``` 1 source ~/.bashrc 2 # Finally Now we can load and run our CAMx module 3 module load camx/7.2.0-mine 4 # The executable will be available once the module is loaded. 5 CAMx.v7.20.openMPI.NCF4.ifortomp ``` We can set up a symbolic link to point to the CAMx executable to shorten the command to run camx: ``` 1 cd ~/src/camx-7.20 2 # Create the symbolic link to the CAMx exectuable 3 ln -s CAMx.v7.20.openMPI.NCF4.ifortomp camx 4 # The symlink will be available once the module is loaded and 5 # The symlink will make it easier to call camx without specifying the full name of the executable. 6 camx ``` ## VASP 5.3.3 Reading the comments in the VASP makefiles, we can compile VASP with the PGI compiler or the Intel compiler. As the makefile comments mention there is no performance change with the PGI compiler versions, we will use the Intel compiler in this example: ``` # Create a directory for our VASP project. mkdir ~/src/vasp-5.3.3 cd ~/src/vasp-5.3.3 # Copy the source code from the admin directory. cp -arv /shared/admin/sw-src/rhel6/vasp/vasp.5.3.3.tar.gz . cp -arv /shared/admin/sw-src/vasp/vasp.5.lib.tar.gz . # Unpack the sources. tar -xvpf vasp.5.3.3.tar.gz tar -xvpf vasp.5.lib.tar.gz # Load the Intel compiler. # # List available Intel compiler versions: module avail intelics # Get rid of any other modules that might interfere with our compilation. module purge # The latest version at this time is 2017. module load intelics/2017 # Compile the VASP 5 library. cd vasp.5.lib/ # There are several makefiles to compile VASP for different types of CPUs and compilers. # These are the linux compatible makefiles: ls -1 makefile.linux* # The best supported for our cluster is makefile.linux_ifc_P4 # Let's see what the makefile will do before running it. make -n -f makefile.linux_ifc_P4 # Now compile by running make without the `-n` flag. # Also overwrite Intel's old Fortran compiler name from `ifc` to be `ifort` by passing as a variable to `make`. make -f makefile.linux_ifc_P4 FC=ifort # Go back to src directory cd .. # Compile the VASP program. cd vasp.5.3/ # Compile by running make without the `-n` flag. # With VASP we cannot use `-j` for simultaneous compilation as it is unreliable. # Overwrite BLAS variable as Intel now calls the "guide" library as "iomp5" per https://software.intel.com/en-us/forums/intel-c-compiler/topic/284445 make -f makefile.linux_ifc_P4 BLAS=-liomp5\ -mkl ``` Create a module file for VASP that so that we can conveniently load VASP and it's dependencies. The name that you choose for your module file is important as that is what module uses to reference it. We will make our name different by adding the "-mine" suffix to help separate it from the system installed vasp. ``` mkdir -p ~/mod/vasp cd ~/mod/vasp nano 5.3.3-mine ``` ``` #%Module1.0 # Throw an error if any of these modules are loaded. conflict vasp conflict intelics # Load the particular Intel compiler module we used for the Math Kernel library, etc. module load intelics/2017 # Modify the PATH to use our compiled VASP. Do not use a trailing slash. prepend-path PATH /gpfs/homefs1//src/vasp-5.3.3/vasp.5.3 ``` If you are interested, in learning about module files you can read `man modulefile` Finally, make sure that `module` knows to look in your `~/mod` directory for your module files by setting the `MODULEPATH` environmental variable: ``` nano ~/.bashrc # Add the lines below. ``` ``` 1 # My modules 2 source /etc/profile.d/modules.sh 3 MODULEPATH=${HOME}/mod:${MODULEPATH} ``` Reload your ~/.bashrc file in your current shell: ``` 1 source ~/.bashrc 2 # Finally Now we can load and run our VASP module 3 module load vasp/5.3.3-mine 4 which vasp 5 vasp -h ``` ## Local library install and loadable module creation. This section of the knowledge base article will provide a guide and show a basic setup example to install a library package locally under the /home directory. ### libarchive Reading the comments in the CAMx makefile, we can compile CAMx with the GCC compiler or the Intel compiler. We will use the Intel compiler (to invoke ifort) in this example. (CAMx has a block data issue with newest versions of GCC and building with gfortran will not work): ``` # Create a directory for our Libarchive project. mkdir -p ~/src/libarchive-3.6.2 cd ~/src/libarchive-3.6.2 # Download the libarchive source tarball into the directory. libarchive-3.6.2]$ wget downloadHTTPspathforlibArchive3.6.2 # Unpack the source file. tar -xvpf libarchive-3.6.2.tar.gz #cd into the new folder after the tarball unpacks libarchive-3.6.2]$ cd libarchive-3.6.2/ # Load a compiler if the library needs to install/run with a compiler (gcc or intel/oneapi). # In this case, libarchive does not need a compiler, so we will ignore the compiler step. # libarchive will generate a ./configure script under the above directory to configure the library for install. # Run the following configure command: ./configure --prefix=/path/where/you/would/like/libarchive/installfiles/to/go/ # We will configure libarchive to install under the main libarchive-3.6.2/ folder. ./configure --prefix=/home/netidhere/src/libarchive-3.6.2/ # The makefile to compile libarchive will be generated after the configure script runs. # Let's see what the makefile will do before running it. make -n -f Makefile # Compile by running the following make command without the `-n` flag, the -j 4 will run make with 4 cores. make -j 4 # After the make file finishes building libarchive, the final install would need to be performed. make install #Wait for libarchive to finish installing, once insalled, libarchive will generate the necessary install executables, libraries, and header files. #The following folders should have been generated under the install path specified in the ./configure command: bin/ include/ lib/ share/ ``` Create a module file for libarchive that so that we can conveniently load libarchive. The name that you choose for your module file is important as that is what module uses to reference it. We will make our name different by adding the "-mine" suffix to help separate it from the system installed libarchive. ``` mkdir -p ~/mod/libarchive cd ~/mod/libarchive vi 3.6.2-mine ``` ``` #%Module1.0 # Throw an error if any of these modules are loaded. conflict libarchive # Modify the PATH(s) to use our compiled libarchive install. Do not use a trailing slash. prepend-path PATH ~/src/libarchive-3.6.2/bin prepend-path LIBRARY_PATH /home/netidhere/libarchive-3.6.2/lib prepend-path LD_LIBRARY_PATH /home/netidhere/libarchive-3.6.2/lib prepend-path INCLUDE /home/netidhere/libarchive-3.6.2/include prepend-path CPATH /home/netidhere/libarchive-3.6.2/include prepend-path PKG_CONFIG_PATH /home/netidhere/libarchive-3.6.2/lib/pkgconfig prepend-path MANPATH /home/netidhere/libarchive-3.6.2/share/man ``` If you are interested, in learning about module files you can read `man modulefile` Finally, make sure that `module` knows to look in your `~/mod` directory for your module files by setting the `MODULEPATH` environmental variable: ``` vi ~/.bashrc # Add the lines below. ``` ``` # My modules source /etc/profile.d/modules.sh MODULEPATH=${HOME}/mod:${MODULEPATH} ``` Reload your ~/.bashrc file in your current shell: ``` source ~/.bashrc # Finally Now we can load and run our CAMx module module load libarchive/3.6.2-mine # The executables will be available once the module is loaded. ```