6. Using software in a hpc environment

http://hpctoolkit.org/software-instructions.html https://apptainer.org/docs/user/main/index.html

• https://ashki23.github.io/spack.html

• https://developer.amd.com/spack/build-customization/

6.1. Spack: automatic load for programs and containers

Spack was designed by llnl and is targeted to simplify the installation and managment of HPC programs with many possible versions and dependencies. Check the docs at https://spack.readthedocs.io/en/latest/tutorial.html or https://bluewaters.ncsa.illinois.edu/webinars/software-ecosystems/spack . For now let’s just install it, configure it, and install some simple tools.

6.1.1. Installation

Following the official docs, just clone the repository

git clone https://github.com/spack/spack.git # clones the repo
cd spack
#git checkout releases/v0.17 # Checkout the latest stable release
source share/spack/setup-env.sh # Setup the environment , this command should go in ~/.bashrc

Now you can check what can be installed

spack list

To be able to use spack easily in the future, it is recommended to add the source command to your ~/.bashrc, so just add the following at the end of the file

source $HOME/PATHTOREPO/share/spack/setup-env.sh

and then close and open the terminal.

In our computer room there has been a problem with spack and openssl. It is better to instruct spack to use the local openssl version instead of building one. To do so, add the following to your spack package config file, $SPACK_REPO/etc/spack/packages.yaml:

packages:
    openssl:
        externals:
        - spec: openssl@1.1.1m
          prefix: /usr
        buildable: False

You can check the correct version with \(openssl version\). Furthermore, to be able to run your installed programs on several computers with different processors, use the flag target=x86_64 .

6.1.2. Installing some tools with spack

Now let’s just install the gsl scientific library and some eigen alternative versions:

spack info gsl # Get some information about versions, deps, etc
spack install gsl@2.5
spack install gsl@2.4
module avail

To check the installed software, you can use the module command as (installed when you used bootstrap)

module avail

Now you will see that you have two versions of the gsl. If you want to use one of them, you will load it with spack. The check the change in environment, first check the PATH, then load, then compare

echo $PATH
echo $C_INCLUDE_PATH
echo $LD_LIBRARY_PATH

Now load the gsl version 2.5,

spack load gsl@2.5

and check the new paths

echo $PATH
echo $C_INCLUDE_PATH
echo $LD_LIBRARY_PATH

If you unload the gsl 2.5, everything goes back to normal,

spack unload gsl@2.5
echo $PATH
echo $C_INCLUDE_PATH
echo $LD_LIBRARY_PATH

To learn more about spack, check the official docs and tutorials. In the following we will use it to play with several packages in parallel programming. Is voro++ available? what about eigen?

6.2. How to install programs from source

In this workshop we will learn how to install a program or library from source to the home directory of the user. This is useful when you need a programa but it is not available in the system you are working on. Also, It could be that you actually need a newer version than the installed one. We will use the following programs, some of them already available at the computer room:

Name	installed version	latest version
fftw	3.3.4	3.3.8
Eigen C++	3.2.7	3.3.7
voro++	Not installed	0.4.6
g++ 9.x	Not installed	9.2

We will learn how to do it by compiling a package directly, something very useful to know, and also using a new tool aimed for supercomputers, called spack , that simplifies the installation of software and also allows to have many versions of the same library, something not easy done manually.

6.2.1. Preliminary concepts

It is important for you to know a little how your operating system and the shell look for commands. In general, the PATH variable stores the directories where the shell interpreter will look for a given command. To check its contents, you can run the following,

If, for instance, you want to add another directory to the PATH, like the directory $HOME/local/bin, you can run the following

export PATH=$PATH:$HOME/bin

This appends the special directory to the old content of the PATH.

When you want to compile a given program that uses some other libraries, you must specify any extra folder to look for include files, done with the -I flag, and for libraries, done with the -L and -l flags. For instance, let’s assume that you installed some programs in your HOME/local, the include files inside $HOME/local/include, and the libraries inside $HOME/local/lib. If you want to tell the compiler to use those, you must compile as

Finally, whenever you are installing a program from source you must be aware that this reduces to basically the following steps:

1. Download and untar the file. Enter the unpacked directory.

2. Read the README/INSTALL files for any important info.

3. If the program uses cmake, create a build dir and then use cmake to generate the Makefiles:

   mkdir build
   cd build
   cmake ../ -DCMAKE_INSTALL_PREFIX=$HOME/local
   

   On the other hand, if the program uses configure, then configure the system to install on the required path

   ./configure --prefix=$HOME/local
   

4. Compile and install the program, maybe using many threads

   make -j 4 # uses for threads to compile
   make install
   

   Done. The program is installed.

   Finally, when compiling, do not forget to use the flags -L and -I appropriately.

Setting all the flags and making sure to use the right version is sometimes difficult, so tools like spack aim to manage and simplify this.

6.2.2. Checking the version for already installed programs

If you are used to apt-get or something related, you can use the package manager to check. But, in general, you can check the versions by looking at the appropriate places on your system. Typically, if you are looking for a library, they are installed under /usr/lib or /usr/local/lib, while include files are installed under /usr/include or /usr/local/include . For instance, if you are looking for library foo, then you should look for the file libfoo.a or libfoo.so . One useful utility for this is the command locate or find .

locate libfoo
find /usr/lib -iname "*libfoo*"

Execute these commands to check the actual versions for fftw, eigen, and git. What versions do you have? If you a re looking for a program, or an specific version of program, you must check if the program exists by executing it. For command line programs you usually can check the version by using the following

where programname is the name of the command.

6.2.3. Preparing the local places to install the utilities

In this case we will install everything under the $HOME/local subdirectory inside your home, so please create it. Remember that the symbol $HOME means your home directory. The utilies will then create the appropriate folders there. NOTE: Better use the $HOME var instead of ~, which is another way to state the name of your home.

6.2.4. Typical installation algorithm

1. Download the source from the project page. This normally implies downloading a tarball (file ending with .tar.gz or .tar.bz2) .

2. Un-compress the downloaded file. For a tarball, the command will be

   tar xf filename.tar.gz
   

3. Enter to the newly uncompressed folder (almost always usually cd   filename).

4. READ the README and/or the INSTALL file to check for important info regarding the program. SOmetimes these files tell you that the installation is special and that you are required to follow some special steps (that will happen with voro++ )

5. CONFIGURATION: You have two options, each one independent on the other:

   1. If the program has a configure script, then just run

      ./configure --help
      

   to check all the available options. Since we want to install on the $HOME/local directory, then we need to run

   ./configure --prefix=$HOME/local
   

   If you don’t specify the prefix, then the program will be installed on the /usr/bin or /usr/local/bin directories, whatever is the default. If these commands ends successfully, it will print some info to the screen and will tell you to go to the next step. Otherwise you will need to read the log and fix the errors (like installing a dependency).

   1. If the program uses cmake, a makefile generator and

   configurator, then you need to do the following:

   mkdir build # creates a build directory to put there the temporary built files
   cd build 
   cmake ../ -DCMAKE_INSTALL_PREFIX:PATH=$HOME/local # configure the building process for the source code located on the parent directory
   

6. COMPILATION: Now that you have configured your installation, you need to compile by using the GNU make utility (Note: All this build utilities come from the gnu organization and are free software as in freedom). If you have several cores, you can use them in parallel, assuming the that the Makefile and your make versions supports it:

   make -j 3 # for three cores, but, if you are unsure, just use one core.
   

   Any errors in this stage should be fixed before going to the next one.

7. INSTALLATION After successful compilation, you can install by using

   make install
   

   This will install the program (libraries, binaries, include files, manual files, etc) onto the prefix directory. If you want to instll system-wide (you did not set the prefix), then you need to use sudo make install . In this case you don’t need sudo since you are installing on your own home.

8. TESTING In this case use a program to test your installation. When you compile your program and you want to use the version that you installed, you need to tell the compiler where to find the libraries/includes, so you need something like

   g++ -L $HOME/local/lib -I $HOME/local/include  programname.cpp -llibname
   

   • -L $HOME/local/lib tells the compiler to look for libraries on

   the $HOME/local/lib directory.

   • -I $HOME/local/include tells the compiler to look for include

   files on the $HOME/local/include directory.

   • -llibname tells the compiler to link with the given

   library. Sometimes is not needed. Sometimes is crucial. Be careful, if your library is called libfftw, you need to write -lfftw, not -llibfftw.

6.2.5. Workshop

For each of the proposed utilities written at the beginning, follow the next procedure:

• Check the installed version number and compare with the latest one.

• Install the latest version on your home directory by following the procedure stated above.

• Run each of the following example program but make sure you a re using you installed version. Show to the instructor the compilation line.

Important NOTE: for g++, use the prefix -9 in the configure line to put that as suffix to the commands and avoid collisions with the compiler already installed in the system. This can be done by adding the flag --program-suffix=-9 to the configure command.

6.2.6. Test Programs

6.2.6.1. fftw

This is a c code. Save it as testfftw.c and compile with gcc instead of g++ .

// From : https://github.com/undees/fftw-example
// This ia a c code (save it as testfftw.c)
/* Start reading here */

#include <fftw3.h>

#define NUM_POINTS 128


/* Never mind this bit */

#include <stdio.h>
#include <math.h>

#define REAL 0
#define IMAG 1

void acquire_from_somewhere(fftw_complex* signal) {
  /* Generate two sine waves of different frequencies and
   * amplitudes.
   */

  int i;
  for (i = 0; i < NUM_POINTS; ++i) {
    double theta = (double)i / (double)NUM_POINTS * M_PI;

    signal[i][REAL] = 1.0 * cos(10.0 * theta) +
      0.5 * cos(25.0 * theta);

    signal[i][IMAG] = 1.0 * sin(10.0 * theta) +
      0.5 * sin(25.0 * theta);
  }
}

void do_something_with(fftw_complex* result) {
  int i;
  for (i = 0; i < NUM_POINTS; ++i) {
    double mag = sqrt(result[i][REAL] * result[i][REAL] +
                      result[i][IMAG] * result[i][IMAG]);

    printf("%g\n", mag);
  }
}


/* Resume reading here */

int main() {
  fftw_complex signal[NUM_POINTS];
  fftw_complex result[NUM_POINTS];

  fftw_plan plan = fftw_plan_dft_1d(NUM_POINTS,
                                    signal,
                                    result,
                                    FFTW_FORWARD,
                                    FFTW_ESTIMATE);

  acquire_from_somewhere(signal);
  fftw_execute(plan);
  do_something_with(result);

  fftw_destroy_plan(plan);

  return 0;
}

6.2.6.2. Eigen C++

These are C++ codes. Save them, compile, run and explain what they do.

    #include <iostream>
    #include <Eigen/Dense>
    #include <Eigen/Core>
    using Eigen::MatrixXd;
    int main()
    {
      //std::cout << EIGEN_MAYOR_VERSION << std::endl;
      std::cout << EIGEN_MINOR_VERSION << std::endl;
      MatrixXd m(2,2);
      m(0,0) = 3;
      m(1,0) = 2.5;
      m(0,1) = -1;
      m(1,1) = m(1,0) + m(0,1);
      std::cout << m << std::endl;
    }

---

    #include <iostream>
    #include <Eigen/Dense>
    using namespace Eigen;
    int main()
    {
      Matrix2d a;
      a << 1, 2,
        3, 4;
      MatrixXd b(2,2);
      b << 2, 3,
        1, 4;
      std::cout << "a + b =\n" << a + b << std::endl;
      std::cout << "a - b =\n" << a - b << std::endl;
      std::cout << "Doing a += b;" << std::endl;
      a += b;
      std::cout << "Now a =\n" << a << std::endl;
      Vector3d v(1,2,3);
      Vector3d w(1,0,0);
      std::cout << "-v + w - v =\n" << -v + w - v << std::endl;
    }

---

    #include <iostream>
    #include <Eigen/Dense>
    using namespace std;
    using namespace Eigen;
    int main()
    {
      Matrix3f A;
      Vector3f b;
      A << 1,2,3,  4,5,6,  7,8,10;
      b << 3, 3, 4;
      cout << "Here is the matrix A:\n" << A << endl;
      cout << "Here is the vector b:\n" << b << endl;
      Vector3f x = A.colPivHouseholderQr().solve(b);
      cout << "The solution is:\n" << x << endl;
    }

---

    #include <iostream>
    #include <Eigen/Dense>
    using namespace std;
    using namespace Eigen;
    int main()
    {
      Matrix2f A;
      A << 1, 2, 2, 3;
      cout << "Here is the matrix A:\n" << A << endl;
      SelfAdjointEigenSolver<Matrix2f> eigensolver(A);
      if (eigensolver.info() != Success) abort();
      cout << "The eigenvalues of A are:\n" << eigensolver.eigenvalues() << endl;
      cout << "Here's a matrix whose columns are eigenvectors of A \n"
           << "corresponding to these eigenvalues:\n"
           << eigensolver.eigenvectors() << endl;
    }

6.2.7. Voro++

Use the example http://math.lbl.gov/voro++/examples/random_points/

// Voronoi calculation example code
//
// Author   : Chris H. Rycroft (LBL / UC Berkeley)
// Email    : chr@alum.mit.edu
// Date     : August 30th 2011

#include "voro++.hh"
using namespace voro;

// Set up constants for the container geometry
const double x_min=-1,x_max=1;
const double y_min=-1,y_max=1;
const double z_min=-1,z_max=1;
const double cvol=(x_max-x_min)*(y_max-y_min)*(x_max-x_min);

// Set up the number of blocks that the container is divided into
const int n_x=6,n_y=6,n_z=6;

// Set the number of particles that are going to be randomly introduced
const int particles=20;

// This function returns a random double between 0 and 1
double rnd() {return double(rand())/RAND_MAX;}

int main() {
  int i;
  double x,y,z;

  // Create a container with the geometry given above, and make it
  // non-periodic in each of the three coordinates. Allocate space for
  // eight particles within each computational block
  container con(x_min,x_max,y_min,y_max,z_min,z_max,n_x,n_y,n_z,
                false,false,false,8);

  // Randomly add particles into the container
  for(i=0;i<particles;i++) {
    x=x_min+rnd()*(x_max-x_min);
    y=y_min+rnd()*(y_max-y_min);
    z=z_min+rnd()*(z_max-z_min);
    con.put(i,x,y,z);
  }

  // Sum up the volumes, and check that this matches the container volume
  double vvol=con.sum_cell_volumes();
  printf("Container volume : %g\n"
         "Voronoi volume   : %g\n"
         "Difference       : %g\n",cvol,vvol,vvol-cvol);

  // Output the particle positions in gnuplot format
  con.draw_particles("random_points_p.gnu");

  // Output the Voronoi cells in gnuplot format
  con.draw_cells_gnuplot("random_points_v.gnu");
}

On gnuplot do the following:

splot "random_points_p.gnu" u 2:3:4, "random_points_v.gnu" with lines

6.2.8. g++ 9.2

Just run the command and check the version,

Now run any of the special functions examples that required -std=c++17 .