ScaLAPACK and ATLAS on Linux installation tutorial

ScaLAPACK, LAPACK, BLACS and ATLAS on OpenMPI & Linux installation tutorial

I wrote this simple tutorial just to collect all data available on the web. I spent some time to dig this information, so hope you'll enjoy!

1. You should have GCC 3.4.6 with g77 installed. Check that you do not have spare GCC 4 available to compile sources, by trying to run gfortran (should not be able to run) and checking versions by gcc -v, g77 -v. I tried with GCC 4.2.0 but some strange errors about unknown references later appeared, so go ahead just with GCC 3.4.6 and everything should be fine. Obviously it's not the best solution, so I'll try to update this page someday.

2. Be sure that OpenMPI is installed and configured with Fortran, check mpif77 -v

3. Some of sources have hardcoded their folder names, so for simplicity unpack all files to home directory. In our tutorial I assume you are root, so after downloading all necessary files you'll have to have:

/root/ATLAS
/root/BLACS
/root/LAPACK
/root/SCALAPACK

4. Download ATLAS (atlas3.8.0.tar.gz) from math-atlas.sourceforge.net

cd /root
tar -xzvf atlas3.8.0.tar.gz
mkdir /root/ATLAS_BUILD
cd /root/ATLAS_BUILD
/root/ATLAS/configure

If you have cpu frequency throttle issues, you'll have to install cpufreqd and run (see INSTALL.txt)

/usr/bin/cpufreq-set -g performance

Then compile ATLAS:

make
make check
make ptcheck
make time
make install

ATLAS installs by default to /usr/local/atlas, so remember to add /usr/local/atlas/lib path to your LD_LIBRARY_PATH

5. Download BLACS (mpiblacs.tgz and mpiblacs-patch03.tgz) from www.netlib.org/blacs

cd /root
tar -xzvf mpiblacs.tgz
tar -xzvf mpiblacs-patch03.tgz
cd BLACS
cp ./BMAKES/BMake.MPI-LINUX Bmake.inc

Edit Bmake.inc and perform these changes (taken from OpenMPI FAQ):

# Section 1:
# Ensure to use MPI for the communication layer

   COMMLIB = MPI

# The MPIINCdir macro is used to link in mpif.h and
# must contain the location of Open MPI's mpif.h.  
# The MPILIBdir and MPILIB macros are irrelevant 
# and should be left empty.

   MPIdir = /path/to/openmpi-1.2.4
   MPILIBdir =
   MPIINCdir = $(MPIdir)/include
   MPILIB =

# Section 2:
# Set these values:

   SYSINC =
   INTFACE = -Df77IsF2C
   SENDIS =
   BUFF =
   TRANSCOMM = -DUseMpi2
   WHATMPI =
   SYSERRORS =

# Section 3:
# You may need to specify the full path to
# mpif77 / mpicc if they aren't already in
# your path.
 
   F77            = mpif77
   F77LOADFLAGS   = 

   CC             = mpicc
   CCLOADFLAGS    =

Then compile BLACS with MPI:

make mpi

Edit the BLACS/TESTING/Makefile and change that line:

blacstest.o : blacstest.f
	$(F77) $(F77NO_OPTFLAGS) -c $*.f

to:

blacstest.o : blacstest.f
	$(F77) $(F77NO_OPTFLAGS) -fno-globals -fno-f90 -fugly-complex -w -c $*.f

and then compile BLACS tests:

cd /root/BLACS/TESTING
make

run tests:

cd /root/BLACS/TESTING/EXE
mpirun -np 5 xCbtest_MPI-LINUX-0
mpirun -np 5 xFbtest_MPI-LINUX-0

if tests passed, you can install the BLACS library to /usr/local/lib for example by:

chmod 555 /root/BLACS/LIB/*.a 
cp /root/BLACS/LIB/*.a /usr/local/lib

5. Download LAPACK (lapack.tgz) from www.netlib.org/lapack

cd /root
tar -xzvf lapack.tgz

Rename the directory where it was extracted, lapack-3.1.1 to LAPACK

cd /root/LAPACK
cp make.inc.example make.inc

Edit make.inc and perform these changes:

#BLASLIB = ../../blas$(PLAT).a
BLASLIB = /usr/local/atlas/lib/libf77blas.a /usr/local/atlas/lib/libatlas.a

Then compile LAPACK:

make

When it's done, install the LAPACK library:

chmod 555 /root/LAPACK/*.a
cp /root/LAPACK/*.a /usr/local/lib

6. Download ScaLAPACK (scalapack.tgz) from www.netlib.org/scalapack

cd /root
tar -xzvf scalapack.tgz

Rename the directory where it was extracted, scalapack-1.8.0 to SCALAPACK and enter:

cd SCALAPACK
cp SLmake.inc.example SLmake.inc

Edit SLmake.inc and make these changes:

# Make sure you follow the instructions to build BLACS with Open MPI,
# and put its location in the following.

   BLACSdir      = /usr/local/lib

# The MPI section is commented out.  Uncomment it. The wrapper
# compiler will handle SMPLIB, so make it blank. The rest are correct
# as is.

   USEMPI        = -DUsingMpiBlacs
   SMPLIB        = 
   BLACSFINIT    = $(BLACSdir)/blacsF77init_MPI-$(PLAT)-$(BLACSDBGLVL).a
   BLACSCINIT    = $(BLACSdir)/blacsCinit_MPI-$(PLAT)-$(BLACSDBGLVL).a
   BLACSLIB      = $(BLACSdir)/blacs_MPI-$(PLAT)-$(BLACSDBGLVL).a
   TESTINGdir    = $(home)/TESTING

# The PVMBLACS setup needs to be commented out.

   #USEMPI        =
   #SMPLIB        = $(PVM_ROOT)/lib/$(PLAT)/libpvm3.a -lnsl -lsocket
   #BLACSFINIT    =
   #BLACSCINIT    =
   #BLACSLIB      = $(BLACSdir)/blacs_PVM-$(PLAT)-$(BLACSDBGLVL).a
   #TESTINGdir    = $(HOME)/pvm3/bin/$(PLAT)

# You may need to specify the full path to mpif77 / mpicc if they
# aren't already in your path.
 
   F77            = mpif77
   F77LOADFLAGS   = 

   CC             = mpicc
   CCLOADFLAGS    = 

# Make sure that the BLASLIB points to the right place.  

   BLASLIB = /usr/local/atlas/lib/libf77blas.a /usr/local/atlas/lib/libatlas.a
   LAPACKLIB = /usr/local/lib/lapack_LINUX.a

Ok, so now compile ScaLAPACK by:

cd /root/SCALAPACK
make
make exe

and optionally:

make example

Then you can try to run tests:

cd /root/SCALAPACK/TESTING
mpirun -np 5 ./xdlu ./LU.dat
mpirun -np 5 ./xdllt ./LLT.dat

etc... And finally install the ScaLAPACK library by:

chmod 555 /root/SCALAPACK/libscalapack.a
cp /root/SCALAPACK/libscalapack.a /usr/local/lib

In principle you are done with installing all necessary libraries.

7. To compile your own program you have to link it against all these libraries, also there's a trick that BLACS init routines have to be linked twice, as on this example:

mpic++ -c example.cpp
mpic++ -o example example.o                             \
        /usr/local/lib/libscalapack.a                   \
        /usr/local/lib/blacsCinit_MPI-LINUX-0.a         \
        /usr/local/lib/blacs_MPI-LINUX-0.a              \
        /usr/local/lib/blacsCinit_MPI-LINUX-0.a         \
        /usr/local/lib/lapack_LINUX.a                   \
        /usr/local/atlas/lib/libf77blas.a               \
        /usr/local/atlas/lib/libcblas.a                 \
        /usr/local/atlas/lib/libatlas.a                 \
        -lg2c

See how blacsCinit_MPI-LINUX-0.a was linked? The same rule is when you link the Fortran code (blacsF77init_MPI-LINUX-0.a).
As for programming in C++ you have to remember that we link against ANSI C/Fortran code, so remember to add extern "C" everywhere you are going to use ScaLAPACK routines, also you have to define all extern library calls that you are going to use, but I suppose that there are some include files available on the web already. Remember that Fortran uses different matrix definitions, etc... just read about calling Fortran subroutines from C/C++ differences. Anyway, relax and see the example.cpp by yourself.

Hope you will enjoy your ScaLAPACK ;-)

Marcin.Skoczylas@pb.edu.pl