Code_Aster 14.2 in parallelo su Ubuntu Bionic

Questo post mostra come costruire una versione di Code_Aster 14.2 in parallelo a partire dai sorgenti. Questa procedura, che richiede una conoscenza intermedia dell’ambiente Linux, non è ufficiale ma è stata creata a partire da suggerimenti e procedure esistenti.

La distribuzione di partenza utilizzata è una Ubuntu Bionic 18.04 LTS, anche se è possibile adattare la procedura ad altre versioni di Ubuntu e di Code_Aster. I comandi non devono essere eseguiti come “root” ma come utente normale (in particolare l’installazione di Code_Aster), la maggior parte dell’installazione viene effettuata all’interno della directory home dell’utente (ma è possibile metterla altrove, purchè l’utente abbia i diritti necessari).

Per prima cosa occorre installare alcuni strumenti e librerie:

sudo apt-get install -y curl mercurial gfortran g++ python-dev python-numpy liblapack-dev libblas-dev tcl tk zlib1g-dev bison flex checkinstall openmpi-bin cmake hdf5-tools swig grace gettext libboost-dev libboost-python-dev libopenmpi-dev tkpng libatlas-base-dev libblacs-mpi-dev

Definiamo alcune variabili di ambiente con le versioni e i percorsi per i software da installare e costruiamo alcune directory dove verranno messi i sorgenti da compilare (DEV_DIR) e i risultati delle compilazioni (INSTALL_DIR):

export ASTER_VER=14.2
export DEV_DIR=~/dev
export INSTALL_DIR=~/install
export HDF5_VER=1.8.14
export MED_VER=3.3.1
export METIS_VER=5.1.0
export PARMETIS_VER=4.0.3
export SCOTCH_VER=6.0.4
export MUMPS_VER=5.1.2
export MFRONT_VER=3.1.1
export PETSC_VER=3.8.2
export SCALAPACK_VER=2.0.2
export ASTER_ROOT=$INSTALL_DIR/aster
export ASTER_PUBLIC=$ASTER_ROOT/public
export ASTER_FULLSRC_DIR=$DEV_DIR/aster-full-src-${ASTER_VER}.0

mkdir $INSTALL_DIR
mkdir -p $DEV_DIR/aster-prerequisites
mkdir -p $DEV_DIR/codeaster

Installiamo gli strumenti di sviluppo per Code_Aster, utili per verificare l’installazione una volta effettuata:

cd $DEV_DIR/codeaster

curl -o ~/.hgrc https://bitbucket.org/code_aster/codeaster-devtools/raw/tip/etc/hgrc_template # modify and set user name and email

hg clone https://bitbucket.org/code_aster/codeaster-devtools devtools

Scarichiamo e compiliamo la versione sequenziale di Code_Aster:

curl -Sl https://www.code-aster.org/FICHIERS/aster-full-src-$ASTER_VER.0-1.noarch.tar.gz | tar -xzC $DEV_DIR

cd $ASTER_FULLSRC_DIR
python setup.py --prefix $INSTALL_DIR/aster

Scarichiamo ptscotch:

cd $DEV_DIR/aster-prerequisites
mkdir ptscotch

tar xf $ASTER_FULLSRC_DIR/SRC/scotch-${SCOTCH_VER}-aster5.tar.gz -C ptscotch --strip-components 1

Prima di poterlo compilare, occorre modificare i sorgenti per disattivare l’opzione SCOTCH_PTHREAD (a questo scopo costruiamo ad applichiamo una patch), poi possiamo compilare:

cat >> patch_ptscotch << 'EOF'
diff -ruN ptscotch/src/Makefile.inc ptscotch-without-pthread/src/Makefile.inc
--- ptscotch/src/Makefile.inc	2016-10-07 07:43:22.000000000 +0000
+++ ptscotch-without-pthread/src/Makefile.inc	2018-06-08 08:41:07.693702800 +0000
@@ -9,7 +9,7 @@
 CCS		= gcc
 CCP		= mpicc
 CCD		= gcc
-CFLAGS		= -O3 -fPIC -DCOMMON_FILE_COMPRESS_GZ -DCOMMON_PTHREAD -DCOMMON_RANDOM_FIXED_SEED -DSCOTCH_RENAME -DSCOTCH_RENAME_PARSER -DSCOTCH_PTHREAD -Drestrict=__restrict
+CFLAGS		= -O3 -fPIC -DCOMMON_FILE_COMPRESS_GZ -DCOMMON_PTHREAD -DCOMMON_RANDOM_FIXED_SEED -DSCOTCH_RENAME -DSCOTCH_RENAME_PARSER -Drestrict=__restrict
 CLIBFLAGS	=
 LDFLAGS		= -fPIC -lz -lm -pthread -lrt
 CP		= cp
EOF

patch -s -p0 < patch_ptscotch
cd ptscotch/src
make scotch esmumps ptscotch ptesmumps CCD=mpicc
mkdir ${ASTER_PUBLIC}/ptscotch-${SCOTCH_VER}
make install prefix=${ASTER_PUBLIC}/ptscotch-${SCOTCH_VER}

Scarichiamo e compiliamo parmetis:

cd $DEV_DIR/aster-prerequisites
wget --no-check-certificate --quiet ${PARMETIS_SRC} -O parmetis.tar.gz
mkdir parmetis
tar xf parmetis.tar.gz -C parmetis --strip-components 1
cd parmetis
make config prefix=${ASTER_PUBLIC}/parmetis-${PARMETIS_VER}
make -j
make install

Stessa cosa per scalapack:

cd $DEV_DIR/aster-prerequisites
mkdir scalapack

curl -Sl http://www.netlib.org/scalapack/scalapack-${SCALAPACK_VER}.tgz | tar -xzC $DEV_DIR/aster-prerequisites/scalapack --strip-components 1

cd scalapack && mkdir build && cd build

cmake -DCMAKE_INSTALL_PREFIX=${ASTER_PUBLIC}/scalapack-${SCALAPACK_VER} -DBUILD_SHARED_LIBS=ON -DBUILD_STATIC_LIBS=ON ..

make -j && make install
cd ${ASTER_PUBLIC}/scalapack-${SCALAPACK_VER}/lib
cp libscalapack.so libblacs.so

Finalmente possiamo compilare il risolutore Mumps (notare che la lunga riga di configurazione è un comando unico):

cd $DEV_DIR/aster-prerequisites
mkdir mumps
tar xf $ASTER_FULLSRC_DIR/SRC/mumps-${MUMPS_VER}-aster5.tar.gz -C mumps --strip-components 1
cd mumps

INCLUDES="${ASTER_PUBLIC}/metis-${METIS_VER}/include ${ASTER_PUBLIC}/parmetis-${PARMETIS_VER}/include ${ASTER_PUBLIC}/ptscotch-${SCOTCH_VER}/include" LIBPATH="${ASTER_PUBLIC}/metis-${METIS_VER}/lib ${ASTER_PUBLIC}/parmetis-${PARMETIS_VER}/lib ${ASTER_PUBLIC}/ptscotch-${SCOTCH_VER}/lib ${ASTER_PUBLIC}/scalapack-${SCALAPACK_VER}/lib" ./waf configure --prefix=${ASTER_PUBLIC}/mumps-${MUMPS_VER}_mpi --install-tests --enable-mpi

./waf build --jobs=1
./waf install --jobs=1

Possiamo ora scaricare ed installare il risolutore iterativo Petsc:

cd $DEV_DIR/aster-prerequisites
mkdir petsc

curl -Sl https://bitbucket.org/petsc/petsc/get/v${PETSC_VER}.tar.gz | tar --strip-components 1 -xzC $DEV_DIR/aster-prerequisites/petsc

cd petsc

./configure --COPTFLAGS="-O2" --CXXOPTFLAGS="-O2"  --FOPTFLAGS="-O2"  --with-debugging=0 --with-shared-libraries=1  --with-scalapack-dir=${ASTER_PUBLIC}/scalapack-${SCALAPACK_VER}  --with-mumps-dir=${ASTER_PUBLIC}/mumps-${MUMPS_VER}_mpi  --with-metis-dir=${ASTER_PUBLIC}/metis-${METIS_VER}  --with-ptscotch-dir=${ASTER_PUBLIC}/ptscotch-${SCOTCH_VER} --download-hypre --download-ml  --LIBS="-lgomp" --prefix=${ASTER_PUBLIC}/petsc-${PETSC_VER}

make all && make install

cp ./arch-linux2-c-opt/lib/libpetsc.so ${ASTER_PUBLIC}/petsc-${PETSC_VER}/lib/

Possiamo ora prepararci a compilare Code_Aster in parallelo:

cd $DEV_DIR/
source ${ASTER_ROOT}/${ASTER_VER}/share/aster/profile_mfront.sh
export METISDIR=${ASTER_PUBLIC}/metis-${METIS_VER}
export TFELHOME=${ASTER_PUBLIC}/tfel-${MFRONT_VER}
export GMSH_BIN_DIR=${ASTER_PUBLIC}/gmsh-3.0.6-Linux64/bin
export HOMARD_ASTER_ROOT_DIR=${ASTER_PUBLIC}/homard-11.10

mkdir aster

tar xf $ASTER_FULLSRC_DIR/SRC/aster-${ASTER_VER}.0.tgz -C aster --strip-components 1

cd aster

echo >> $DEV_DIR/aster/wafcfg/Ubuntu_mpi.py << 'EOF'
def configure(self):
    self.env.append_value('LIB_METIS', ('parmetis'))
    self.env.append_value('LIB_SCOTCH', ('ptscotch', 'ptscotcherr', 'ptscotcherrexit'))
EOF

INCLUDES="${ASTER_PUBLIC}/hdf5-${HDF5_VER}/include ${ASTER_PUBLIC}/med-${MED_VER}/include ${ASTER_PUBLIC}/metis-${METIS_VER}/include ${ASTER_PUBLIC}/parmetis-${PARMETIS_VER}/include ${ASTER_PUBLIC}/ptscotch-${SCOTCH_VER}/include ${ASTER_PUBLIC}/mumps-${MUMPS_VER}_mpi/include ${ASTER_PUBLIC}/petsc-${PETSC_VER}/include ${ASTER_PUBLIC}/tfel-${MFRONT_VER}/include" LIBPATH="${ASTER_PUBLIC}/hdf5-${HDF5_VER}/lib ${ASTER_PUBLIC}/med-${MED_VER}/lib ${ASTER_PUBLIC}/metis-${METIS_VER}/lib ${ASTER_PUBLIC}/parmetis-${PARMETIS_VER}/lib ${ASTER_PUBLIC}/ptscotch-${SCOTCH_VER}/lib ${ASTER_PUBLIC}/scalapack-${SCALAPACK_VER}/lib ${ASTER_PUBLIC}/mumps-${MUMPS_VER}_mpi/lib ${ASTER_PUBLIC}/petsc-${PETSC_VER}/lib ${ASTER_PUBLIC}/tfel-${MFRONT_VER}/lib" ./waf configure --use-config=Ubuntu_mpi --prefix=${ASTER_ROOT}/${ASTER_VER}_mpi --install-tests --enable-mpi

./waf build && ./waf install

Dobbiamo registrare la nuova versione di Code_Aster per renderla disponibile e la macchina attuale come nodo MPI, inoltre occorre aggiungere i binari di Code_Aster nel PATH (eventualmente aggiungerli a ~/.bashrc):

echo "vers : testing_mpi:${ASTER_ROOT}/${ASTER_VER}_mpi/share/aster" >> ${ASTER_ROOT}/etc/codeaster/aster

echo "vers : ${ASTER_VER}_mpi:${ASTER_ROOT}/${ASTER_VER}_mpi/share/aster" >> ${ASTER_ROOT}/etc/codeaster/aster

echo "localhost" > ${ASTER_ROOT}/etc/codeaster/mpi_hostfile
export PATH=$ASTER_ROOT/bin:$PATH
as_run --info

Occorre anche modificare un file di configurazione di Code_Aster, che è configurato per Mpich, mentre questa procedura usa OpenMPI:

echo >> asrun_patch.diff << EOF
230c230,231
< mpi_get_procid_cmd : echo $PMI_RANK
---
> #mpi_get_procid_cmd : echo $PMI_RANK
> mpi_get_procid_cmd : echo $OMPI_COMM_WORLD_RANK
EOF 

patch ${ASTER_ROOT}/etc/codeaster/asrun < asrun_patch.diff

Eseguiamo un semplice test per provare l’installazione:

./waf test -n sslp114a

Per eseguire una validazione più completa, possiamo usare gli strumenti di sviluppo:

source $HOME/dev/codeaster/devtools/etc/env.sh
ln -s $DEV_DIR/aster/ $DEV_DIR/aster/src

run_testcases --root=$DEV_DIR/aster --builddir=build --testlist=verification --resutest=/tmp/resutest --force-mpi  --vers=${ASTER_ROOT}/${ASTER_VER}_mpi/share/aster

Questa procedura non è perfetta: si tratta di un punto di partenza su cui lavorare per far passare più test possibili e viene fornita per mostrare le varie tappe necessarie più che come risultato finale. Sarebbe inoltre utile in futuro modificarla per utilizzare di più le librerie di sistema (come i pacchetti “libscalapack-mpi-dev libmetis-dev libparmetis-dev libscotch-dev libptscotch-dev”) invece di ricompilare tutti i prerequisiti da zero. Un altro punto di miglioramento consisterebbe nel ricompilare i prerequisiti direttamente, invece di appoggiarsi su quelli della versione sequenziale.

10 risposte a “Code_Aster 14.2 in parallelo su Ubuntu Bionic”

  1. Hello,

    thank you very much for your contribution to the parallel compilation of code_aster!

    Your prescriptions work perfect, up to the point where petsc is to be installed: When I issue

    curl -Sl https://bitbucket.org/petsc/petsc/get/v${PETSC_VER}.tar.gz | tar –strip-components 1 -xzC $DEV_DIR/aster-prerequisites/petsc

    it turns out that the respective directory /petsc/petsc no longer exists on bitbucket. It was moved to github. But my attempts to download the petsc archive from there all failed.

    Furthermore, the ./configure command does not work with other petsc archives that I tried to download from the internet. The configuration always stops at the point where something with mumps-5.1.2_mpi does not work.

    So I ask myself: Do you think you could continue your great work with providing instructions for CA14.2 that include current, up-to-date packages? That would be extremely helpful to a great number of code aster practitioners. It would certainly contribute to your “fame” in the community … 🙂

    With my best regards
    Hanno Hammer

    1. Hello,

      Many projects are currently being moved from Bitbucket, sorry if this “broke” the procedure… I will try to fix it (or at least to publish a new procedure as soon as the new version 15 is released).
      I will try to do it as soon as my “urgent TODO list” is a bit smaller 😉
      For the time being, you might try to:
      – Download the corresponding PETSC version from the main site (is this the one with the “configure” issue?): https://www.mcs.anl.gov/petsc/download/index.html
      – Try to disable PETSC (it is not always needed, depending on your target)
      – Share the mumps error message, maybe we can workaround it ?
      Thank you for writing,

      Best regards,
      Luca Dall’Olio

      1. Dear Mr. Dall’Olio,

        thank you for your quick reply!

        As to your first item, “download PETSC from main site”:

        The thing is that the main site (link as provided above by yourself) does NOT contain the PETSC version you are referencing (3.8.2). Instead, they have 3.7.7, followed by 3.8.4.
        I tried your instructions with both of the above, but both produce the error. Then I tried to download the 3.8.2 from another archive, but the config stage failed as well.

        Your second item: “Disable PETSC”:

        Well, I don’t know how to do that …

        Your third item: “Share the mumps error”:

        Yes, I will do that tomorrow. I will post the error as terminal message and, if possible, will attach the config.log.

        In the meantime,
        Kind regards
        Hanno Hammer

        1. Hello,

          Sorry for the quick, and untested, hints: all I can do at the moment… Did you try to download Petsc from here ?
          http://ftp.mcs.anl.gov/pub/petsc/release-snapshots/petsc-3.8.2.tar.gz

          Abount “disabling” PETSC, you could try to add
          the option –disable-petsc when calling waf configure and (if it is not enough) remove all references to PETSC in the INCLUDES and LIBPATH environment variable definitions.
          Sorry again if this seems cryptic, I am in a hurry this morning 🙂

          All the best,

          Luca

  2. Hi Luca,

    thank you very much for your quick response.

    I have tried twice to post the mumps error message via this online form, but they do not get published. I suspect the culprit to be an over-zealous spam filter which swallows text containing control characters.

    Do you have any suggestions how we can communicate on this matter?

    Best regards
    Hanno

    1. Hello again,

      I just tried to disable PETSC according to your instructions, together with removing the references to PETSC in the INCLUDES and LIBPATH environment variable definitions. The waf build failed, with error message:

      Build failed
      -> task in ‘asterexec’ failed (exit status 1):
      task details in: /home/ubuntu/dev/aster/build/release/asterexec.log

      Kind regards
      Hanno

  3. Hi Hanno,

    I wrote you an email, I think we could go on exchanging there (will be easier). About posting error messages, we could also try something like postbin.com but personal messages would probably be better in this case. I think that I need to work on a new procedure (as for your message, the information is probably in your /home/ubuntu/dev/aster/build/release/asterexec.log file).

    Sorry for all this effort, recompiling Code_Aster in parallel can be challenging (it shouldn’t be of course)
    All the best,
    Luca

  4. Hello Luca,

    Thank you for the helpful procedure!
    I encountered the same issue as Hanno. Configuring petsc I obtain the error:

    –with-mumps-dir=/home/install/aster/public/mumps-5.1.2_mpi did not work

    Do you have any suggestion to solve the problem?
    Many thanks,
    Aldo

    1. Hello Aldo,
      Hard to say, these compilations might fail for a lot of reasons (different operating system, are you on Bionic ?, different environment, …)
      In some future I plan to update this procedure to the new 14.4 Code_Aster installation hoping that it will solve some of these issues

      All the best,
      Luca

Lascia un commento

Il tuo indirizzo email non sarà pubblicato. I campi obbligatori sono contrassegnati *

Questo sito usa Akismet per ridurre lo spam. Scopri come i tuoi dati vengono elaborati.