Homogenization3D fails when using MPI
In MR !59, the example Homogenization3D
fails with an error when run with multiple MPI processes:
[be770a7b8dc2:06932] Read -1, expected 14112, errno = 1
[be770a7b8dc2:06932] *** Process received signal ***
[be770a7b8dc2:06932] Signal: Aborted (6)
[be770a7b8dc2:06932] Signal code: (-6)
[be770a7b8dc2:06932] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x46210)[0x7f8ce74ee210]
[be770a7b8dc2:06932] [ 1] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb)[0x7f8ce74ee18b]
[be770a7b8dc2:06932] [ 2] /lib/x86_64-linux-gnu/libc.so.6(abort+0x12b)[0x7f8ce74cd859]
[be770a7b8dc2:06932] [ 3] /usr/lib/petscdir/petsc3.12/x86_64-linux-gnu-real/lib/libpetsc_real.so.3.12(PetscTraceBackErrorHandler+0x2ee)[0x7f8ce79c2bbe]
[be770a7b8dc2:06932] [ 4] /usr/lib/petscdir/petsc3.12/x86_64-linux-gnu-real/lib/libpetsc_real.so.3.12(PetscError+0x20b)[0x7f8ce79bf4fb]
[be770a7b8dc2:06932] [ 5] /usr/lib/petscdir/petsc3.12/x86_64-linux-gnu-real/lib/libpetsc_real.so.3.12(PetscCommBuildTwoSidedFReq+0x12b)[0x7f8ce79ff7db]
[be770a7b8dc2:06932] [ 6] /usr/lib/petscdir/petsc3.12/x86_64-linux-gnu-real/lib/libpetsc_real.so.3.12(+0x652b7b)[0x7f8ce7ee9b7b]
[be770a7b8dc2:06932] [ 7] /usr/lib/petscdir/petsc3.12/x86_64-linux-gnu-real/lib/libpetsc_real.so.3.12(+0x6553db)[0x7f8ce7eec3db]
[be770a7b8dc2:06932] [ 8] /usr/lib/petscdir/petsc3.12/x86_64-linux-gnu-real/lib/libpetsc_real.so.3.12(MatAssemblyBegin_MPIAIJ+0x72)[0x7f8ce7d80122]
[be770a7b8dc2:06932] [ 9] /usr/lib/petscdir/petsc3.12/x86_64-linux-gnu-real/lib/libpetsc_real.so.3.12(MatAssemblyBegin+0x9a)[0x7f8ce7f1c6ea]
[be770a7b8dc2:06932] [10] build/examples/ae108-examples-Homogenization3D(+0x353db)[0x55fe57df03db]
[be770a7b8dc2:06932] [11] build/examples/ae108-examples-Homogenization3D(+0x2e586)[0x55fe57de9586]
[be770a7b8dc2:06932] [12] build/examples/ae108-examples-Homogenization3D(+0x27024)[0x55fe57de2024]
[be770a7b8dc2:06932] [13] build/examples/ae108-examples-Homogenization3D(+0x20701)[0x55fe57ddb701]
[be770a7b8dc2:06932] [14] build/examples/ae108-examples-Homogenization3D(+0x182ef)[0x55fe57dd32ef]
[be770a7b8dc2:06932] [15] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7f8ce74cf0b3]
[be770a7b8dc2:06932] [16] build/examples/ae108-examples-Homogenization3D(+0x170fe)[0x55fe57dd20fe]
[be770a7b8dc2:06932] *** End of error message ***
--------------------------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
with errorcode 59.
NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[0]PETSC ERROR: or see https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run
[0]PETSC ERROR: to get more information on the crash.
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Signal received
[0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.12.4, Feb, 04, 2020
[0]PETSC ERROR: build/examples/ae108-examples-Homogenization3D on a named be770a7b8dc2 by Unknown Wed Dec 22 10:06:13 2021
[0]PETSC ERROR: Configure options --build=x86_64-linux-gnu --prefix=/usr --includedir=${prefix}/include --mandir=${prefix}/share/man --infodir=${prefix}/share/info --sysconfdir=/etc --localstatedir=/var --with-silent-rules=0 --libdir=${prefix}/lib/x86_64-linux-gnu --runstatedir=/run --with-maintainer-mode=0 --with-dependency-tracking=0 --with-debugging=0 --shared-library-extension=_real --with-shared-libraries --with-pic=1 --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 --with-cxx-dialect=C++11 --with-opencl=1 --with-blas-lib=-lblas --with-lapack-lib=-llapack --with-scalapack=1 --with-scalapack-lib=-lscalapack-openmpi --with-mumps=1 --with-mumps-include="[]" --with-mumps-lib="-ldmumps -lzmumps -lsmumps -lcmumps -lmumps_common -lpord" --with-suitesparse=1 --with-suitesparse-include=/usr/include/suitesparse --with-suitesparse-lib="-lumfpack -lamd -lcholmod -lklu" --with-ptscotch=1 --with-ptscotch-include=/usr/include/scotch --with-ptscotch-lib="-lptesmumps -lptscotch -lptscotcherr" --with-fftw=1 --with-fftw-include="[]" --with-fftw-lib="-lfftw3 -lfftw3_mpi" --with-superlu=1 --with-superlu-include=/usr/include/superlu --with-superlu-lib=-lsuperlu --with-superlu_dist=1 --with-superlu_dist-include=/usr/include/superlu-dist --with-superlu_dist-lib=-lsuperlu_dist --with-hdf5-include=/usr/include/hdf5/openmpi --with-hdf5-lib="-L/usr/lib/x86_64-linux-gnu/hdf5/openmpi -L/usr/lib/openmpi/lib -lhdf5 -lmpi" --CXX_LINKER_FLAGS=-Wl,--no-as-needed --with-hypre=1 --with-hypre-include=/usr/include/hypre --with-hypre-lib=-lHYPRE_core --prefix=/usr/lib/petscdir/petsc3.12/x86_64-linux-gnu-real --PETSC_ARCH=x86_64-linux-gnu-real CFLAGS="-g -O2 -fstack-protector-strong -Wformat -Werror=format-security -fPIC" CXXFLAGS="-g -O2 -fstack-protector-strong -Wformat -Werror=format-security -fPIC" FCFLAGS="-g -O2 -fstack-protector-strong -fPIC -ffree-line-length-0" FFLAGS="-g -O2 -fstack-protector-strong -fPIC -ffree-line-length-0" CPPFLAGS="-Wdate-time -D_FORTIFY_SOURCE=2" LDFLAGS="-Wl,-Bsymbolic-functions -Wl,-z,relro -fPIC" MAKEFLAGS=w
[0]PETSC ERROR: #1 User provided function() line 0 in unknown file
--------------------------------------------------------------------------
mpirun noticed that process rank 1 with PID 0 on node be770a7b8dc2 exited on signal 6 (Aborted).
--------------------------------------------------------------------------