Simulation crashed on an Electromagnetic scatteirng problem

I copied my github issue report here in case people in the forum have encountered this before.
Hi, I tried Bempp on electromagnetic scattering problems and the jupyter kernel crashed. The simulation was running for several hours, and I don’t think I ran out of memory. It’s kind of hard for me to locate the cause of such a crash.

I encountered this problem earlier at a smaller resolution and I was able to bypass it by forcing specific versions of libraries (see below). But now I have this issue again on a larger resolution. I attached the jupyter log file and the links to the notebook file and mesh file I used. I’m wondering if there is something suspicious to you in the script or the log file; or if there is a specific python environment setting I should try. Thank you!

I’m using
python=3.8.5
ipykernel=5.3.4
ipython=7.19.0
numba=0.51.2
nbformat=5.0.8
Bempp-cl=0.2.4

jupyter.log
The script I used is here: maxwell_cylinder_crash.ipynb - Google Drive
The mesha I used i here: cylinder_r4_res0.2.msh - Google Drive

Should I expect the memory usage to have a big increase during matrix assembly? From Py-spy it seems that the program has been running matrix assembly and the memory usage went from 22G to 50G. I’m not so sure about the not-out-of-memory argument anymore. I said that in the above based on my observation of the memory usage at the beginning of the matrix assembly and I just realized the memory usage increased quite a lot during matrix assembly.

Hi, Mandy,
  I have some good experience about the memory usage, although it is an acoustic scattering problem. If using dense mode of Bempp instead of FMM, the memory usage increases rapidly. The rough evaluation is that the ratio of memory and frequency is: Memory \propto O(N^2) \propto O(f^4), where N is the mesh point number. It means that the memory consumption will be quadruple if the mesh point number is doubled (increasing resolution). Besides, the peak point(matrix assembly) of Bempp consumes almost 2.5 times memory than the memory occupation of a single dense matrix(my test). That means you need more available memory to run Bempp along the whole process.

  There are possibly three choices:

  • (1) increase your server’s memory or virtual memory;
  • (2) consider FMM to reduce the complexity and memory;
  • (3) revise Bempp to distributed version.

Hope it is useful to you!

Best wishes,
Long