Optimization and parallelization of 3D TDEM simulations

For reference: the forward problem

For a single electric source term, the forward problem can be defined as:

\begin{bmatrix}
\mathbf{A_{dc}} & & & & & \\
\mathbf{B_1 G} & \mathbf{A_1} & & & & \\
& \mathbf{B_2} & \mathbf{A_2} & & & \\
& & & \ddots & & \\
& & & & \mathbf{B_n} & \mathbf{A_n}
\end{bmatrix}
\begin{bmatrix}
\phi_0 \\ \mathbf{u_1} \\ \mathbf{u_2} \\ \vdots \\ \mathbf{u_n}
\end{bmatrix}
=
\begin{bmatrix}
\mathbf{q_{dc}} \\ \mathbf{q_1} \\ \mathbf{q_2} \\ \vdots \\ \mathbf{q_n}
\end{bmatrix}

For the e-field formulation

$$ \begin{align} \mathbf{A_k} &= \mathbf{C^T M_\mu C} + \Delta t_k^{-1} \mathbf{N_e^T M_\sigma N_e} \ \mathbf{B_k} &= -\Delta t_k^{-1} \mathbf{N_e^T M_\sigma N_e} \ \mathbf{q_k} &= -\Delta t_k^{-1} \mathbf{N_e^T s} \Big [ f_k - f_{k-1} \Big ] \end{align} $$

Issue 1: Fields for many sources

At each time step, the right-hand side for all sources is computed, then the solve is done for all right-hand sides simultaneously. The efficiency of the this is dependent on the solver being used and how it handles many right-hand sides.

• Q: Is there a limitation for storing the fields for an extremely large problem? (I.e. large mesh and/or large number of sources)
• Q: Can a more light-weight fields object be used for most practical applications?

Issues 2: Projection from fields to receivers

The discretized fields for all sources have been computed on the mesh and stored. At this point, we loop over all sources and receivers to project from the fields object to the receivers. I think the projection is done as needed and the projection matrices are not stored explicitly.

• Q: Is there a more efficient way to project from the fields to the receivers?

Issue 3: Storing factorizations of A:

For both the forward problem, and implicit computations of $\mathbf{Jv}$, we need to solve many linear systems of the form:

$$ \mathbf{A}\mathbf{x} = \mathbf{b} $$

for each time-step length.

Current SimPEG approach:

When the time-step length changes, the factorization of $\mathbf{A}$ for the previous step-length is discarded. If all we need is to solve the forward problem, this is fine. However if we need many implicit computations of $\mathbf{Jv}$, everything will need to be re-factored for each vector. This is not efficient.

Improvements:

Can we have the options to:

• Discard factorizations when step length changes
  • forward simulation when RAM is severely limited
  • wouldn’t be required if sensitivities were computed and stored explicitly (done using Jtv)
• Store all factorizations for the current model to RAM
  • for inversion Jv must be solved implicitly
• Store all factorizations for the current model to disk
  • for inversion Jv must be solved implicitly

Storing Factorizations of A^T

For implicit computations of $\mathbf{J^T v}$ (which is required to compute and store the sensitivities), we need to solve many linear systems of the form:

$$ \mathbf{A^T x} = \mathbf{b} $$

for each time-step length.

Current SimPEG approach:

When the time-step length changes, the factorization of $\mathbf{A^T}$ for the previous step-length is discarded. If we need many implicit computations of $\mathbf{J^T v}$, everything will need to be re-factored for each vector. This is not efficient. This issue occurs when:

• $\mathbf{J^T v}$ is computed implicitly for many vectors to compute and store the sensitivities explicitly
• $\mathbf{J^T v}$ is computed implicitly many times for the CG solve when sensitivities are not stored

Improvements:

Can we have the options to:

• Store all factorizations for the current model to RAM
• Store all factorizations for the current model to disk

[JKutt]

That all sound good @dccowan. Totally agreed.

@domfournier, I am on board with you re-working stuff with fields. @jcapriot and I were chatting recently and the fields topic came up. Hopefully after the holidays we could all sit down and start mapping this out 🚀 . I should be able to put some dev time in soon.

[domfournier]

Might be a seperate issue, but it would be very beneficial to re-work the way fields are stored, mostly trying to remove the mesh object as much as possible. Mostly since it is mostly used as a simple index at the moment. Fields should be a dictionary, that’s it.