GPU Computing openEMS FDTD RCS

Why GPU Acceleration Did Not Make openEMS a Fast RCS Engine

A CUDA backend accelerated the core Yee stencil, but large RCS workloads exposed a deeper bottleneck: openEMS' CPU-oriented simulation interface.

Section 1 - The RCS compute problem

The first RCS report in this repository showed that openEMS can be a useful open-source tool for radar observability studies. It validated a sphere against the Mie series, used field dumps to reason about spatial backscatter, and showed that small enclosed structures can dominate the radar return from a UAV-like geometry.

That result immediately creates the next problem: useful RCS design studies are not single simulations. They are sweeps over aspect angle, frequency, polarization, material assumptions, mesh resolution, and geometry variants. For finite-difference time-domain (FDTD), each refinement in physical scale or frequency pushes the domain volume upward. Memory scales roughly with the number of cells, and cell count scales as the cube of the domain resolution.

For early experiments, openEMS was attractive because it already provided a complete simulation front end: geometry import, meshing, boundary conditions, sources, probes, near-field to far-field transforms, and Python scripting. The natural follow-on question was whether the expensive part of the workflow could be accelerated without throwing away that infrastructure.

The CUDA project in repo759_public was an attempt to answer that question. The goal was not to build a separate toy FDTD solver. The goal was to keep normal openEMS problem setup intact while replacing the core field-update engine with a CUDA backend.

Section 2 - The CUDA experiment

openEMS advances electromagnetic fields on a Yee grid. The local voltage/current updates are a natural GPU target: each field cell depends on nearby staggered field values and coefficients, so there is abundant parallelism and a regular memory-access pattern. In openEMS terminology, the CUDA backend implements an Engine_CUDA replacement for the standard CPU engine.

Project architecture: the existing openEMS front end builds the simulation, while Engine_CUDA accelerates the FDTD update.
The CUDA project preserved the openEMS front end and inserted a GPU-backed FDTD engine underneath it. That choice kept openEMS usability, but also preserved CPU-oriented extension interfaces.

The first working CUDA backend was correct but slow. On a 100^3 cavity benchmark it took 48.7 ms per timestep, roughly 11x slower than the CPU baseline. The optimization sequence improved that to 1.793 ms per timestep, a 2.107x speedup over the 4-core CPU baseline for the same small benchmark.

The optimization path included:

  • fixing an engine-dispatch bug that accidentally routed extension calls through CPU-side buffers;
  • reducing full-field host/device transfers;
  • improving managed-memory residency with cudaMemAdvise;
  • aligning CUDA thread layout with the field-array memory order;
  • adding shared-memory tiling for neighbor field reuse;
  • packing coefficients into float2 loads;
  • adding a probe-gather fast path for small sets of CPU-visible field reads.
Optimization history from the first working CUDA backend through the best merged implementation.
The small benchmark did improve: the project moved from a slow but correct prototype to a real speedup. The later question was whether that improvement translated into RCS-scale workloads.

Section 3 - The core kernels were not the final bottleneck

The Yee stencil is memory-bound. Each update performs a modest amount of floating-point work while moving several field and coefficient values. The right optimization target is therefore memory traffic, coalescing, cache behavior, and host/device synchronization, not peak FP32 throughput.

Roofline model for the CUDA FDTD update.
The core FDTD update sits in the memory-bound region. That makes memory layout, transfer avoidance, and synchronization more important than raw floating-point peak.

The important result is subtle. The CUDA kernels did become meaningfully faster than the 4-core CPU update path, but the full openEMS timestep did not continue to scale. In the large-grid sweep, the core voltage/current kernels stayed in the right performance regime while the total CUDA timestep became dominated by everything around those kernels.

Grid CPU 4-core ms/step CUDA full ms/step CUDA kernel subtotal Full CUDA speedup
100^3 2.959 1.732 0.457 1.71x
150^3 11.450 6.842 1.431 1.67x
200^3 28.513 18.967 3.456 1.50x
250^3 55.382 51.176 8.653 1.08x
300^3 106.250 118.489 16.826 0.90x
400^3 215.000 1027.500 31.060 0.21x
500^3 447.500 3284.167 67.309 0.14x

At 500^3, the CUDA kernels required about 67 ms per timestep. The full CUDA timestep reported by openEMS was about 3284 ms. In other words, the kernels were only about 2 percent of the measured CUDA timestep. The GPU was not out of memory, and the stencil kernels were not the dominant runtime.

Large-grid CPU/CUDA scaling.
The CUDA backend is faster on small grids, reaches parity near 250^3, and then loses badly at larger sizes.
CUDA kernel fraction of full CUDA timestep.
This is the central result. At larger grid sizes, the accelerated kernels are only a small fraction of the full openEMS timestep.

Section 4 - What openEMS was really exposing

The useful finding is that openEMS is not simply "an FDTD kernel." It is a scientific application framework built around a CPU-oriented engine interface. That interface is one of the reasons openEMS is useful: sources, probes, ports, boundary conditions, and near-field/far-field infrastructure can be composed from the high-level Python workflow.

The same interface becomes expensive when the field arrays live on a GPU. Many openEMS extensions expect synchronous CPU-side access to field values during every timestep through methods such as:

  • GetVolt()
  • SetVolt()
  • GetCurr()
  • SetCurr()

In the CPU engine, those calls are ordinary memory accesses. In a GPU engine, they can imply synchronization, remote reads over PCIe, page migration, or fallback paths that defeat the advantage of keeping fields resident on the device.

The small 100^3 benchmark could be helped by a probe-gather fast path: record which field cells were being read, gather just those cells on the GPU, and copy a tiny buffer back to the host. That does not solve the larger architectural problem. Boundary conditions, PML regions, probes, material updates, and near-to-far reductions may touch many cells in patterns that were not designed as GPU-resident dataflow.

The project therefore changed from "make the kernels faster" to "identify the application boundary that prevents kernel speed from becoming application speed."

flowchart TD
    A["RCS goal: larger sweeps and finer grids"] --> B["Try CUDA backend inside openEMS"]
    B --> C["Yee stencil kernels improve"]
    C --> D["Small benchmark reaches 2.107x speedup"]
    D --> E["Large-grid sweep exposes interface overhead"]
    E --> F["CPU-oriented extension callbacks dominate"]
    F --> G["Conclusion: future speed requires GPU-resident simulation dataflow"]

    C --> H["Kernel-only path is promising"]
    E --> I["Full openEMS timestep is not"]

    style H fill:#1f6feb,color:#fff
    style I fill:#8b0000,color:#fff
    style G fill:#2ea44f,color:#fff

Section 5 - Why this matters for RCS

For the RCS workflow, the target capability is not a single 100^3 cavity benchmark. The target is design exploration:

  • many observation angles;
  • multiple radar bands;
  • different polarizations;
  • material models such as anisotropic CFRP;
  • geometry variants;
  • increasingly fine meshes around cavities, edges, and apertures.

A 2x small-grid speedup is useful, but it does not transform the RCS workflow. The original goal was closer to a 20x practical capability jump plus a path to multi-GPU scaling. The CUDA experiment did not reach that as a drop-in openEMS backend.

It did, however, answer a valuable question. The limiting factor was not that FDTD is impossible to accelerate on a GPU. The core field update accelerated. The limiting factor was that the existing application boundary kept pulling the simulation back toward CPU-visible state every timestep.

That distinction matters. If the conclusion had been "FDTD kernels are too slow on this GPU," the next step would be more kernel optimization or bigger hardware. Instead, the result points toward architecture: data layout, extension APIs, boundary-condition execution, probes, reductions, and multi-device decomposition all need to be designed around resident parallel data.

Estimated and sampled memory use across the large-grid sweep.
The 500^3 run did not exhaust RTX 4000 Ada memory. The performance failure came before memory capacity became the main limit.

Section 6 - Future architecture options

There are three plausible paths from here.

Option A: keep patching openEMS

This is the least disruptive path. More openEMS extension work could be moved onto the GPU incrementally: boundary conditions, probe reductions, lumped-port accumulation, near-to-far transforms, and material updates. That would preserve the existing ecosystem while reducing the worst synchronization paths.

The risk is that the interface remains CPU-shaped. Each improvement may fix one callback path while another extension reveals the same architectural mismatch. This can still be worthwhile, especially if the goal is a moderate speedup for existing openEMS users.

Option B: use unified-memory hardware as a bridge

Platforms such as NVIDIA Grace Blackwell make the memory boundary between CPU and GPU less painful. A coherent high-bandwidth CPU/GPU memory system could make some of openEMS' CPU-visible access patterns less catastrophic than they are on a PCIe-attached GPU.

That does not automatically make the software architecture good. Unified memory can reduce the penalty for crossing the boundary, but it does not remove the need for parallel algorithms, GPU-resident reductions, or distributed dataflow. It may be a practical bridge for research and prototyping, but it risks becoming a crutch if it delays the solver redesign that large RCS studies eventually need.

Option C: build a GPU-native EM/RCS platform

The cleanest long-term architecture is a solver designed from the start around GPU-resident field arrays, domain decomposition, halo exchange, and reduction pipelines. In that model, probes, boundaries, sources, material updates, and near-to-far transforms are not external CPU callbacks. They are stages in a device-resident simulation graph.

This path is more expensive, but it also lines up better with multi-GPU and multi-node scaling. The same redesign needed to avoid CPU/GPU synchronization is also the redesign needed for distributed simulation: explicit subdomains, boundary exchanges, asynchronous communication, and reductions that do not centralize every timestep through one host process.

Section 7 - Current recommendation

For this repository, openEMS remains the right open-source tool for validated, inspectable RCS experiments at modest scale. It is transparent, scriptable, and good enough to expose real electromagnetic mechanisms such as cavity-driven backscatter and polarization-sensitive material behavior.

For large-scale GPU-accelerated RCS design studies, the CUDA experiment suggests that a drop-in backend is not enough. The next serious capability step should probably be architectural:

  1. keep using openEMS as a validation and methodology reference;
  2. isolate the RCS-specific simulation patterns that matter most;
  3. prototype a GPU-resident FDTD/RCS subset with explicit boundary, probe, and near-to-far dataflow;
  4. design domain decomposition early, so the single-GPU solver can evolve toward multi-GPU and multi-node runs;
  5. treat unified-memory systems as useful hardware, not as a substitute for the distributed data model.

The negative result is therefore productive. The CUDA project did not turn openEMS into a fast aircraft-scale RCS engine. It showed why that is hard, where the actual bottleneck sits, and what kind of solver architecture would be needed to move beyond the current open-source capability level.

References