OpenFOAM on GPUs: integration paths and validation

A GPU-enabled external solver does not make the complete OpenFOAM application GPU-accelerated. Verify the distribution, integration, supported boundary conditions, matrix transfers, precision and case shape. The existence of an accelerated linear-algebra backend is not evidence that every part of your CFD workflow runs on the GPU.

This comparison covers OpenFOAM.com’s external PETSc solver module, the third-party OpenFOAM Ginkgo Layer (OGL), and a community bridge to NVIDIA AmgX. They do not have the same support status. Treat each as a version-pinned integration and validate it against a CPU baseline before using it for production results.

OpenFOAM GPU support at a glance

OpenFOAM.com’s external PETSc solver module

The README in the OpenFOAM external-solver repository describes petsc4Foam, an interface between OpenFOAM.com and PETSc. This identifies a specific module and distribution, not universal support across every OpenFOAM fork or release. The module’s existence does not establish GPU compatibility for a particular case.

PETSc has CUDA, HIP, and Kokkos device paths, with backend maturity varying by vector, matrix, solver, and preconditioner. Its current GPU roadmap marks some combinations as supported and others as still in development. The same roadmap recommends using PETSc's main branch for GPU work, so a build that worked with an older release should not be treated as a durable recipe.

Use the module’s requirements and the selected PETSc backend’s documentation to define a testable configuration. A roadmap entry or repository description is not a compatibility certificate. Validate the exact combination before describing it as production-ready.

Route 1: OpenFOAM external solver with PETSc

This route starts with the OpenFOAM external-solver module and a PETSc build configured for the accelerator you intend to use. PETSc's current installation guidance uses options such as --with-cuda or --with-hip, and Kokkos builds add the matching Kokkos packages. Exact OpenFOAM library names and dictionary entries vary by distribution and module version, so copy them from the module shipped for your version rather than from an undated tutorial.

1. Pin the complete stack. Record the OpenFOAM distribution and version, compiler, MPI, PETSc commit or release, accelerator backend, driver, and toolkit.
2. Build a CPU PETSc path first. Confirm the external-solver interface reproduces the native CPU result before adding device-specific behavior.
3. Enable one GPU backend. Follow the current PETSc GPU guidance for its matrix and vector types and runtime options.
4. Inspect data movement. Repeated CPU-to-GPU copies can consume the time saved by the accelerated solve.
5. Test solver and preconditioner combinations. A backend being available does not mean every preconditioner stays on the device or performs well for your matrix.

PETSc provides an external-solver route whose integration must be validated for the selected versions and case. It does not establish that mesh handling, equation assembly, turbulence models, boundary conditions and I/O all run on the GPU.

Route 2: the OpenFOAM Ginkgo Layer

OGL wraps Ginkgo solvers and preconditioners for OpenFOAM. Its documented executors include CUDA, HIP, SYCL, OpenMP and reference execution. Check the requirements and supported case features before selecting one; the executor list alone does not prove portability of your complete workflow.

The limitations matter. OGL documents support for only basic cyclic boundary conditions, with no AMI boundary conditions or block-coupled matrices. It also exposes choices such as whether to update the matrix and initial guess on the device. Those settings change transfer overhead and can change the result of a benchmark.

Use OGL only when your case fits its supported matrix and boundary-condition path and you can maintain the third-party build. Test the exact OpenFOAM and Ginkgo versions together; repository activity alone is not support from your OpenFOAM distributor.

Route 3: NVIDIA AmgX through a community wrapper

NVIDIA AmgX is a GPU-accelerated AMG and linear-solver library. Check the selected release’s build requirements against your CUDA toolkit and GPU. Library support does not establish support for the separate bridge to OpenFOAM.

An issue opened in the NVIDIA AmgX repository on January 22, 2026 asks where to find an official amgx4foam version and links a community wrapper. The issue was open when checked on September 28, 2026. It is a user’s question, not a vendor statement proving that no official integration exists. Verify the provenance and supported versions of the wrapper you choose.

Choose this route only when NVIDIA-only deployment is acceptable and you can review, build, and maintain the wrapper. Validate matrix conversion, MPI behavior, supported OpenFOAM versions, convergence, and restart behavior before relying on it.

When a GPU is likely to help

A GPU backend has the best chance when sparse linear solves consume a large share of the measured runtime and the problem is large enough to amortize setup and transfers. Pressure-heavy incompressible cases can be candidates, but the solver share must be measured. A fixed threshold such as “60–70%” is not universal.

A GPU is less likely to help when the mesh is small, equation assembly or boundary processing dominates, I/O and post-processing dominate, or the selected solver repeatedly moves matrices and vectors between host and device. Cases using unsupported boundary conditions or block coupling may be excluded by the wrapper before performance becomes relevant.

Precision and hardware fit

Start from the numerical requirements, not the GPU name. CFD workloads often use double precision, but the acceptable precision and residual behavior depend on the solver, case, and quantities being reported. Consumer GPUs can execute FP64 while offering much less FP64 throughput than accelerator-class hardware.

Compare CPU and GPU runs using conserved quantities, field values at representative locations, forces or coefficients, residual histories, iteration counts, and final convergence criteria. Our FP64 decision checklist helps separate a genuine double-precision requirement from a default setting.

VRAM also needs headroom for matrix formats, vectors, preconditioners, temporary storage, and runtime workspaces. A mesh fitting in host RAM does not prove the accelerated solver will fit in GPU memory.

A benchmark that answers the deployment question

Use at least one case that represents the intended production workload and several mesh sizes. Warm up the environment, repeat runs, and report the spread rather than one best result.

• Case: solver, physics, boundary conditions, mesh cells, time-step settings, and convergence criteria.
• Software: OpenFOAM distribution and version, wrapper commit, PETSc, Ginkgo, or AmgX version, compiler, MPI, driver, and CUDA or HIP version.
• Hardware: CPU model and threads, GPU model and VRAM, host RAM, storage, and interconnect.
• Timing: initialization, assembly, linear solve, transfers where observable, I/O, and total wall time.
• Numerics: iterations, residual history, convergence status, and differences from the CPU reference.
• Economics: total cost per converged case, including failed builds and tuning runs when comparing deployment options.

Multi-GPU scaling needs its own test. The backend, partitioning, MPI implementation, GPU awareness, interconnect, and communication-to-compute ratio all affect the result. A single-GPU speedup cannot be multiplied by the GPU count.

Build and test the stack on Compute with Hivenet

Compute with Hivenet provides GPU and CPU instances with SSH access, templates and OS images, and per-second billing. It gives you control of the environment; it does not provide a managed OpenFOAM GPU integration.

1. Start with a CPU baseline or the smallest GPU instance with enough VRAM for the planned test.
2. Build OpenFOAM and the selected external solver or wrapper in a versioned environment.
3. Save the build inputs, compiler output, dictionary files, and benchmark script.
4. Run correctness checks before measuring speed.
5. Move to a different GPU only when memory, precision, or measured runtime justifies it.

The wider scientific-modeling GPU fit guide covers the same measure-before-scaling principle across simulation workloads. For a commercial multiphysics example with a narrower supported GPU scope, compare the COMSOL 6.3 GPU support guide.

Troubleshooting

The GPU is visible but utilization stays low

Confirm that the selected matrix and vector types use the device backend, then profile whether the linear solve dominates the run. Host-side assembly, transfers, or an unsupported preconditioner may be the bottleneck.

The wrapper fails to load

Check that the wrapper was compiled against the same OpenFOAM distribution, compiler ABI, MPI, and dependency versions used at runtime. Verify library paths and read the wrapper's version-specific instructions.

The GPU run converges differently

Match tolerances and algorithms first. Compare iteration histories and field-level results with the CPU baseline. Do not accept a faster run that solves a materially different numerical problem.

The case runs out of VRAM

Measure the matrix, preconditioner, vector, and workspace footprint. Reduce the case only for diagnosis; use a larger-memory device or a different solver path for the production comparison.

OpenFOAM GPU decision checklist

• Profile the CPU case and identify the actual dominant phase.
• Choose one integration whose limitations fit the case.
• Pin every software and driver version.
• Reproduce the CPU result before enabling the GPU backend.
• Check boundary-condition, matrix, precision, and multi-GPU support explicitly.
• Measure total wall time and cost per converged case.
• Preserve the environment and benchmark method for repeatability.

Primary sources

• OpenFOAM external-solver repository: petsc4Foam module requirements
• OpenFOAM HPC Technical Committee: background context, not version-specific compatibility evidence
• PETSc GPU support roadmap
• PETSc getting started: GPU backends and data movement
• OpenFOAM Ginkgo Layer documentation
• NVIDIA AmgX releases
• NVIDIA AmgX issue: status of an official amgx4foam bridge