One Element Short: the ONNX Runtime Web Bug That Kept Our Inpainting Model Off WebGPU

For three weeks we believed our flagship inpainting model could not run on WebGPU. Session creation succeeded, inference blew up inside a Fourier convolution, and the failure was inconsistent enough that we filed it under "GPU driver instability" and pinned the model to WASM. That diagnosis was wrong in every part: the failure was deterministic, it had nothing to do with drivers, and the fix is a one-line change to which build of onnxruntime-web we import. The bug is an off-by-one in the default build's handling of a negative-step Slice — one element out of 31, in the middle of LaMa's fast Fourier convolutions.

Everything below is reproducible from the checked-in lab notes. The measurements come from our own app's inpaint() path on an Apple M4, Chromium 153, ONNX Runtime Web 1.30.0.

The symptom: a kernel error hiding behind a "deadlock"

Our HQ inpainting engine is LaMa. It is the opt-in quality tier behind the fast MI-GAN path, and after part 13 it ships as a 102.8 MiB fp16 weight file with the fp32 original kept as a fallback spec. It ran on WASM at roughly 4 seconds per 512×512 tile — acceptable for an opt-in mode, and part 5 explained why we had pinned it there: WebGPU sessions "deadlocked" on some GPU/driver combinations with no error and no timeout that could rescue them.

When we finally reproduced it in the lab, the error was not silent at all:

[WebGPU] Kernel "[Add] /generator/model/model.5/conv1/ffc/convg2g/Add" failed.
Error: Can't perform binary op on the given tensors

Session creation succeeds. The first run() fails. It failed identically on fp32 and fp16 weights, across ORT Web 1.27, 1.29 and 1.30, in a worker and on the main thread. That reproducibility should have been the tell: this was not a driver, and it was not a race.

Step one: stop trusting the layer name

convg2g is the Fourier branch of a LaMa FFC block: transform → convolution in the frequency domain → inverse transform. The Add sits at the point where the spectral path and the spatial path are summed, so the mismatch was between two shapes, not inside the Add itself.

The useful move was to ask ONNX which shapes it had inferred. Using onnx.shape_inference.infer_shapes we read the value_info entries, walked backwards from the failing Add to collect its ancestors, and exported just that prefix as a small standalone graph with both of the Add's inputs promoted to graph outputs. Now we could print the actual tensor shapes per runtime instead of guessing from the error text:

Context	fu/Slice_21	fu/Concat_23	convg2g/Add
ort.webgpu.min.mjs / ort.webgpu.bundle.min.mjs	[1,192,64,31]	[1,192,64,64]	works
ort.bundle.min.mjs / ort.all.bundle.min.mjs	[1,192,64,30]	[1,192,64,63]	fails
our app, default import, proxy worker	[1,192,64,30]	[1,192,64,63]	fails
our app, onnxruntime-web/webgpu import	[1,192,64,31]	[1,192,64,64]	works

Same weights, same input, same machine. The only variable is the entry point. We had spent hours blaming our worker proxy; the default build fails on the main thread too.

Step two: read the slice

Two Slice nodes feed that branch, and they are mirror images of each other:

Slice_20: starts=[1]  ends=[-1]  axes=[3]  steps=[1]   # 33 → 31  (drop first and last)
Slice_21: starts=[-1] ends=[-9223372036854775807] axes=[3] steps=[-1]   # 31 → 31 (reverse)

Slice_21 reverses a 31-element axis. Its end is INT64_MIN + 1, which is how the exporter writes "go all the way past the start" for a negative step; ONNX clamps that to -1, i.e. "one position before index 0", and every element survives. The reverse of 31 elements is 31 elements.

The default build clamps the same end to 0 instead. Reversing from index 30 down to index 0 inclusive still yields 31 elements, but ORT's implementation treats the clamped bound as exclusive and drops one. Slice_21 returns 30, the Concat on the spatial path becomes 33 + 30 = 63 instead of 64, and the Add receives 64 and 63. Its calcShape() returns null, and the reported error is the shape-op message rather than anything about slicing — which is exactly why the layer name sent us looking in the wrong place.

The fix

onnxruntime-web ships several builds. The package root resolves to ort.bundle.min.mjs, which contains every execution provider — and carries the buggy Slice. The onnxruntime-web/webgpu subpath is the WebGPU-specific build (ort.webgpu.bundle.min.mjs), which implements the clamp correctly:

// apps/web/src/lib/ort-setup.ts
// before: return (ortPromise ??= import('onnxruntime-web'));
export function getOrt(): Promise<Ort> {
  return (ortPromise ??= import('onnxruntime-web/webgpu'));
}

Two consequences that matter for delivery:

• The WebGPU build depends on ort-wasm-simd-threaded.asyncify.wasm (25.5 MB raw, 3.8 MiB after brotli q11), not the jsep.wasm the older docs assumed. We added it to the R2 mirror and to the runtime's warm-up URL, using the incremental setup:r2 upload from an earlier round.
• With the blocker gone, initLama goes back to the default execution-provider chain, ['webgpu', 'wasm'], with WASM kept as the runtime fallback. The watchdog budget drops from 300 s to 60 s, because a healthy tile is under a second now. The 300 s number was a symptom of the misdiagnosis: we sized the timeout for a hang that was really an exception we had not read.

We still pin one platform to WASM on purpose. Apple mobile devices get ['wasm'] unconditionally: WebGPU on mobile Safari is the one target we cannot regression-test in CI, and the WASM path is the known-good one there. Desktop is where the WebGPU win is banked.

What it is worth

Real inference path, 512×512 single tile, fp16 weights, M4:

Backend	Session init	Per-tile run()
WebGPU	5.3 s	0.84–0.86 s
WASM	5.9 s	4.06 s

That is 4.7× on the step users actually wait for. Init is a wash — most of it is shader compilation and weight upload, which is why we pre-warm the session with a dummy forward pass before the first real tile.

Quality is not a trade-off here. The WebGPU output versus the WASM output on the same tile measures 71.6 dB PSNR, with a maximum per-channel difference of 2 out of 255 — that is ORT's own backend-to-backend rounding noise, not a model change. The model-level probe agrees: fp16 on WebGPU differs from WASM by 65 dB, within the same band as the fp32 backend comparison.

The download win from part 13 compounds with this one, and the two together changed the shape of the HQ tier: half the bytes to fetch, and a fifth of the time to run.

Reproduce it

The smallest version of the bug is a single node. Feed a float tensor whose last axis is 31 long through this slice and compare the output length across builds: ONNX semantics say 31, the default build returns 30.

import numpy as np, onnx
from onnx import TensorProto, helper, numpy_helper as nh

init = [
    nh.from_array(np.array([-1], np.int64), "starts"),
    nh.from_array(np.array([-9223372036854775807], np.int64), "ends"),
    nh.from_array(np.array([3], np.int64), "axes"),
    nh.from_array(np.array([-1], np.int64), "steps"),
]
graph = helper.make_graph(
    [helper.make_node("Slice", ["x", "starts", "ends", "axes", "steps"], ["y"])],
    "reverse_last_axis",
    [helper.make_tensor_value_info("x", TensorProto.FLOAT, [1, 1, 1, 31])],
    [helper.make_tensor_value_info("y", TensorProto.FLOAT, [1, 1, 1, 31])],
    initializer=init,
)
model = helper.make_model(graph, opset_imports=[helper.make_opsetid("", 17)])
onnx.save(model, "slice_reverse.onnx")  # run in ORT Web: 31 correct, 30 broken

Our probes are checked in next to the lab notes and run the real models, not a mock:

# execution provider / build comparison, model level
node lab/lama-quant/scripts/probe_webgpu.mjs \
  --model=lab/lama-quant/models/lama_fp16_fu3_prefix.onnx \
  --ep=webgpu --opt=all --angle=metal --dims=1 \
  --ort=ort.bundle.min.mjs      # Slice_21 → 30 (broken)
  --ort=ort.webgpu.min.mjs      # Slice_21 → 31 (correct)

# the app's real inpaint() path, WebGPU vs WASM
node lab/lama-quant/scripts/probe_app_lama.mjs --eps=webgpu,wasm

Two notes for anyone repeating this: headless Chromium needs --use-angle=metal to get a real GPU instead of the software adapter (400 s per run, which will make you think you have reproduced the "deadlock"), and the prefix-subgraph trick — promote the failing node's inputs to graph outputs via shape_inference value_info — generalizes to any shape-op error in any exported model.

What we got wrong

The expensive part of this bug was not the one element. It was the label we put on it. "Deadlocks on certain GPU/driver combinations" is a conclusion that stopped three weeks of investigation: it sounds environmental, it sounds unmeasurable, and it excuses the absence of a reproduction. The actual failure was deterministic and gave us a tensor shape we could read.

Three specific lessons:

1. An unactionable error message is a bug you have not localized yet. Can't perform binary op on the given tensors names no tensor. Dumping shapes from a prefix subgraph took an afternoon and ended the investigation.
2. A watchdog that fires is not evidence of a hang. Our 300 s timeout turned an exception into an apparent deadlock story. We now log the shape dump and the provider on the first failure, before any fallback runs.
3. Package entry points are part of the dependency. onnxruntime-web and onnxruntime-web/webgpu are different code with the same version number. "Same version" is not "same kernel implementations."

We are preparing a minimal upstream issue for the default build's negative-step Slice clamp; it is present in 1.30.0 and affects any model that reverses an axis with an extreme end value, not just LaMa. Until that ships, the subpath import is the fix.

Try it

The engine described here is live, free, and entirely client-side. clearpix.org/remove-watermark runs the dual-engine MI-GAN/LaMa pipeline — the HQ tier now on WebGPU in Chromium-based desktop browsers, with WASM as the fallback everywhere else. Everything runs in your browser; no image ever leaves your machine.

---

Part 14 of the ClearPix engineering series — how we build free, private, in-browser media tools at clearpix.org.