Deterministic Replay Testing

Overview

The editor renders on a background AsyncRenderer worker thread while the UI thread runs EditorShell::runFrame(). Because a worker render lands whenever it finishes relative to wall-clock, a test that replays a recorded .rnr session and compares specific frames can pass or fail by luck: the frame at which a render becomes visible is render_wall_clock / frame_interval, which varies with machine and load.

The deterministic replay framework makes the GL replay harness reproducible by turning "when a worker result becomes visible" into a function of frame index rather than wall-clock. It provides replay-only worker scheduling modes, a content-only capture mode that excludes editor chrome, and a worker-delay injection hook for proving that a test would catch a timing regression.

Guarantees callers can rely on:

• Production behavior is unchanged at defaults. Every knob defaults to today's behavior (Realtime scheduling, zero delay, chrome on). The real worker still runs free against wall-clock; the deterministic paths are test-only.
• Deterministic captures. Under DrainEachFrame with contentOnlyCapture, a replayed .rnr produces byte-identical capture PNGs and a stable canonical diagnostics projection regardless of pace, injected worker delay, or host CPU load.
• Both worker states are reproducible. DrainEachFrame models a caught-up worker (each render lands one frame after its request); HoldFramesBehind(n) models a worker held a fixed number of frames behind.
• Scheduling windows run on recorded time. In both deterministic modes the render coordinator's scheduling windows (canvas-commit throttle, raster-settle window, pixel-capture commit wake, nothing-to-present retry pacing) read each frame's recorded timestampSeconds, so they close on the same frame however fast the host runs the replay.
• All waits are bounded. A stuck worker fails the replay with the blocking worker state; it can never hang CI.

Architecture Snapshot

The dominant source of nondeterminism is when a worker render becomes visible to the synchronous replay loop. EditorShell::runFrame() polls completed renders at the top (RenderCoordinator::pollRenderResult) and may request a new render near the bottom (maybeRequestRender).

• DrainEachFrame. Before a frame reaches pollRenderResult, the harness waits until AsyncRenderer::hasRenderInFlightForTesting() is false. A staged DoneState result is not in flight, so it is left for the normal poll to consume. Every render then lands exactly one frame after its request, and UI-thread document reads happen after the worker has released document access.
• HoldFramesBehind(n). Installed at the poll seam (AsyncRenderer::setReplayResultHoldFramesForTesting) rather than in the harness, because runFrame() already owns pollRenderResult. A completed result is withheld for n poll attempts; during that window the editor's existing isBusy() checks observe the worker as busy, exactly as a slow render would. Releasing the gate lands the old result on a deterministic frame.
• Recorded-time scheduling clock. In the deterministic modes the harness installs EditorShell::setSchedulingClockForReplay before the first frame and sets it to each frame's recorded timestampSeconds. Without it, an unpaced replay crosses the 120 ms canvas-commit throttle and raster-settle window on whichever frame the host reaches in time, so a faster host posts the same render a frame later. Realtime keeps the steady clock.
• Content-only capture. EditorShell::setContentOnlyCaptureForNextFrameForReplay suppresses non-document render-pane presentation for the readback frame (overlay texture, frame graph, selection/reference chips, pen preview, tool palette, context-menu affordances). It does not clear the overlay texture or skip overlay rasterization, so the next normal frame is unaffected.
• Worker-delay injection. AsyncRenderer::setReplayRenderDelayForTesting(delay) makes the worker sleep a fixed duration per render, outside any mutex-held region. It exists so a deterministic test can demonstrate that it would catch the timing bug; the suite keeps no permanent "realtime must fail" assertion.

A separate non-GL harness (donner/editor/tests/RnrReplay_tests.cc) is deterministic by construction: it reimplements a simplified loop where SyncCanvasSizeDebounced keys the 120 ms canvas-commit throttle on each frame's recorded timestampSeconds (a virtual clock) and drives the worker synchronously. The GL harness cannot reuse that simplified loop because its purpose is fidelity to the real runFrame() and GL texture path; this framework brings equivalent determinism to the real loop.

API Surface

Replay-and-capture entry point (donner/editor/repro/GlRnrReplay.h):

bool RunGlRnrReplay(const GlRnrReplayOptions& options, GlRnrReplayResult* result,
                    std::string* error);

GlRnrReplayOptions (deterministic-replay fields; defaults reproduce production behavior):

Field	Default	Meaning
workerScheduling	Realtime	Realtime / DrainEachFrame / HoldFramesBehind
holdFramesBehind	0	Frames to withhold each result; requires HoldFramesBehind
workerRenderDelayMsForTesting	0	Fixed per-render worker delay (ms)
contentOnlyCapture	false	Suppress non-document chrome on capture frames
cropMode	Full	Full / RenderPane / DocumentCanvas
pace	true	Pace replay by recorded timestamps

Invalid combinations are rejected by RunGlRnrReplay (returns false with an error): a negative holdFramesBehind or workerRenderDelayMsForTesting, or a non-zero holdFramesBehind without HoldFramesBehind scheduling.

GlRnrReplayResult carries captures, per-frame frameDiagnostics (GlRnrReplayFrameDiagnostics, including the scheduling mode, injected delay, held-frame count, and whether a result was withheld), and finalSelectedElementLabel.

Worker-state hooks (donner/editor/AsyncRenderer.h, all test-only, no-ops at defaults):

bool hasRenderInFlightForTesting() const;              // Rendering/Cancelling only, not Done
bool waitUntilNoRenderInFlightForTesting(std::chrono::steady_clock::time_point deadline);
void setReplayRenderDelayForTesting(std::chrono::milliseconds delay);  // negative clamped to 0
void setReplayResultHoldFramesForTesting(int frameCount);
std::uint64_t replayResultHoldPollCountForTesting() const;

hasRenderInFlightForTesting() is deliberately narrower than isBusy(): a staged DoneState result is busy for normal request scheduling but is not in flight, so a drain that waited on isBusy() would hang forever on a staged result.

Testing and Observability

• //donner/editor/tests:async_renderer_tests. Pins the Rendering/Cancelling/Done distinctions, default-zero delay, and the hold semantics.
• //donner/editor/tests:gl_rnr_replay_tests. The end-to-end GL replay coverage. DrainEachFrameContentCaptureIsDeterministicAcrossPaceAndDelay runs a focused fixture over the {pace on/off} x {0,5,10,20,50 ms} matrix and asserts byte-identical captures plus a stable diagnostics projection; ContentOnlyDocumentCanvasCaptureMatchesRendererGroundTruth checks content-only readback against a cropped svg::Renderer::draw ground truth; HoldFramesBehindRecordsWithheldReplayDiagnostics pins the hold semantics.
• //donner/editor/tests:editor_layer_stress_tests. ConcurrentDom safety: busy-gated clicks, cancellation, structural remap, and UI reads under deterministic worker control.
• //donner/editor/tests:rnr_replay_tests. The non-GL deterministic harness.

Capture comparisons use bitmap_golden_compare + pixelmatch identity params, not percentage thresholds. Diagnostics comparisons use the CanonicalReplayDiagnostics test helper, which omits fields that are per-process identity rather than replay invariants (backend texture handles, worker wall-clock durations) so they cannot cause false flakes; the raw diagnostics remain available for failure output.

Maintainer Checklist

• Keep the deterministic knobs test-only. Defaults must reproduce production behavior; the real worker runs free against wall-clock.
• Every replay wait must be bounded by a deadline and report the worker state that blocked it.
• The deterministic modes may change only when staged results become visible and which clock the scheduling windows read (recorded frame time); they must not otherwise change which frames request renders or the drag/presentation state machine.
• Synthesized multi-frame replays must advance timestampSeconds; a constant timestamp never closes a scheduling window.
• HoldFramesBehind must preserve maybeRequestRender semantics: a withheld result looks exactly like a slow in-flight render to the editor.
• Content-only capture is a presenter/readback mode: it must not clear overlay textures, skip overlay rasterization, or perturb the next non-capture frame.
• Do not add a permanent test that asserts the realtime path is flaky; the red-then-green evidence lives in commit history or a local witness run.

Limitations and Future Extensions

• This is not a general virtual clock for the whole editor. It controls the dominant nondeterministic input (worker landing) and the render coordinator's scheduling windows. ImGui frame time already comes from each frame's recorded deltaMs; other wall-clock reads (cost timing, save deadlines) stay real.
• Geode GPU replay targets require a working GPU; they are validated on CI (one Intel Arc + Mesa configuration hits an environmental failure, issue #542).
• Recording per-render virtual durations into the .rnr format is deferred unless the synthetic scheduling modes prove unfaithful.