//donner/svg/renderer/tests:resvg_test_suite uses https://github.com/RazrFalcon/resvg-test-suite to validate Donner's rendering end-to-end. The test suite provides .svg files that can be rendered with the static subset of SVG (and some SVG2), and resvg's golden images to compare against.
To validate against this suite continuously, https://github.com/jwmcglynn/pixelmatch-cpp17 is used to perceptually difference the images and wrap it in a gtest. Execution is single-threaded but it's fast enough to be run in CI, and sufficiently fast to be run as part of inner loop development.
To run the suite:
bazel run //donner/svg/renderer/tests:resvg_test_suite
Or in debug mode:
bazel run -c dbg //donner/svg/renderer/tests:resvg_test_suite
Since this is a gtest, it will also be run as part of any bazel test targeting this directory:
To run as part of gtest, parameter-driven tests are generated by scanning the suite's category directories and registering one test per .svg file found.
Generated Parity Ledger
Generate a deterministic JSON ledger of every vendored case, active or disabled category, skip/render-only state, pixel budget, custom golden, backend mode, text-full gate, and recorded reason:
bazel run //tools/resvg_parity:parity_report -- --output /tmp/resvg-parity.json
The generator joins the C++ registration table to the vendored corpus and fails closed on duplicate registrations, unregistered categories, missing overrides, or malformed parameter expressions. //tools/resvg_parity:parity_report_tests verifies parser fixtures and the exact live-tree totals in Bazel runfiles. The report distinguishes explicit pixel-budget sources from effective cases after category defaults are applied, and counts the two render-without-compare buckets separately. The same test fails when a case renders without a comparison through anything but Params::UndefinedBehavior or Params::DeprecatedInSVG2, when an undefined-behavior case's reference is not the resvg UB marker image and the case is not in SPEC_UB_ALLOWLIST, or when a deprecated-in-SVG-2 case is not under a feature in DEPRECATED_IN_SVG2_FEATURES (both in parity_report.py; see the policy below).
--head-to-head compares Donner with resvg's own recorded results for the same snapshot, the resvg column of third_party/resvg-test-suite/results.csv:
bazel run //tools/resvg_parity:parity_report -- --head-to-head --output /tmp/resvg-head-to-head.json
It reports both pass counts, the margin (Donner passes minus resvg passes), a strict Donner count (no pixel budget or golden override), the cases Donner passes and resvg does not (with resvg's recorded result), and the gap: cases resvg passes and Donner does not, with their state and reason. A Donner pass is a case that compares, by default or within a recorded budget, so the count assumes the comparison tests are green. Skipped and render-only cases are not passes; neither are resvg's unknown or crashed rows. The results.csv values are manual visual judgments recorded with the snapshot, not a pixel diff, and current upstream resvg may score differently.
Some tests require a more lenient threshold, or must be skipped entirely due to incomplete Donner functionality. To do this, per-test params may be specified. A test registration appears as:
INSTANTIATE_TEST_SUITE_P(
StrokeLinecap, ImageComparisonTestFixture,
Combine(
ValuesIn(getTestsInCategory(
"painting/stroke-linecap",
{
{"zero-length-path-with-round-cap.svg", Params::Skip("Bug: zero-length subpath caps")},
})),
ValuesIn(ActiveComparisonModes())),
TestNameFromFilename);
getTestsInCategory(category, overrides, defaultParams) scans one category directory under the resvg-test-suite tree (e.g. painting/fill, filters/feBlend, text/text) and registers every .svg it finds. overrides is keyed by the bare filename (with extension) and sets per-test Params (skip, threshold, golden override, …); files not listed use defaultParams. Combine(..., ValuesIn(ActiveComparisonModes())) runs each test under every active comparison mode: one mode on CPU builds (TinyGolden) and two on the Geode build (TinyGolden and GeodeGolden). The former GeodeTinyParity pixel-vs-pixel mode is retired; both backends are independently gated against their reference goldens.
The test name is <SuiteName>/ImageComparisonTestFixture.ResvgTest/<sanitized-filename>, where the filename stem has every non-alphanumeric character replaced by _. For the example above:
StrokeLinecap/ImageComparisonTestFixture.ResvgTest/zero_length_path_with_round_cap
On the Geode build (multiple comparison modes) the mode is appended, for example ..._TinyGolden / ..._GeodeGolden.
To run a single test:
bazel run -c dbg //donner/svg/renderer/tests:resvg_test_suite -- \
--gtest_filter="*zero_length_path_with_round_cap"
If a test is skipped, it is still useful to manually run it without editing resvg_test_suite.cc. Params::Skip(...) prefixes the generated gtest name with DISABLED_. Run those tests with gtest's disabled-test flag and a narrow filter; text-full-only tests still require a text-full build.
To run a skipped test:
bazel run -c dbg //donner/svg/renderer/tests:resvg_test_suite -- \
--gtest_filter="*zero_length_path_with_round_cap" --gtest_also_run_disabled_tests
With suffix-matching, the same test identifier can be used.
Triaging Test Failures
When tests fail, follow this systematic approach to triage and document them:
1. Run the Failing Tests
First, identify which tests are failing:
bazel run //donner/svg/renderer/tests:resvg_test_suite -c dbg -- '--gtest_filter=TextFontWeight/*'
Look for tests marked as FAIL and note the pixel difference count. Tests pass if pixel differences are under the threshold (default 100 pixels).
2. Examine the Test Output
When a test fails, the framework provides detailed diagnostic information:
Test Failure Header
[ COMPARE ] .../tests/text/word-spacing/simple-case.svg [TinySkia]: FAIL (8234 pixels differ, with 100 max)
- FAIL: Test failed (vs PASS for success)
- [TinySkia]: which backend rendered this comparison (TinySkia, or a Geode* mode)
- 8234 pixels differ: Number of pixels that don't match between actual and expected
- with 100 max: Threshold for passing (tests pass if pixel diff ≤ 100)
Verbose Rendering Output
When a test fails, it re-renders with verbose logging showing:
Document world from canvas transform: matrix(2.5 0 0 2.5 0 0)
Instantiating SVG id=svg1 #1
Instantiating Text id=text1 #5
Rendering Text id=text1 #5 transform=matrix(2.5 0 0 2.5 0 0)
This shows:
- Canvas transform: The scaling/transform applied to the entire SVG
- Instantiating: Elements being created from the SVG DOM
- Rendering: Elements being drawn to the canvas with their transforms
SVG Source Display
The complete SVG source is printed inline:
SVG Content for simple-case.svg:
---
<svg id="svg1" viewBox="0 0 200 200" xmlns="http://www.w3.org/2000/svg"
font-family="Noto Sans" font-size="48">
<title>word-spacing</title>
<text id="text1" x="30" y="100" word-spacing="10">Two words</text>
</svg>
---
Look for:
- <title>: Describes what the test validates
- Element attributes: Features being tested (e.g., multiple x/y values)
- Complexity: Number of elements and their properties
Output File Paths
On failure, three PNGs are written to $TEST_UNDECLARED_OUTPUTS_DIR (Bazel exposes this under bazel-testlogs/.../test.outputs/), named after the sanitized test:
actual_<name>.png — Donner's output (what the renderer currently produces)
expected_<name>.png — the resvg golden (reference rendering)
diff_<name>.png — per-pixel difference highlight
actual (Donner's output): what the renderer currently produces.
expected (golden reference): the resvg-test-suite PNG for this test, from the bazel runfiles directory, generated by resvg.
diff (visual comparison): highlights where actual and expected differ — red/orange pixels mark differing pixels, colored outlines mark positional differences.
Interpreting Output Files
What to look for:
- Diff image:
- Solid red areas = completely different pixels
- Colored outlines = positional/alignment differences
- Minimal differences = may just need threshold adjustment
- Large differences = missing feature or wrong implementation
- Actual vs Expected:
- Compare side-by-side to understand the failure
- Missing elements = not implemented
- Wrong position = baseline/positioning issue
- Wrong style = font/styling issue
- Whole-shape or per-glyph offset = a real coordinate-space / layout bug, not "AA"
3. Analyze the Failure
Based on the output, determine what's causing the failure:
Check the SVG source (printed in test output):
- Look at the <title> to understand test intent
- Identify which SVG features are being tested
- Note complex attributes or patterns
Compare images:
- Open the diff image to see where differences are
- Compare actual vs expected side-by-side
- Assess the magnitude of differences (pixel count)
Review verbose output:
- Check if elements are being instantiated
- Verify transforms are being applied
- Look for errors or warnings in the render log
4. Categorize the Failure
Common failure categories:
- Not implemented: Feature doesn't exist yet (e.g., <tspan>, writing-mode)
- Undefined behavior: The specification leaves the behavior undefined or implementation-defined, or the vendored reference is the resvg UB marker image (a byte-for-byte copy of third_party/resvg-test-suite/resources/ub.png); Params::UndefinedBehavior(reason) renders without a comparison.
Deprecated in SVG 2: The case exercises a feature SVG 2 removed or deprecated and Donner deliberately does not implement (see docs/unsupported_svg1_features.md); Params::DeprecatedInSVG2(reason) renders without a comparison. A removed feature with a UB marker reference belongs here.
Nothing else renders without a comparison: bugs, missing features, reference disagreements, harness or background mismatches, and API policy differences are Params::Skip with a bug number or reason, and a case that passes the default comparison compares. Skipped cases do not render, so they lose no-crash and sanitizer coverage.
- Bug: Wrong output for an implemented feature — find the root cause (wrong transform, coverage geometry, color space, premultiplication, layer compositing). Note: pixelmatch already excludes anti-aliased edge pixels, so a diff our harness reports is never "just AA" — a diff large enough to fail a test has a real cause (see CLAUDE.md §"Anti-Aliasing Is Never the
Root Cause").
5. Document in resvg_test_suite.cc
Add the failing test to the appropriate INSTANTIATE_TEST_SUITE_P block with a skip comment:
INSTANTIATE_TEST_SUITE_P(
TextWordSpacing, ImageComparisonTestFixture,
Combine(
ValuesIn(getTestsInCategory(
"text/word-spacing",
{
{"simple-case.svg", Params::Skip("Not impl: word-spacing")},
{"negative.svg", Params::Skip("Not impl: word-spacing")},
})),
ValuesIn(ActiveComparisonModes())),
TestNameFromFilename);
Comment format:
- Not impl: <feature> - Feature not yet implemented
- Params::UndefinedBehavior("<reason>") - Undefined behavior or a UB marker reference (renders, no compare). The factory name carries the classification, so the reason only explains it.
- Params::DeprecatedInSVG2("<feature> (removed in SVG 2)") - Removed or deprecated SVG 2 feature (renders, no compare). Add the feature to DEPRECATED_IN_SVG2_FEATURES in tools/resvg_parity/parity_report.py first; //tools/resvg_parity:parity_report_tests fails otherwise.
- Bug: <description> - Known bug
- Params::WithThreshold(t, maxPx) - only after a root-cause investigation, with the reason. Never widen a threshold to absorb a diff you haven't explained (see CLAUDE.md / AGENTS.md).
- Reference rasterizer generation: <cause>; analytic residual N px, supersampled M px - a per-case pixel budget (withMaxPixelsDifferent or WithMaxPixels, default color threshold, no other exception) for a reference drawn with 4x4 supersampled edge coverage where Donner's analytic coverage differs, typically on the 1.25 px crosshair. Allowed only when a 4x4 supersampled render of the case compares within the default allowance (M <= 100), the residual is edge coverage with matching layout, and the budget is the largest residual on macOS and Linux for both CPU text tiers (and Geode where it runs) plus 3% rounded up to 5 pixels. Record what a 1 px text shift does against the budget.
- Reference label glyphs: <cause>; analytic residual N px, supersampled M px - the same form for a reference set whose label glyphs differ beyond coverage (the color-adjust-functions cases), where the residual is confined to the labels and deliberate filter regressions are shown to exceed the budget. parity_report_tests checks the form of both classes; the evidence and per-case numbers are in design doc 0021.
6. Group Related Failures
Within a category's override map, group the entries by reason with a short comment so the gaps read at a glance:
{
{"variable-weight.svg", Params::Skip("Not impl: variable-font weight")},
{"weight-interpolation.svg", Params::Skip("Not impl: variable-font weight")},
{"bolder-keyword.svg", Params::Skip("Bug: synthetic-bold advance width")},
}
Each resvg category is its own getTestsInCategory(...) block, so a feature that shows up in more than one category (e.g. text/font-weight and painting/fill) is tracked once per category, not in a shared list.
7. Verify Skip Configuration
After adding skips, verify tests run correctly:
bazel run //donner/svg/renderer/tests:resvg_test_suite -c dbg -- '--gtest_filter=TextFontWeight/*'
You should see:
- Skipped tests don't run
- Passing tests still pass
- Clear count of passed/skipped tests
Example Triage Workflow
- Run tests
bazel run //donner/svg/renderer/tests:resvg_test_suite -c dbg -- '--gtest_filter=TextFontWeight/*'
- Examine the SVG of the failing test (printed as output)
- Open the diff image to see where differences are
- Identify the cause of the failure. Either fix the root cause in Donner, modify the test parameters in resvg_test_suite.cc, or mark the test skipped to defer resolving the issue while keeping the suite operational.
Tips
- Visual inspection: Always view the diff images to understand the nature of failures
- Magnitude is the tell: hundreds of differing pixels, per-glyph drift, or whole-shape offsets are a real bug — pixelmatch already excludes AA, so the diff is not edge anti-aliasing
- Categorize systematically: Group tests by missing feature for easier tracking
- Keep comments concise: Use the established format from existing tests
MCP Servers
The resvg-test-triage MCP server provides automated test analysis. When available, use it to:
Batch analyze test failures:
result = await mcp.call_tool("batch_triage_tests", {
"test_output": test_output_string
})
Analyze individual tests:
result = await mcp.call_tool("analyze_test_failure", {
"test_name": "e-text-023.svg",
"svg_content": svg_source,
"pixel_diff": 8234
})
Get implementation guidance (NEW):
result = await mcp.call_tool("suggest_implementation_approach", {
"test_name": "e-text-031.svg",
"features": ["writing_mode"],
"category": "text_layout",
"codebase_files": []
})
Find related tests for batch implementation (NEW):
result = await mcp.call_tool("find_related_tests", {
"feature": "writing-mode",
"skip_file_content": resvg_test_suite_cc_content
})
Track feature progress (NEW):
result = await mcp.call_tool("generate_feature_report", {
"category": "e-text",
"test_output": bazel_test_output,
"skip_file_content": resvg_test_suite_cc_content
})
Analyze visual differences (NEW):
result = await mcp.call_tool("analyze_visual_diff", {
"diff_image_path": "/tmp/diff_e-text-031.png",
"actual_image_path": "/tmp/e-text-031.png",
"expected_image_path": "/path/to/resvg-test-suite/png/e-text-031.png"
})
Setup:
- Install: pip install -e tools/mcp-servers/resvg-test-triage
- Configure in MCP settings:
- Claude Code: See tools/mcp-servers/resvg-test-triage/mcp-config-example.json
- VSCode: Add to .vscode/mcp.json (see README for format)
- Use tools during test triage
Benefits:
- Consistent categorization across all tests
- Auto-detection of SVG features being tested
- Batch processing of 50+ test failures
- Properly formatted skip comments
- Vision model analysis with actual, expected, and diff images
- Implementation guidance - suggests files to modify
- Batch opportunities - find all tests for same feature
- Progress tracking - monitor feature completion
- Visual analysis - categorize diff types automatically
See resvg-test-triage README for full documentation.
1.15.0 