Comparisons

Comparisons detect correctness drift between endpoints running the same chain.

Comparison model

A comparison run usually includes:

one reference endpoint
one or more candidate endpoints
optional baseline context from previous runs

Outputs are grouped into verdict categories.

Verdicts

Each comparison produces one of three verdicts:

pass: candidate responses match reference within configured tolerance
warn: non-critical differences detected (e.g. latency approaching threshold)
fail: significant mismatch in fields, behavior, or performance

The verdict endpoint provides a CI-friendly exit_code (0 = pass, 1 = fail).

Verdict checks

Each comparison run evaluates up to four independent checks. The overall verdict is the worst status across all checks.

Check	What it evaluates	When it runs
`latency_p95`	Absolute p95 latency against threshold	Always
`latency_vs_reference`	Candidate p95 vs reference p95	When reference endpoint exists
`correctness`	Response match rate between candidate and reference	When reference exists and `correctness_enabled` is true
`latency_vs_baseline`	Candidate p95 vs previous baseline run	When baseline run exists

Each check produces pass, warn, or fail independently. A check warns at 90% of its threshold and fails when exceeded.

Verdict vs run status

These are independent signals:

Run status answers “Did the test execute successfully?”
Verdict answers “How does the endpoint perform compared to expectations?”

A run can be success with a fail verdict (endpoint works but regressed), or failed with no verdict at all (test crashed before comparison).

What gets compared

Template-dependent examples:

block number consistency
RPC method response shapes
transaction receipt field agreement
latency deltas between reference and candidate

Thresholds

Comparison thresholds control pass/warn/fail boundaries:

Threshold	Default	Description
`latency_p95_max_ms`	700	Absolute p95 latency ceiling
`latency_regression_max_pct`	20	Max regression % vs baseline
`latency_vs_reference_max_pct`	50	Max delta % vs reference endpoint
`correctness_min_pct`	100	Minimum response match rate
`correctness_enabled`	true	Whether to run the correctness check

Thresholds can be configured at the suite level and overridden per method via method_overrides.

When to use comparisons

Comparisons are useful in several scenarios:

client diversity checks (reth vs nethermind, geth vs besu)
provider parity checks
canary validation before traffic shift
post-upgrade validation

See Multi-Endpoint Comparison and Upgrade Testing.

Reference endpoint strategy

Choose reference endpoints that are:

stable and well-understood
representative of expected production behavior
isolated from experimental changes

Avoid switching reference endpoint frequently, or trend analysis becomes noisy.

Comparison hygiene

Keep comparison suites focused and maintainable:

keep candidate set small per suite
compare similar endpoint classes together
separate performance and correctness suites when possible
track threshold changes in version control

Triage workflow for `warn` and `fail`

Confirm reference endpoint health first.
Check which specific checks failed in the verdict.
Filter run samples by method and failing candidate endpoint.
Identify whether mismatch is deterministic across retries.
Separate semantic mismatches (field/value) from timing-related drift.
Consider acknowledging known-benign differences.
Decide whether threshold tuning is justified or behavior is a true regression.

Example mismatch categories

blockNumber lag: candidate consistently behind reference
null vs object: method output shape divergence
field formatting: hex formatting or quantity normalization mismatch
revert payload: differing error object/message structures

Mitigation options

remove unstable endpoint from production candidate set
pin client version and retest
lower request concurrency in suite parameters
split one large compare suite into focused method-specific suites
acknowledge known-benign mismatches so they stop triggering failures