Benchmarks · evidence policy
This page contains no performance numbers. That is deliberate.
A benchmark is a claim about the world, and most of the ones you have read were not reproducible. Here is what it takes for a number to appear on this site.
The policy
Targets are not measurements.
We build a GPU database because we believe it is a different tier of hardware for analytics. That belief is not evidence. A performance goal in a plan, an engineer's observation on a development cluster, a number from a historical log — none of that is a measurement we will show you.
A measurement is published only when it is bound to an immutable artifact that anyone with the same hardware can reproduce. The rules below are the same ones our own repository enforces mechanically: a gate refuses to merge unevidenced claims, and every benchmark script emits a structured evidence record that says unverified when the artifacts are missing.
Six requirements
What a published number carries with it.
Artifact
Commit-bound raw output.
Every published measurement is bound to an immutable raw artifact produced by a named commit of the engine: the JSON record, the logs, the binary provenance. Artifacts are never edited after the fact.
Workload
Manifest, not a description.
The dataset (row counts, digests), the exact SQL text, and the expected results live in a versioned manifest. A run whose scanned rows do not match the manifest is invalid, so a short scan cannot pose as a fast one.
State
Cold or warm, said out loud.
Cache state, GPU residency, result caching (always disabled in the benchmark path) and the number of warm-up iterations are part of the record. So is the timing scope: client wall-clock versus engine time, both reported.
Correctness
The right answer, proven.
Row counts and result digests are compared against the manifest for every iteration. A fast wrong answer is not a measurement.
Hardware
Described precisely.
GPU model and count, memory, CPU, RAM, storage and interconnect. Cross-host comparisons say which host produced which number.
Comparisons
Equivalent, or not published.
A cross-engine claim additionally requires a versioned comparator artifact: same data, same SQL semantics, same execution state, same timing scope, same hardware. Without it, the comparison does not exist for us.
Reference hardware
What we develop and measure on.
Our development cluster is built from consumer GPUs on purpose: it is hardware any team can buy. Eight GPUs per node in the reference development cluster.NVIDIA GeForce RTX 4090: 24 GB GDDR6X, 1,008 GB/s memory bandwidth, 16,384 CUDA cores.
When we publish, the record names the exact host, driver and CUDA version. Numbers from the iteration host are relative evidence for our own before/after decisions; only the certified host produces absolute figures.
What we will publish
Methodology first. Then workloads. Then numbers.
Now: this policy, and the shape of the evidence record (per-query latency distribution, wall and engine time, rows scanned versus manifest, GPU count and memory, commit provenance).
Next: the public workload manifests — a billion-row open dataset with a standard query suite, and a video-analytics suite over synthetic sessions — with the tooling to run them.Then: results, one artifact at a time, each with an evidence identifier you can cite. If you are an early-access partner you will see them first, on your own data.
Measure it on your workload instead.
Early-access partners run RayoDB on their own data with our engineers. That is the benchmark that matters.