Ten deterministic evaluators
exact_match, contains, contains_all, contains_any, regex, json_valid, non_empty, length_bounds, levenshtein, numeric_tolerance. No model in the loop, so a score is reproducible.
Apache-2.0 · one binary · v0.2.0
A 2xx means it is on disk.
Point your OpenInference or OTel exporter at localhost:4318.
evald normalises each span, fsyncs it to a write-ahead log before
acknowledging, compacts it into time-partitioned Parquet you can query with
DuckDB, and runs your evals against it. One static binary — no database,
no Python runtime, no container.
§01 — read this before the throughput table
Trace stores acknowledge at different points, and that is the whole comparison
A spans-per-second number means nothing on its own, because it is not a measure of the same promise. Here is what a 2xx actually commits to in each store this was benchmarked against.
| store | what a 2xx ACK means |
|---|---|
| evald | the span is fsynced to the WAL before the ACK — crash-safe at ACK |
| Phoenix | request parsed and queued; the insert happens async — spans were still landing 20+ s after load stopped |
| OpenObserve | written to an in-process WAL buffer; fsync is periodic, not per-request |
Method: a Rust OTLP/HTTP load generator, protobuf
ExportTraceServiceRequest, LLM-shaped spans with globally
unique trace and span ids so no target gets an upsert or dedup
shortcut. 10 spans per request, 15 s measured after a 3 s warm-up,
keep-alive connections. “Accepted” means 2xx-ACKed;
429 and 503 sheds are counted separately and never as throughput.
| spans/s accepted, 10 spans/req | 1 conn | 8 conns | 32 conns |
|---|---|---|---|
| evald — group-commit writer, native | 4,732 | 17,348 | 56,029 |
| evald — per-request fsync, before group commit | 4,719 | 5,150 | 5,261 |
| Phoenix — Docker, SQLite default | 1,053 | 160 | 128 |
| OpenObserve — Docker, single node | 18,747 | 103,468 | 248,555 |
OpenObserve wins that row by 4.4×, and we are not going to dim it
It owns raw single-binary OTel ingest throughput and we do not claim that
slice. It is also faster on latency at every concurrency we measured. What
it does not do is fsync before acknowledging — so the two numbers are
answers to different questions. evald's claim is the strongest ACK at
per-core parity, not the biggest number. Against Phoenix, which does shed,
evald ingests roughly 50× more spans per second under concurrency.
§02 — the promise, tested
Killed mid-compaction, everything came back once
The hot-to-cold commit protocol was verified with a real kill -9
during compaction — the worst moment to die, because a segment is
half-moved between the log and the Parquet blocks. Spans come back from cold
Parquet plus WAL replay with no loss and no double-count.
the ACK boundary
fsynced WAL
A span is durable before the client is told it was accepted. Nothing is acknowledged from memory.
then, in the background
Parquet
A compactor flushes sealed segments to time-partitioned Snappy Parquet, with a redb trace_id index and a watermark, then truncates the log.
what you can do with it
SQL
Those blocks are plain columnar files. Query them in DuckDB or pandas — or through evald's own DataFusion endpoint, where spans and scores join in one query.
under overload
429 + Retry-After
Backpressure and shedding, with the sheds counted separately. It refuses work rather than dropping it silently.
# the durability check the benchmark ground rules require after every run
$ curl -s localhost:4318/v1/sql -d '{"sql":"SELECT COUNT(*) FROM spans"}'
# must equal the accepted-span count. If it does not, the run does not count.
§03 — the part nobody else ships in a binary
A regression gate that can tell a real drop from noise
evald eval run scores a JSONL dataset with deterministic
evaluators, persists every per-item and aggregate score, and exits non-zero
on a threshold regression. That much is a normal CI gate. The second gate is
the interesting one.
evald eval compare runA runB reads those persisted aggregates
back and diffs two runs by evaluator. With --fail-on-regression
it fails the build when a run gets worse — and with
--significance that gate becomes statistically honest: a
drop only fails the build when a dependency-free Welch's two-sample
t-test shows the (1 − α) confidence interval
excludes zero.
So sampling noise on a small dataset does not flag a phantom regression, and nobody learns to ignore the gate. The aggregate score carries the per-run n, mean and variance the test needs, which is why this works without a statistics runtime.
Scores are first-class
One universal Score object — eval, human or API — targeting a span, trace, session or run, stored durably. There is a Phoenix-compatible /v1/span_annotations endpoint so existing tooling keeps working.
Keyed to the exact span
A score is not a number in a spreadsheet next to a run id. It points at the span it judged, which is what makes the OTel-native join worth having.
§04 — what it costs to keep running
8.5 MB idle, and one file to copy
The deployment model is the other half of the argument. A store you can put on a CI runner is a different product from one that needs six services.
| idle RSS | under load | CPU | at | |
|---|---|---|---|---|
| evald — native | 8.5 MB | 747 MB peak on a 1M-span burst, 276 MB after compaction | 1.8 cores | 59k spans/s |
| Phoenix | 436 MB | 549 MB and climbing — the queue | ~0.15 core | 726 spans/s |
| OpenObserve | ~170 MB | ~340 MB, flat | 5–9 cores | 245k spans/s |
evald
Copy one static binary. evald serve --data-dir ./data. A laptop, a CI runner, an air-gapped host, a systemd unit. No runtime, no database, no container.
Phoenix
A container, or pip install inside a Python environment. SQLite by default, Postgres for production. Python or a container either way.
OpenObserve
Single binary or container for one node. Production is cluster mode — multiple nodes plus object storage plus etcd or NATS — or their hosted service.
Langfuse
Docker Compose or Helm: web, worker, Postgres, ClickHouse, Redis and MinIO. Or their cloud. Against that, this comparison is six services against one file.
The console is embedded in the binary too — trace list, trace tree,
scores and a SQL console, served at / and working air-gapped.
There is no separate front end to deploy or keep in version lockstep.
§05 — getting it
Five platforms, a signed checksum file and an SBOM
Every v0.2.0 release publishes musl and macOS archives for both
architectures, a Windows zip, an SPDX bill of materials, and a
SHA256SUMS carrying a Sigstore signature and certificate.
# run it
$ evald serve --data-dir ./data # OTLP/HTTP on :4318, console on /
# point anything OTel at it, then gate your build on the result
$ evald eval run --dataset cases.jsonl
$ evald eval compare runA runB --fail-on-regression --significance
# or query the Parquet directly, no server involved
$ evald query "SELECT * FROM spans JOIN scores USING (span_id)"
Channels
$ cargo install evald
$ docker run -p 4318:4318 mancube/evald:v0.2.0
Or take a tarball from the release and verify it against the signed SHA256SUMS.
Both wire formats
The receiver accepts POST /v1/traces in protobuf (gzip-aware) and OTLP-JSON, and normalises each span into one model unifying the OpenInference and gen_ai.* conventions — so you do not have to pick a convention first.
Apache-2.0
Every number here is reproducible from the repository
The benchmark harness ships in bench/ — the same load
generator and the same fixed matrix, pointed at whichever store you want to
check. Including the one that beats us.