tracelint

Deterministic · judge-free

Your agent said it worked. The trace says otherwise.

tracelint is a structural linter for tool-calling agents. It reads the trace you already collect and proves the defects a fluent final answer hides — with a CI exit code, and no model in the loop.

$pip install tracelint Get started
Works with Arize Phoenix · Langfuse · LangSmith · OpenAI · OTel GenAI

A sample agent run. Hover any finding to see what tracelint caught.

step 0 user refund order A100 step 2 get_order → 500 error step 3 refund_order → refunded step 5 refund_order → refunded A100 reused ran twice !
The agent refunded an order it never confirmed.

get_order failed with a 500 — but its order id A100 was passed straight into the side-effecting refund_order. The refund ran on data from a call that had already errored.

hard_defectR2b · error_mishandled · fails CI (exit 2)hover the trace to explore all 3 findings

Why it exists

The bugs that never reach the answer.

A fluent final message passes your evals and your LLM judge — while the run underneath already went wrong. These defects are structural, and decidable straight from the trace:

ignored error
A deploy tool returns a 500, and the agent reports success and moves on. The error is right there in the span.
HTTP 200 · declined
A payment comes back {"status":"declined"} over a 200. Nothing technically errored, so exception handling never fires.
double write
A non-idempotent send_email runs twice with identical args. It fires twice; the answer looks perfect.

The tool changes — a charge, a database write, a file delete, a deploy — but the structural bug is the same, and it's decidable straight from the trace.

How it works

It runs after the run, on the trace — never on a model.

Many agent failures aren't a matter of taste; they're decidable from what the agent actually did. tracelint decides them, the same way every time.

01 · INGEST

Read the trace

Point it at the spans you already emit — Phoenix, Langfuse, LangSmith, OpenAI, OTel GenAI — normalized into one canonical schema.

02 · CHECK

Apply the rules

Deterministic checks replay the trace: schema violations, ignored errors, reused failed values, loops, duplicate side effects — each with the exact steps as evidence.

03 · GATE

Return an exit code

A structurally-provable defect exits non-zero and fails the build. Everything else is disclosed, never silently passed.

≠ judge
No LLM in the loop. No prompt, no second model, no stochastic verdict — the same trace always gives the same report.
tiers
Candidate, not verdict. Only structurally-provable defects fail CI; heuristics are shown as candidates with their evidence.
closed
Fails closed. If a check can't run — a field it needs is missing — it says so, with the reason. A clean report means everything was actually checked, not that nothing ran.
coverage
Shows its work. Per-rule "evaluatable / total" says how much of the run was actually verified — not just that nothing fired.
+ evals
Composes with your evals. It doesn't judge answer quality — it owns the structurally-decidable part before the probabilistic interpretation, and runs alongside LLM-judge evals, not instead of them.

The checks

What it catches

Deterministic checks across schema, error handling, provenance, control flow, and side effects. Every finding is filed under one of three tiers — and only the first one fails your build:

hard_defect

Provable from the trace itself. Fails CI — exit 2, the build stops.

hard_event

A certain fact — a tool errored, a side effect repeated. Reported, but doesn't fail CI.

candidate

A heuristic, shown with its evidence for you to judge. Never fails CI on its own.

R1
Schema violationRecorded arguments don't satisfy the tool's declared JSON Schema.
hard_defect
R2a
Tool error eventA tool returned a structured error — or a declared failure_when fired on a 200.
hard_event
R2b
Error mishandledA value from a failed call is reused as an argument to a later side-effecting call.
hard_defect
R3
Hallucinated argumentAn argument that isn't derivable from anything the agent actually observed.
candidate
R4
LoopThe same call repeated with no change in state — excluding legitimate polling.
candidate
R5
Redundant callAn identical call with an identical result and no mutation in between.
candidate
R6
Malformed argumentsTool arguments that aren't well-formed against the call contract.
hard_defect
R7
Unknown toolA call to a tool absent from the registry — its behavior can't be verified.
candidate
R8
Duplicate side effectA non-idempotent side-effecting call repeated with equivalent args after the first succeeded.
hard_event

Works on the traces you have

Bring your own telemetry.

tracelint reads the exports from the tools you already run. One canonical schema underneath, so every rule reaches every source.

Arize Phoenix openinference

Lint the OpenInference spans Phoenix collects — reads get_spans_dataframe() directly.

for t in load_source("spans.json", "openinference"):
    print(render_report(lint_trace(t, default_rules(), reg)))

Langfuse langfuse

Fetch a trace with the SDK and hand it straight to tracelint — object or raw JSON.

trace = langfuse.api.trace.get("id")
print(render_report(lint_langfuse_trace(trace, registry=reg)))

OpenLLMetry / Traceloop otel

The event-list reader understands the OTel GenAI semantic convention out of the box.

tracelint check spans.json --format otel --tools tools.json

OpenAI & LangSmith openai · langsmith

Chat-completion message lists (incl. ShareGPT) and nested LangSmith run trees.

tracelint check run.json --format langsmith --tools tools.json

Custom format? Map it to the canonical Trace schema — messages, tool calls, and paired results — in a few lines, and every rule applies. The tool contract (side_effecting, failure_when, per-field x-value-origin) lives in your tools.json, declared once and never guessed from a name.

In your pipeline

Gate the build on it.

Add it in a few lines. A hard defect fails the run — the same way a failing test does.

# .github/workflows/ci.yml
- uses: AshwinUgale/tracelint@v0.5.0
  with:
    path: spans.json
    format: openinference
    tools: tools.json

# …or the CLI, anywhere
$ tracelint check spans.json \
    --format langfuse --tools tools.json
0
Clean. No hard defects — findings and coverage still reported.
2
Hard defect. A structurally-provable defect. The build fails.
3
Input error. A bad trace or tools file — never a false pass.