Engineering
A lightweight Session directory for finding runs, with captured traces kept in DuckLake for investigation.
Bill Zuo
Softprobe
Founder
Every coding agent session leaves a record of tool calls, model turns, errors, token usage, and timing. To investigate a failure, teams need to find the right session and inspect what happened.
Our original Sessions list aggregated spans across the selected time window on every request. As trace volume grew, list queries did more work even when the page showed the same number of sessions.
We separated those workloads. A lightweight Postgres directory serves the list; DuckLake stores the captured traces used for investigation. This post explains the design and its tradeoffs.
What Softprobe is optimizing for
For Session capture and review, we care about:
Capture once, retain for review. Captured OTLP traces and logs for agent runs live in a lake-backed store. Evidence is not a temporary cache.
Ask bounded questions. Humans and AI agents triage sessions, open one session, and dig into observations — not build real-time multi-tenant dashboards over every span ever written.
Stay simple. Keep the architecture small: one trace store and one rebuildable Session directory.
We chose a design that reuses our existing storage and job infrastructure.
The UX that broke under growth
Explorer’s Sessions list is a triage surface:
Which sessions ran in this window?
Which agent?
How many steps? Errors? Tokens / cost?
Filter → page → open detail.
Detail is easy in principle: read one session from traces. Cost scales with that session.
List was the expensive path. On the original design, every list load effectively meant:
Aggregate all spans in the time window, group by session_id, then page.
That works for demos. It does not work when a team runs agents all day. List latency and cost started tracking spans in the window, not rows on the page.
So we had a classic product fork:
Temptation | Why it fails Softprobe |
|---|---|
Drop old spans / shorten retention | Sacrifices retained evidence to solve a list-query problem |
Dual-write a lake session_facts table | Two derived truths; compaction and ops get harder |
Push a counter into the UI / Explorer DB | Explorer does not own ingest; multi-instance breaks |
Stand up ClickHouse / Elastic “for lists” | New system, new cost center, new drift |
We wanted list queries to read a small indexed directory rather than aggregate every span in the window. Page size and filter selectivity still matter; the summary table is not a constant-time lookup.
The design we chose
We keep a lightweight Session directory in Postgres. Captured traces remain in DuckLake.
Evidence vs index
Data | Lives in | Role |
|---|---|---|
Span bodies, attrs, events | DuckLake traces | Source of truth |
Triage fields for the list | Postgres session_summary | Derived, rebuildable |
“These sessions changed” | Postgres session_summary_dirty | Short-lived work queue |
Who runs the reducer | Postgres job lease | Coordination across replicas |
The Session directory can be rebuilt from retained traces. Losing the index does not delete those traces. The lake is the product’s memory; the index is a cache with schema.
Why not maintain absolute counters in memory?
Process-local counters cannot remain authoritative across restarts and multiple ingest replicas. We use the trace store as the authority:
All index numbers come from re-aggregating traces over a time scope.
Dirty rows only say which sessions to refresh, not what the counts are.
Late spans trigger another refresh: touch dirty → next reduce replaces the row within the supported time window. No fragile OPEN/FINALIZED state machine in v1.
Batching is part of the product contract
A dirty table that UPSERTs per span would hammer Postgres harder than the list problem we set out to fix.
Ingestion is already batched (collectors + Softprobe soft coalesce). So we lock the contract:
Directory updates are batched after traces are committed, rather than written for every span.
Batching frequency determines how often the directory needs to be refreshed.
Coalesce is not an afterthought. It is how list indexing stays aligned with how the lake already wants to be written.
Reduce as a leased async job, not on the hot path
Ingest must stay fast and boring: write Parquet/DuckLake, mark dirty, return.
The reducer runs on the same async job runner we use for lake maintenance (compaction, metadata), under a Postgres lease that coordinates reducer ownership per tenant.
That gives us:
Coordinated reducer ownership across replicas
No second “session-index microservice”
One ops mental model: jobs + leases
Worker coordination reduces duplicate work, but it does not guarantee exactly-once execution.
And the SQL always carries a mandatory time window, with partition and event-time bounds derived together, so DuckLake can prune Parquet files. A reducer that scans “all time for these session_ids” would recreate the original problem in the background.
The refresh process preserves updates that arrive while it is running.
What this means for users
Before: opening Sessions could feel like querying your entire telemetry window.
After: the list reads the Session directory — filter, page, open — while the captured observation stream remains in the lake.
You get:
Session-list queries that no longer aggregate the trace lake on every page load
Unchanged deep dive into captured prompts, tools, and errors
A rebuildable directory without shortening evidence retention to speed up the list
The tradeoff is freshness: the directory updates asynchronously. If refresh fails, list entries can be stale or missing even though traces were stored. On the Postgres path, an empty directory produces an empty list; it does not silently fall back to a lake scan. Recovery uses bounded rebuilds from retained traces. Very long sessions can exceed the reducer’s supported window, so their summaries may be incomplete until the relevant history is rebuilt.
This design removes repeated trace aggregation from the list path. Query latency still depends on the summary’s size, indexes, filters, and refresh health.
How the design evolved (and what we refused)
Designs do not arrive fully formed. Ours went through several wrong-but-tempting shapes:
Explorer-owned directory — rejected; Explorer is read-only for this path; ingest lives in our telemetry store.
Dual DuckLake session summary table — rejected; one derived store is enough; lake stays evidence-only.
In-memory touch set as source of dirty truth — rejected under multi-replica; dirty must be durable.
Inline reduce on every ingest — rejected; accepting asynchronous list freshness beats putting aggregation on the write path.
Per-span dirty UPSERT — rejected; dirty writes follow lake flush batches.
These decisions keep the Session directory derived from the trace store, without adding another analytics system.
Why this matters beyond one screen
Agent systems are getting denser. More tools, more retries, more tokens per “session.” Teams that aggregate every span for every list request pay for work that session review only needed once, in the background.
The result:
The Session directory handles browsing; DuckLake retains the captured traces for investigation.
If you are building or evaluating agent products and want session review that stays practical as volume grows — without adding a second analytics database just for the list — this is the design we chose.
Try Softprobe
Connect OpenCode and capture agent Sessions.
Explore your Sessions and inspect their details.
Dig into captured traces when a session misbehaves.
Our free plan includes unlimited Session ingestion, unlimited agents, and 365 days of Session history. LangChain support and our first evidence-linked Agent Diagnosis checks are next.
Questions, feedback, or “we hit the same list-vs-lake wall” stories: we want them.