APACHE-2.0

OPEN-SOURCE STORAGE

Open storage built for long agent runs

Open storage built for long agent runs

Open storage built for long agent runs

thelake writes complete traces and HTTP bodies to Zstd-compressed Parquet on object storage, with a small Postgres index so session search stays fast.

thelake writes complete traces and HTTP bodies to Zstd-compressed Parquet on object storage, with a small Postgres index so session search stays fast.

View on GitHub
View on GitHub
Start free on Softprobe
Start free on Softprobe
thelake architecture·github.com/softprobe/thelake
Apache-2.0 · Rust + DuckLake
01 · CAPTURE 02 · THELAKE (APACHE-2.0) 03 · OPEN ACCESS async reduce AGENT TRACES Steps, tools & logs OTLP HTTP :8090 · gRPC :4317 BACKEND HTTP Full API payloads Linked via W3C traceparent RUST OTLP Arrow commit DERIVED INDEX Session Directory Postgres session_summary DURABLE STORE DuckLake + Parquet Zstd · S3 / GCS / EU VPC Softprobe Explorer Fast session list + full trace view DuckDB SQL & Grafana make duckdb-shell · Loki & Tempo Scores & CI Replay traces, logs, and scores tables

WHY WE BUILT THELAKE

Traditional trace databases break on multi-turn agents

Traditional trace databases break on multi-turn agents

Traditional trace databases break on multi-turn agents

Multi-turn runs carry long prompts, tool outputs, and raw HTTP bodies. Putting all of that in a hot search database gets expensive fast, so most tools cut retention to 30 days.

Multi-turn runs carry long prompts, tool outputs, and raw HTTP bodies. Putting all of that in a hot search database gets expensive fast, so most tools cut retention to 30 days.

01

Built for full payloads

Traces land in Zstd-compressed Parquet sorted by session_id and timestamp on object storage, keeping 365-day capture free on Softprobe Cloud.

02

Fast session listing

A rebuildable Postgres session_summary directory powers search and pagination without scanning raw Parquet files on every page load.

03

Open format, your files

Read the Rust code on GitHub, query traces directly with DuckDB SQL or Grafana Loki/Tempo routes, or run thelake inside your own VPC.

BUILT FOR ENGINEERS

Query directly, ingest via OTLP, or self-host

Query directly, ingest via OTLP, or self-host

Query directly, ingest via OTLP, or self-host

Standard protocols on the way in, open Parquet and SQL on the way out.

Standard protocols on the way in, open Parquet and SQL on the way out.

query_retries.sql
-- Query traces directly via make duckdb-shell (DuckLake)
SELECT
  session_id,
  COUNT(*) AS charge_calls,
  MAX(http_response_status_code) AS last_status
FROM traces
WHERE http_request_method = 'POST'
  AND http_request_path = '/api/charges'
  AND timestamp >= NOW() - INTERVAL '7 days'
  AND session_id <> ''
GROUP BY session_id
HAVING COUNT(*) > 1
ORDER BY charge_calls DESC
LIMIT 25;

STORAGE ARCHITECTURE

Why full traces stay practical to keep

Why full traces stay practical to keep

Why full traces stay practical to keep

Traditional observability stacks are built for short incident windows. thelake is built for complete session history.

Traditional observability stacks are built for short incident windows. thelake is built for complete session history.

DESIGN CHOICE

TRADITIONAL TRACE DATABASES

THELAKE (DUCKLAKE + PARQUET)

Storage layer

Hot SSD and indexed search clusters

Zstd-compressed Parquet on S3, GCS, or local disk

Large payloads

Sampled out or truncated to protect cluster cost

Full prompts, tool calls, and HTTP bodies kept intact

Session listing

Aggregates raw spans across the time window

Rebuildable Postgres directory (session_summary)

Query & schema

Proprietary query syntax and fixed index fields

DuckDB SQL, MAP attributes, and typed column promotion

Retention & price

Metered per gigabyte; typically capped at 14 to 30 days

365 days free on Softprobe Cloud, or self-hosted in your VPC

Written in Rust on Apache Arrow, DuckDB, and DuckLake (Apache-2.0).

DEPLOYMENT MODELS

Start in two minutes, or keep data in your VPC

Start in two minutes, or keep data in your VPC

Start in two minutes, or keep data in your VPC

Start on our free hosted cloud, or run the exact same engine inside your own VPC.

Start on our free hosted cloud, or run the exact same engine inside your own VPC.

Softprobe Cloud

$0 · FREE FOREVER

Hosted thelake storage wired straight into Softprobe Explorer, Fix Briefs, and Softprobe MCP.

✓

Unlimited session and backend HTTP capture

✓

365-day session history included

✓

1-prompt setup (OpenCode, LangChain, Vercel)

✓

Instant Explorer access with nothing to manage

Start free

Enterprise BYOC

YOUR OWN VPC

Run thelake in your own AWS, GCP, Azure, or EU cloud while using Softprobe Explorer and QA checks.

✓

Parquet files stay in your own S3 or GCS buckets

✓

Prompts, tool calls, and HTTP bodies never leave your VPC

✓

Query DuckLake directly with DuckDB inside your network

✓

Custom retention, SSO/RBAC, and enterprise SLAs

Book a technical demo

APACHE-2.0 OPEN SOURCE

Own every byte of your agent history

Own every byte of your agent history

Own every byte of your agent history

Inspect thelake on GitHub or start capturing production runs for free today.

Inspect thelake on GitHub or start capturing production runs for free today.

View on GitHub
View on GitHub
Start free
Start free
Softprobe

Capture, inspect, and fix production AI agents.

© 2026 Softprobe, Inc. All rights reserved.

Softprobe

Capture, inspect, and fix production AI agents.

© 2026 Softprobe, Inc. All rights reserved.

Softprobe

Capture, inspect, and fix production AI agents.

© 2026 Softprobe, Inc. All rights reserved.