Skip to main content

Trace Channel

The SSE stream carries two kinds of items. Production items are what your user sees: messages, components, statuses, errors. Trace items are observability data: which block ran, the resolved generator prompt, state snapshots, router decisions. Both flow on a single endpoint; trace items are server-side filtered out by default.

What's on each side​

ChannelItem typesWho sees them
Production (default)message, reasoning, tool_output, component, container, source, status, error, state_change, resource_changeEvery client.
Trace (opt-in)block_trace, router_decision, state_snapshotDevTool, opted in via ?include=trace.

Trace items resolve to itemVisibility: { client: false, history: false } based on their item.type (e.g. block_trace, router_decision, state_snapshot). The visibility resolver recognizes these trace types and short-circuits them to hidden.

block_trace follows a three-event lifecycle: item.added (status in_progress, input known), zero or more item.updated patches (connector input, generator bundle, model usage), and item.done (terminal status, output written). When a block is invoked as a tool, both block_trace and tool_output are emitted — the called block's block_trace.output is a ref to the tool_output item, so the result is stored once and surfaced in two places. See Items for the field-level shape.

Subscribing​

// Production stream — what your end-user UI subscribes to.
new EventSource("/api/flows/myFlow/requests/req_abc/stream");

// Trace stream — what the DevTool subscribes to.
new EventSource("/api/flows/myFlow/requests/req_abc/stream?include=trace");

The ?include=trace parameter doesn't bypass any filtering — it widens the production filter to include the four trace types. There's no second SSE endpoint and no separate transport.

Emitting trace items from a block​

Framework auto-emitters use a typed namespace:

ctx.emit.trace.blockTrace(item);      // block lifecycle (in_progress → updated → terminal)
ctx.emit.trace.routerDecision(item);  // router selection
ctx.emit.trace.stateSnapshot(item);   // sequencer state at step boundary

User code rarely calls these directly. They exist so the framework's auto-emission sites flow through one path that resolves trace visibility by item.type and persists to the trace store in one shot.

Trace retention​

Trace events live in stores.traces, a new entry on StoreRegistry that's independent of stores.request. The retention policy that GCs RequestRecords leaves the trace store alone, so the DevTool can replay traces from a completed request even after its request record is gone.

Backends​

Three implementations ship with the framework. createInMemoryStores, createFilesystemStores, and createSQLiteStores each wire a paired trace store automatically — no separate config step.

  • In-memory (createInMemoryTraceStore). Per-request ring buffer. FIFO over distinct request IDs plus a per-request maxBytesPerRequest soft cap to bound heap usage. Events are gone when the process exits.
  • Filesystem (createFilesystemTraceStore). Append-only .ndjson files under {rootDir}/traces/. A _roster.json file records insertion order for FIFO eviction. Survives process restarts. Used by fsdev dev and by kitchen-sink with STORE_TYPE=filesystem.
  • SQLite (createSQLiteTraceStore from @flow-state-dev/store-sqlite). Two tables, trace_events and trace_request_roster, joined by ON DELETE CASCADE so eviction is one row delete. Survives restarts; runs in the same database as the request store.

Filesystem layout for reference:

.fsdev/data/traces/
  _roster.json
  req_abc.ndjson
  req_def.ndjson

Each .ndjson file holds one trace event per line. Filenames are URL-encoded so arbitrary request IDs round-trip safely.

Local development​

fsdev dev and kitchen-sink with STORE_TYPE=filesystem both wire the filesystem trace store. When NODE_ENV=development, the registry factory raises the maxRequests cap to 1000 so a multi-request iteration session doesn't silently evict its own history. Trace data survives fsdev dev restarts: kill the server, run fsdev dev again, open the DevTool against an earlier request — the trace tree replays.

To override the cap explicitly:

import { createFilesystemStores } from "@flow-state-dev/server";

const stores = createFilesystemStores({
  rootDir: ".fsdev/data",
  traceStore: { maxRequests: 200 }
});

The override always wins, in either direction.

Production​

Outside of NODE_ENV=development, all three factories default to maxRequests: 50 — enough to debug a recent failure without unbounded growth. Filesystem and SQLite both survive process restarts; in-memory does not. Pass traceStore: { maxRequests } to widen or narrow the window.

See also​