Storage

The SQLite index, the content-addressed blob store, sinks, and how to plug in a remote store.

coding.sh keeps two stores with different shapes, plus any number of write-only mirrors. All three are interfaces (src/core/types.ts), chosen by URL scheme in the config.

Index (built)

Small, queryable rows: sessions, events, eval tasks, harness bundles, runs, labels, datasets and experiments. The default is one SQLite file through the built-in node:sqlite, in WAL mode, with an FTS5 table over prompt, reply and tool text for coding.sh search. Writes are idempotent on event id, so re-ingesting the same spool changes nothing.

coding.sh sql "select …" runs one read-only query against it. coding.sh export sessions|events|labels streams rows as JSONL or CSV, on every store. The tables and columns are in Query and export.

Blobs (built)

Large, write-once content: full tool output, raw source payloads, diffs. Event payloads larger than maxInlineBytes (4 KB) move to a blob and the event keeps a reference. Each blob is keyed by the sha256 of its uncompressed bytes, so the same AGENTS.md read in a thousand sessions is stored once.

Blobs are compressed with zstd level 3. One that compresses to under 4 KiB is a row of blobs/pack.db (SQLite); a larger one is blobs/ab/cd/<sha256>.zst, written to a temp file then renamed, so readers never see half a blob.

Config

# ~/.coding.sh/config.toml  (coding.sh config get prints the effective values)
[store]
index = "sqlite://~/.coding.sh/index.db"
blobs = "file://~/.coding.sh/blobs"

[capture]
mode = "full"                 # or "metadata": no prompts, code or tool output
redact = ["secrets"]          # "secrets", "env", or "none"
redactPatterns = []           # extra regexes
exclude = []                  # repo globs never captured
maxInlineBytes = 4096         # larger payloads move to blobs

[retention]
keepSpoolDays = 7
toolOutputDays = 90

A repository can carry its own .coding.sh/config.toml, but because it arrives with git clone it is untrusted: it may only tighten capture (add redaction, exclusions or metadata mode) and adjust snapshot settings. It can never add a sink or move a store, which would let a repository send your sessions somewhere.

Redaction and metadata mode

• Redaction runs before anything is written to the index, blobs or sinks. The built-in secrets rules key on vendor token prefixes and on secret-looking names next to values, so code, paths and hashes pass through.
• capture.mode = "metadata" keeps ids, timings, kinds, counts and usage, and drops prompts, code and tool output.
• A session in which a vendor token was redacted gets the secret:exposed tag: the agent and its model provider saw it, so rotate it. coding.sh sessions --flag secret:exposed lists them.
• The model API proxy redacts each request and response before storing it, and never stores Authorization, x-api-key or cookies.

A value that got past redaction can be removed afterwards. Put it in a file, one value per line, and count the copies first:

coding.sh purge --literal leaked.txt --dry-run   # count the copies
coding.sh purge --literal leaked.txt             # replace them
coding.sh purge --rule 'sk-ant-api03-[A-Za-z0-9_-]{80,}'   # every match of a shape

Every copy is replaced, in the index, blobs, spool and eval runs: with [redacted:purged] in the index and blobs, with asterisks in spool and eval run files. SQLite store only; rotate the secret too. purge has the details, and Storage, sinks and privacy covers every redaction rule.

Sinks (built)

A sink mirrors events to somewhere else. Each sink keeps its own cursor and retries with backoff, so a remote that is down never blocks local capture. Remote sinks default to metadata mode: content leaves your machine only when you set mode = "full".

[[sinks]]
kind = "otlp"                           # OTLP/HTTP JSON traces to {endpoint}/v1/traces
name = "collector"
endpoint = "http://localhost:4318"
mode = "metadata"                       # the default for every sink

[[sinks]]
kind = "http"                           # POSTs {schema, v, mode, events, sessions}
endpoint = "https://relay.example.com/ingest"
mode = "full"
headers = { Authorization = "Bearer ${RELAY_TOKEN}" }   # read from the environment

• otlp: OTLP/HTTP JSON. A session becomes an invoke_agent span, tools execute_tool spans and replies chat spans, with gen_ai.* attributes pinned to semconv 1.37.0. Span ids derive from event ids, so a resent batch de-duplicates.
• http: posts coding.sh/trace JSON as is, for a collector or relay that speaks it.
• Sinks mirror every event local ingest stores, from hooks and from imported transcripts, and events from the OTLP receiver.
• coding.sh serve mirrors every batch it commits to the sinks in its own config, so a team can set one there instead of on each laptop.

Plugging in a remote store

Built-in schemes are sqlite:// and memory:// for the index, and file:// and memory:// for blobs. For any other scheme the CLI imports an npm package by convention:

• coding.sh-store-<scheme> exporting indexStore(url, config) and/or blobStore(url, config);
• coding.sh-sink-<kind> exporting sink(sinkConfig, config).

The package must be resolvable from where coding.sh is installed. A missing package gives an error that names it; a package that fails to load surfaces its own error.

// coding.sh-store-postgres/index.mjs (the real package, abridged)
export function indexStore(url, config) {
  return PgIndexStore.open(url);   // resolves to an IndexStore
}

// coding.sh-sink-kafka/index.mjs (hypothetical package)
export function sink(sinkConfig, config) {
  return { name: sinkConfig.name ?? "kafka", mode: sinkConfig.mode, async write(events, sessions, blobs) { /* … */ return { ok: true }; } };
}

The interfaces a plugin implements, abridged. A test suite in src/store/conformance.ts runs against the built-in stores and can be pointed at a new one.

interface IndexStore {
  readonly kind: string;
  migrate(): Promise<void>;
  close(): Promise<void>;
  putEvents(batch: AgentEvent[]): Promise<{ inserted: number }>;   // idempotent on event.id
  upsertSession(session: Session): Promise<void>;
  getSession(id: string): Promise<Session | undefined>;
  listSessions(filter?: SessionFilter): Promise<Session[]>;
  listEvents(sessionId: string, opts?: { kinds?: string[]; limit?: number; offset?: number }): Promise<AgentEvent[]>;
  search(text: string, opts?: { limit?: number; agent?: string }): Promise<SearchHit[]>;
  getCursor(source: string): Promise<number>;
  setCursor(source: string, offset: number): Promise<void>;
  // plus tasks, harness bundles, runs, labels, referencedBlobs(),
  // and optional datasets, experiments, judges, kv state and compact()
}

interface BlobStore {
  readonly kind: string;
  put(data: Uint8Array | string): Promise<string>;   // returns the sha256; dedupes
  get(ref: string): Promise<Uint8Array | undefined>;
  getText(ref: string): Promise<string | undefined>;
  has(refs: string[]): Promise<boolean[]>;
  list(): AsyncIterable<string>;
  delete(ref: string): Promise<void>;
}

interface Sink {
  readonly name: string;
  readonly mode: "full" | "metadata";
  write(batch: AgentEvent[], sessions: Session[], blobs: BlobStore): Promise<{ ok: boolean; error?: string; retryAfterMs?: number }>;
}

Retention and gc

coding.sh gc deletes spool files older than keepSpoolDays once every sink has them, tool-output blobs older than toolOutputDays (diffs and eval tasks are kept), and unreferenced blobs, and trims the judge cache to its size cap. --compact also runs SQLite VACUUM; --dry-run shows what would go.