Deploy

How to run the engine in a container (Railway today, Kubernetes later) and how to measure performance and load.

This page deploys the repo, not the gem. The reference deployment is a checkout of this same repository — config/ is its composition root, and the Gemfile consumes the engine through the gemspec. An adopter deploys gem install insika (or a Gemfile line) instead.

Image (Docker)

The Dockerfile (multi-stage, YJIT on) serves config.ru under Falcon. The Studio ships with its dist/ built and vendored — no Node in the build. The backend is durable SQLite (WAL) at INSIKA_DB; mount a volume and point it inside.

docker build -t insika .
docker run -p 9292:9292 -v insika-data:/data \
  -e DEEPSEEK_API_KEY=sk-... \
  -e INSIKA_GATEWAY_TOKEN=change-me \
  insika
curl localhost:9292/up      # {"status":"ok"}

The process model

The image boots N Falcon worker processes over one SQLite file (WEB_CONCURRENCY, default 1). That number is a contract input, not a tuning knob: it decides which engine semantics hold cluster-wide and which are per-worker. The default is 1 because the per-worker semantics are the product — the queue modes (collect/steer) and FIFO ordering need one session actor; raise N only with sticky routing per session in front. It is also a Soak precondition: with more than one worker the hourly RSS series is a random process per sample, and a soak refuses to start. The contract:

Everything here describes N workers of one deployment — one graph, replicated. N graphs inside one process is a different contract, and it is Embedding: there each graph owns its own store and credentials, and the host — not the engine — installs the drain.

N workers share one SQLite store. Everything durable — sessions, tasks, checkpoints, outbox, delegations — is cross-process state. Any status transition that hands work to “whoever gets there first” goes through a transactional claim (Store#transaction); a bare read-check-write on a shared status field is a bug by definition.
A session’s live semantics are per-worker. Per-session FIFO ordering, steer, interrupt, pause/cancel and the SSE watch operate on the worker that holds the session’s actor. The engine does not promise them across workers. A deploy that needs those semantics for a session must route that session’s traffic to one worker (sticky routing) — or accept per-worker best-effort. On Railway there is no sticky-routing option at any layer — confirmed against Railway’s own docs, which distribute traffic randomly and explicitly do not support sticky sessions — so N>1 there is not “per-worker best-effort,” it is a guaranteed cross-session leak the first time two requests for the same session land on different workers. See the Railway section below; insika doctor errors on WEB_CONCURRENCY>1 there — unless insika-router, the session-sticky proxy, is in front, which is what makes N>1 safe on Railway too (N local workers behind it).
Recovery is part of boot, in every wiring. Every worker boots through Server::Boot, which runs recovery before the listen. The per-record sweeps (undelivered outbox records, undelivered delegation results) run in every worker — each record carries its own transactional claim (item 1), so at-most-once holds however many workers sweep. The task sweep runs once per boot generation: the sweep’s “orphaned :running” test cannot see a fiber living in a sibling process, so the first worker to claim INSIKA_BOOT_ID (one id per container start, exported by deploy/entrypoint.sh) sweeps and the rest skip. A worker respawned mid-generation skips too — sweeping then would steal its siblings’ live turns; its own orphans wait for the next generation (the next deploy). Without INSIKA_BOOT_ID (single-process runs) every boot sweeps.
Shutdown is a drain, not a kill. On SIGTERM (or SIGINT) a worker stops accepting new turns — a turn that arrives mid-drain stays :queued and the next boot’s recovery replays it — and waits up to INSIKA_DRAIN_TIMEOUT (default 20s) for the in-flight ones. A second signal skips the wait. Whatever the deadline abandons dies :running, and item 3 picks it up at the next boot. The layers above must grant the time: deploy/entrypoint.sh passes Falcon --graceful-stop = drain + 5 (Falcon’s own default is 1s), and the platform’s SIGTERM→SIGKILL buffer must be ≥ drain + 15: with INSIKA_WORKERS > 1 the router keeps serving the replies in flight for up to drain + 10, then exits. On Railway that buffer defaults to 0 — SIGKILL right after SIGTERM, which cancels the whole drain — so set RAILWAY_DEPLOYMENT_DRAINING_SECONDS=35 on the service.
A periodic tick closes the gap between boots. Serving workers run a tick every INSIKA_TICK_INTERVAL (default 60s, 0 disables) as a child of the turn supervisor: it re-drives any outbox record left :pending (each carries its own claim — every worker may drain), and sweeps orphaned tasks without waiting for a deploy — which is what recovers the orphans of a worker respawned mid-generation (item 3’s hole). One worker per window sweeps (a single transactional claim), and only :queued/:running tasks untouched for INSIKA_TICK_STALE_AFTER (default 900s) are candidates: a live turn is bounded by turn_timeout, so anything older cannot be alive. If you raise turn_timeout past it, raise INSIKA_TICK_STALE_AFTER too — the threshold must exceed the largest turn_timeout of the deployment. :waiting/:paused tasks are idle by nature and stay boot recovery’s.

deploy/entrypoint.sh sets WEB_CONCURRENCY next to a pointer to this section; this section is the single source of truth for what changing it means.

Environment variables

Env	Default	Effect
`INSIKA_DB`	`/data/insika.db` (in the image)	durable SQLite path (mount a volume!)
`PORT`	`9292`	HTTP bind port
`WEB_CONCURRENCY`	`1`	number of Falcon worker processes — a contract input, see The process model
`INSIKA_BOOT_ID`	set by `deploy/entrypoint.sh`	boot generation id; the recovery task sweep runs once per id (process model, item 3). Unset = every boot sweeps (single-process default)
`INSIKA_DRAIN_TIMEOUT`	`20`	seconds a stopping worker waits for in-flight turns before abandoning them to the next boot’s recovery (process model, item 4). The entrypoint sizes Falcon’s `--graceful-stop` from it; on Railway also set `RAILWAY_DEPLOYMENT_DRAINING_SECONDS` ≥ drain + 15
`INSIKA_TICK_INTERVAL`	`60`	seconds between tick passes — outbox drain + stale recovery sweep (process model, item 5). `0` disables
`INSIKA_TICK_STALE_AFTER`	`900`	seconds a `:queued`/`:running` task must sit untouched before the tick sweeps it. Must exceed the largest `turn_timeout` of the deployment
`INSIKA_STALL_LOG_MS`	unset (off)	diagnostic: a worker whose reactor does not yield for this many ms logs a `[stall]` line with the stack holding it, once a second. Falcon SIGKILLs a worker stuck 30s; `5000` shows what gets there
`INSIKA_GATEWAY_TOKEN`	falls back to `ADMIN_TOKEN`	Bearer for `/v1/responses` and `/v1/agents` (the API contract)
`ADMIN_TOKEN`	`local-demo`	login token for `/studio` (change in production)
`INSIKA_ENV`	`RACK_ENV`, else `local`	environment name on the Studio banner; `production`/`prod` hides the banner
`DEEPSEEK_API_KEY`	—	provider key. Without it the engine still boots (`/up` green), but turns fail until it is configured (env or Studio → LLM providers) — cloud resilience
`DEEPSEEK_MODEL`	`deepseek-v4-flash`	model
`CONSUMER_INTERNAL_URL`	—	base URL for data-tools calling back a consumer’s internal API (see below)
`INSIKA_EGRESS_HOSTS`	—	outbound host allowlist for data tools (SSRF guard). An MCP server is pinned by it only when its host resolves to a private address
`INSIKA_EGRESS_ALLOW_HTTP` / `_ALLOW_PRIVATE`	off	for `http`/loopback callbacks only (never in cloud). Also opens inbound media fetches (audio/image URLs) to `http`/private targets — a local run serving media over `http://` needs it
`INSIKA_RELAY_TOKEN`	—	mounts the relay channel at `POST /channels/relay/events`, and is the Bearer it requires. Empty = the route does not exist (`404`). See Channels
`INSIKA_RELAY_DELIVER_URL`	—	your callback; the engine POSTs each reply there. Goes through the egress guard
`INSIKA_RELAY_DELIVER_TOKEN`	—	Bearer the engine sends to your callback (optional)
`INSIKA_RELAY_DELIVERY`	`at_end`	how the relay flushes the outbox: `at_end` (one POST) or `progressive` (one POST per balloon — delivery policy)
`INSIKA_WIDGET_ORIGINS`	—	exact-match origins allowed to embed the web widget, comma-separated. No wildcards. Half the switch: with `INSIKA_WIDGET_AGENTS` unset, nothing is mounted (`404`)
`INSIKA_WIDGET_AGENTS`	—	agent ids a widget visitor may address, comma-separated. The other half of the switch. A chat rate limit is also required or the widget answers `503`
`INSIKA_DATABASE_URL`	—	Postgres store (`postgres://…`, Postgres 13+). Set, it replaces `INSIKA_DB` and Litestream is skipped. Needs the `pg` gem. See Postgres store
`INSIKA_DATABASE_POOL`	`10`	Postgres connections per process
`LITESTREAM_REPLICA_URL`	—	enables Litestream (backup/DR). Empty = disabled (default). See below
`LITESTREAM_ENDPOINT`	—	S3-compatible endpoint (R2/MinIO). Empty = AWS S3
`LITESTREAM_REGION`	—	bucket region (AWS: `us-east-1`; R2: `auto`)
`LITESTREAM_ACCESS_KEY_ID` / `LITESTREAM_SECRET_ACCESS_KEY`	—	bucket credentials (read natively by Litestream)

Renamed from HARNESS_* → INSIKA_*. Every engine variable uses the INSIKA_ prefix. The old HARNESS_* names were removed in 1.0 and are no longer read: a leftover one is reported as an error naming its replacement (at boot and in insika doctor). Rename it before upgrading.

The database filename changed too — the image now defaults to INSIKA_DB=/data/insika.db (it was /data/harness.db). Existing volumes are adopted automatically: the container entrypoint renames the old file — with its -wal/-shm siblings, before anything opens it — when the configured path does not exist yet. Nothing to run by hand, no data lost, and a no-op from the second boot on. To keep the old filename instead, point INSIKA_DB at it: the variable is the knob, the image only picks a default. (The adoption lives in deploy/entrypoint.sh, not in the engine — it is deploy baggage, not a runtime behavior.)

Tokens & rotation (keep the two separate!)

There are two secrets with distinct purposes — in production use different values (the API token falling back to ADMIN_TOKEN is a dev convenience only):

ADMIN_TOKEN — the /studio login (cookie auth). This is the operator surface (just you). Rotating it is safe and independent: change it, redeploy, log in with the new value. It does not affect any API consumer.
INSIKA_GATEWAY_TOKEN — the Bearer for /v1/responses and /v1/agents. This is the contract with your API consumers. Rotating it means changing both sides together (or the integration breaks): update the runtime var and each consumer’s token in the same step.

Generate a strong token: ruby -rsecurerandom -e 'puts SecureRandom.hex(24)'.

Strict config and `insika doctor`

Config discipline that rejects unknown keys — no silent schema tolerance. Two parts:

1. Boot gate. On boot, the engine validates the environment against a schema of known keys (Insika::EnvSchema): a wrong type (INSIKA_PORT=abc) or an unknown key in the INSIKA_ namespace (a typo like INSIKA_EGRES_ALLOW_HTTP the runtime would silently ignore). By default it only warns and boots anyway (last-known-good — a rotated key or a typo never takes the whole service down). To refuse boot on any finding, set INSIKA_CONFIG_STRICT=1. Unknown-key detection is scoped to the INSIKA_ prefix; the shared OPENCLAW_, LITESTREAM_, and OTEL_ namespaces are never flagged.

2. bin/insika doctor — on-demand diagnostics. Reads the same durable backend the server uses (INSIKA_DB) without booting the whole app (no provider, no seed) — safe to run against a production volume:

insika doctor            # colored report; exits != 0 on any error
insika doctor --json     # machine-readable (CI / monitoring)
insika doctor --fix      # applies the safe autofixes and re-diagnoses
insika env               # lists known keys + current values (secrets masked)
insika tools:report      # tool audit over the stored traces: never-called
                         # allowlisted tools, error rate > 30%, stale tools, gate refusals —
                         # read-only, the operator removes ([--agent ID] [--days N] [--json])

Checks: env (the schema above), settings schema version (a pending migration → --fix applies it), a missing platform default_model (--fix seeds it from DEEPSEEK_MODEL), durable vs ephemeral backend, LLM provider configured, ADMIN_TOKEN set, data-tool definitions still valid, a stored agent whose declared tool allow/deny list does not name the tool_allowlist policy (the engine repairs it on read, but as stored the list is inert — see Agents), prompt files that hold text rather than a serialized object (a file whose content is a stringified Hash serves a mangled prompt on every turn while looking perfectly healthy — present, non-empty, and the agent still answers), a prompt file that outgrew a prompt (WARN past ~6 000 estimated tokens or 600 lines — the LLM-generated pack shape that costs 20%+ extra tokens per turn for no better instruction-following), in-session compaction enabled with no model slot (WARN — neither compaction.model nor the platform utility_model is set, so it can never run; see Context), eval seeding left on (WARN — evals.seeding opens POST /v1/conversations/:id/seed, which writes a fabricated conversation state under the tenant token; fine while running snapshot evals, off in production — see Evals), provenance and presentation tools without an allowed evidence data tool (WARN — the check only sees stored data declarations; verify any code-tool source), fencing off for agents exposed by relay/widget environment configuration, cache-layer declarations (known volatile providers marked identity are errors; custom identity providers need a byte-stability review), and skill drift — a shared skill whose body names one store, a prompt file routing to a skill the agent cannot load, a broken companion pair, a stale eager: key (see Skills). Settings-schema migrations are explicit — no Studio save silently reinterprets old-shape data.

Data-tool callbacks to a backend — via a tunnel

Data-tools call back a consumer’s internal HTTP API. With the engine in the cloud and your backend on your machine (:3000), expose it over a public https tunnel and point the engine at it:

# in the tool/manifest: base_url = {{env.CONSUMER_INTERNAL_URL}}
CONSUMER_INTERNAL_URL=https://your-tunnel.example.dev
INSIKA_EGRESS_HOSTS=your-tunnel.example.dev

Because the tunnel is public https, the strict egress guard (the default) already allows it — you do not need ALLOW_HTTP/ALLOW_PRIVATE (those are only for a fully-local loop). Restricting INSIKA_EGRESS_HOSTS to the tunnel host is the secure posture. See Security.

Railway

railway.json already configures the Dockerfile builder, startCommand, the /up healthcheck, and a restart policy.

Create the project/service from this repo (builder = Dockerfile).
Volume: mount it at /data (the default INSIKA_DB points there) — without a volume, SQLite is ephemeral and recovery resumes nothing after a redeploy.
Vars: DEEPSEEK_API_KEY, INSIKA_GATEWAY_TOKEN, CONSUMER_INTERNAL_URL, INSIKA_EGRESS_HOSTS. Leave WEB_CONCURRENCY at its default of 1 unless you run insika-router in front. Railway’s own docs say it “does not support sticky sessions” and randomly distributes traffic across replicas/workers — there is no way on this platform to satisfy the “sticky routing per session in front” precondition item 2 of the process model requires, at any layer (Railway replicas or Falcon’s own --count workers within one container). Raising WEB_CONCURRENCY without a router in front is not a throughput knob, it is a guaranteed way to leak a reply across sessions the first time two requests for the same session land on different workers — insika doctor errors on this combination for exactly that reason. insika-router is the fix: N local Falcon workers behind one sticky proxy, still one Railway replica.
The healthcheck hits /up.
Point your consumer at the service’s public URL, with a matching API token (see RUNNING-LOCAL.md).

Kubernetes

Same contract as Railway, for the same reason, verified against ingress-nginx’s own docs: run one pod, WEB_CONCURRENCY=1. Scale vertically (bigger pod) for now.

Why a standard K8s setup does not get you out of this: the session id lives in the JSON body of POST /v1/responses (the user field), not in a header, cookie, or URL segment. That rules out every sticky-routing mechanism K8s gives you for free:

A vanilla Service (ClusterIP) load-balances across pod endpoints with no notion of session at all — same failure mode as Railway’s replicas, one layer down.
Service.spec.sessionAffinity: ClientIP does not help either: the caller is the consumer’s own backend (achei-b2b, a server-to-server call), not the end customer’s browser, so many different customers’ sessions arrive from the same source IP and would collide on the same pod instead of spreading — the opposite of what you want, and it still does not restore a real per-session guarantee.
ingress-nginx’s nginx.ingress.kubernetes.io/upstream-hash-by (the standard sticky/consistent-hash annotation) hashes on nginx variables — headers, cookies, $request_uri, client IP — not on a field parsed out of a POST body. Reaching into the JSON body needs a custom Lua/OpenResty snippet (or an Envoy filter) that parses the request and extracts user before hashing. That is real, unbuilt engineering work, not a K8s annotation to flip.

If real horizontal throughput is ever needed, two paths get there, neither of them “just add replicas”:

Build the body-aware sticky layer above (Lua/Envoy consistent-hash on the user field) in front of N pods, each still WEB_CONCURRENCY=1.
Move the session id into the URL, the way the web widget transport already does (POST /api/widget/sessions/:token/messages carries the session token in the path) — that surface, unlike /v1/responses, is sticky-routable today with a plain upstream-hash-by: $request_uri, no custom scripting required.

Neither is urgent while a single pod’s throughput is enough — this section exists so scaling this deployment does not silently reintroduce the same cross-session leak the Railway incident above already found once.

Backup / DR — Litestream (opt-in, configurable)

A single volume is the one point of total loss between a pilot and production (disk corruption/loss = goodbye conversations + config). Litestream does continuous replication of insika.db (its WAL) to an S3/R2 bucket, without changing databases and without a line of Ruby.

It is off by default and turns on by env — a single-box ephemeral deploy pays nothing; a durable deploy enables it by pointing at a bucket. The trigger is one variable, LITESTREAM_REPLICA_URL:

empty (default): the entrypoint execs Falcon directly. The Litestream binary is never invoked — behavior identical to not having it.
set: on a fresh box the entrypoint restores insika.db from the replica before the app opens it (litestream restore -if-replica-exists; a no-op if the bucket is still empty), then supervises the app (litestream replicate -exec), replicating the WAL continuously and doing a final sync on shutdown (Railway’s SIGTERM).

Enable in production (Railway)

Add the vars (keep the volume at /data):

# AWS S3
LITESTREAM_REPLICA_URL=s3://my-bucket/insika
LITESTREAM_REGION=us-east-1
LITESTREAM_ACCESS_KEY_ID=AKIA...
LITESTREAM_SECRET_ACCESS_KEY=...

# Cloudflare R2 (S3-compatible): same, + endpoint and region=auto
LITESTREAM_REPLICA_URL=s3://my-bucket/insika
LITESTREAM_ENDPOINT=https://<accountid>.r2.cloudflarestorage.com
LITESTREAM_REGION=auto
LITESTREAM_ACCESS_KEY_ID=...
LITESTREAM_SECRET_ACCESS_KEY=...

Credentials are read natively by Litestream (they are not in deploy/litestream.yml, which only references URL/endpoint/region via ${VAR}).

One writer per replica path

Never point two deployments at the same LITESTREAM_REPLICA_URL (two services, or staging and production). Both replicate into the same lineage and a later restore can bring back the wrong database.
A database already on the volume skips the restore and is replicated as it is. Give a new deployment an empty volume, and do not set the replica URL on a box whose volume holds the wrong file — it will be copied over the backup.
Moving a database to another deployment or path: stop replication on the old box first (unset the variable and redeploy), and give the new one a fresh path. If the old box’s local state is stale (LTX file is missing), run litestream reset <db>.
Point-in-time recovery into a separate file, to inspect before swapping it in: litestream ltx -level all <url> lists the history, then litestream restore -txid <id> -o /data/recovered.db <url>.

Restore drill (the real “done” — an untested backup does not count)

The pilot→production gap only closes once a restore has been exercised. Two ways:

1. Local, automated (proves the mechanism, zero credentials): uses the real image + a file:// replica; boots → replicates → deletes the volume → boots a new box → restores → confirms the marker row survived and /up is green.

scripts/litestream-restore-drill.sh      # needs docker, sqlite3, curl
# → [drill] PASS — marker … restored from replica; /up green on the new box.

2. Production (the drill that counts for go-live): against the real bucket.

# a. with the service live and replicating, generate some config/conversation and
#    confirm the replica has files (Litestream 0.5 lists them as LTX files):
litestream ltx -config deploy/litestream.yml "$INSIKA_DB"

# b. boot a NEW box (empty volume) with the same LITESTREAM_* vars → the entrypoint
#    restores on boot. Verify manually in /studio that conversations and config came
#    back. Manual restore alternative:
litestream restore -config deploy/litestream.yml -o /tmp/restored.db "$INSIKA_DB"

Postgres store (optional)

SQLite is the default and the right choice for one box, including several worker processes (INSIKA_WORKERS > 1): their writes share commits (group commit), so they only queue behind the file’s single write lock under sustained bursts of writes. Postgres removes that lock — writers only meet on the keys they share — but every store call becomes a network round trip, and a turn makes many store calls, so on one box it is usually slower per turn, not faster.

Choose Postgres when the store must be shared: several nodes or replicas running the engine, or other systems that query the data directly. Measure your own load before switching for speed alone.

Add gem "pg" to your Gemfile (the reference image already has it and libpq).
Set INSIKA_DATABASE_URL=postgres://user:pass@host:5432/db (Postgres 13+). It replaces INSIKA_DB; Litestream is skipped. INSIKA_DATABASE_POOL sets the connections per process (default 10). A transaction keeps its connection while its turn waits for the reactor, so under many concurrent turns a small pool is where calls queue; raise it, keeping workers × pool under the server’s max_connections (100 by default).
The table (insika_kv) is created on first boot.
Moving an existing SQLite store into an empty database (the copy adds and overwrites keys, it does not remove extra ones): stop the app, then INSIKA_DB=/data/insika.db INSIKA_DATABASE_URL=… bundle exec ruby scripts/store_copy.rb, then boot with INSIKA_DATABASE_URL set. Unsetting it goes back to the SQLite file as it was at the copy.

Backups are Postgres’s job (your provider’s snapshots or point-in-time recovery).

Kubernetes (evolution)

SQLite does not share one file across nodes. Paths forward: a StatefulSet + a PVC per pod + sticky-by-agent routing (shard by tenant), or LiteFS, or the Postgres store, which every pod can share. Litestream (above) for backup/DR of SQLite — orthogonal to topology.

For the session-routing half specifically — as opposed to the storage topology above — see insika-router: N pods behind a headless Service, one insika-router Deployment in front doing the consistent-hash routing ingress-nginx’s own upstream-hash-by cannot (it hashes nginx variables, never a field parsed out of a POST body).

Measuring performance / load

1. SQLite write ceiling (no provider) — `bench_store.rb`

Isolates “can SQLite take multi-process writes?” from LLM noise: N processes hammering writes on the same file (WAL + busy_timeout — the real config).

bundle exec ruby scripts/bench_store.rb 1,2,4,8 3000

Measured (mid-2026, laptop, ~481B payload):

procs	writes/s (aggregate)	p50	p95	max
1	~29.6k	0.02ms	0.04ms	2.4ms
2	~29.9k	0.02ms	0.04ms	59ms
4	~23.7k	0.02ms	0.05ms	336ms
8	~28.4k	0.03ms	0.05ms	539ms

Reading: aggregate throughput stays ~25–30k writes/s regardless of process count (the WAL’s one-writer-at-a-time ceiling), with zero “database is locked” (the busy_timeout absorbs contention into tail latency, not errors), and a microscopic p95. A real turn is provider-bound (seconds) and does a handful of writes → the workload sits ~100× under the ceiling. Empirically, SQLite is not the bottleneck on a single box.

2. End-to-end load (with provider) — `loadtest.rb`

Hits POST /v1/responses (SSE), the production path. Measures TTFB, total, tokens, cache hits, P50/P95, error rate. Runs against local or a remote deployment. See LOADTEST.md.

INSIKA_URL=http://localhost:9292 INSIKA_GATEWAY_TOKEN=xxx \
  bundle exec ruby scripts/loadtest.rb --agents assistant --concurrency 16 --iterations 3

3. Baseline vs multi-worker on one box — `loadtest-local.sh`

Boots Falcon with 1 worker, then N, over the same SQLite, and counts “database is locked” in the logs.

DEEPSEEK_API_KEY=sk-... ./scripts/loadtest-local.sh 4 24