Deploy

How to run the engine in a container (Railway today, Kubernetes later) and how to measure performance and load.

This page deploys the repo, not the gem. The reference deployment is a checkout of this same repository — config/ is its composition root, and the Gemfile consumes the engine through the gemspec. An adopter deploys gem install insika (or a Gemfile line) instead.

Image (Docker)

The Dockerfile (multi-stage, YJIT on) serves config.ru under Falcon. The Studio ships with its dist/ built and vendored — no Node in the build. The backend is durable SQLite (WAL) at INSIKA_DB; mount a volume and point it inside.

docker build -t insika .
docker run -p 9292:9292 -v insika-data:/data \
  -e DEEPSEEK_API_KEY=sk-... \
  -e OPENCLAW_GATEWAY_TOKEN=change-me \
  insika
curl localhost:9292/up      # {"status":"ok"}

The process model

The image boots N Falcon worker processes over one SQLite file (WEB_CONCURRENCY, default 1). That number is a contract input, not a tuning knob: it decides which engine semantics hold cluster-wide and which are per-worker. The default is 1 because the per-worker semantics are the product — the RFC-0015 queue modes (collect/steer) and FIFO ordering need one session actor; raise N only with sticky routing per session in front. The contract:

Everything here describes N workers of one deployment — one graph, replicated. N graphs inside one process is a different contract, and it is Embedding: there each graph owns its own store and credentials, and the host — not the engine — installs the drain.

  1. N workers share one SQLite store. Everything durable — sessions, tasks, checkpoints, outbox, delegations — is cross-process state. Any status transition that hands work to “whoever gets there first” goes through a transactional claim (Store#transaction); a bare read-check-write on a shared status field is a bug by definition.
  2. A session’s live semantics are per-worker. Per-session FIFO ordering, steer, interrupt, pause/cancel and the SSE watch operate on the worker that holds the session’s actor. The engine does not promise them across workers. A deploy that needs those semantics for a session must route that session’s traffic to one worker (sticky routing) — or accept per-worker best-effort.
  3. Recovery is part of boot, in every wiring. Every worker boots through Server::Boot, which runs recovery before the listen. The per-record sweeps (undelivered outbox records, undelivered delegation results) run in every worker — each record carries its own transactional claim (item 1), so at-most-once holds however many workers sweep. The task sweep runs once per boot generation: the sweep’s “orphaned :running” test cannot see a fiber living in a sibling process, so the first worker to claim INSIKA_BOOT_ID (one id per container start, exported by deploy/entrypoint.sh) sweeps and the rest skip. A worker respawned mid-generation skips too — sweeping then would steal its siblings’ live turns; its own orphans wait for the next generation (the next deploy). Without INSIKA_BOOT_ID (single-process runs) every boot sweeps.
  4. Shutdown is a drain, not a kill. On SIGTERM (or SIGINT) a worker stops accepting new turns — a turn that arrives mid-drain stays :queued and the next boot’s recovery replays it — and waits up to INSIKA_DRAIN_TIMEOUT (default 20s) for the in-flight ones. A second signal skips the wait. Whatever the deadline abandons dies :running, and item 3 picks it up at the next boot. The layers above must grant the time: deploy/entrypoint.sh passes Falcon --graceful-stop = drain + 5 (Falcon’s own default is 1s), and the platform’s SIGTERM→SIGKILL buffer must be ≥ drain + 10. On Railway that buffer defaults to 0 — SIGKILL right after SIGTERM, which cancels the whole drain — so set RAILWAY_DEPLOYMENT_DRAINING_SECONDS=30 on the service.
  5. A periodic tick closes the gap between boots. Serving workers run a tick every INSIKA_TICK_INTERVAL (default 60s, 0 disables) as a child of the turn supervisor: it re-drives any outbox record left :pending (each carries its own claim — every worker may drain), and sweeps orphaned tasks without waiting for a deploy — which is what recovers the orphans of a worker respawned mid-generation (item 3’s hole). One worker per window sweeps (a single transactional claim), and only :queued/:running tasks untouched for INSIKA_TICK_STALE_AFTER (default 900s) are candidates: a live turn is bounded by turn_timeout, so anything older cannot be alive. If you raise turn_timeout past it, raise INSIKA_TICK_STALE_AFTER too — the threshold must exceed the largest turn_timeout of the deployment. :waiting/:paused tasks are idle by nature and stay boot recovery’s.

deploy/entrypoint.sh sets WEB_CONCURRENCY next to a pointer to this section; this section is the single source of truth for what changing it means.

Environment variables

Env Default Effect
INSIKA_DB /data/insika.db (in the image) durable SQLite path (mount a volume!)
PORT 9292 HTTP bind port
WEB_CONCURRENCY 1 number of Falcon worker processes — a contract input, see The process model
INSIKA_BOOT_ID set by deploy/entrypoint.sh boot generation id; the recovery task sweep runs once per id (process model, item 3). Unset = every boot sweeps (single-process default)
INSIKA_DRAIN_TIMEOUT 20 seconds a stopping worker waits for in-flight turns before abandoning them to the next boot’s recovery (process model, item 4). The entrypoint sizes Falcon’s --graceful-stop from it; on Railway also set RAILWAY_DEPLOYMENT_DRAINING_SECONDS ≥ drain + 10
INSIKA_TICK_INTERVAL 60 seconds between tick passes — outbox drain + stale recovery sweep (process model, item 5). 0 disables
INSIKA_TICK_STALE_AFTER 900 seconds a :queued/:running task must sit untouched before the tick sweeps it. Must exceed the largest turn_timeout of the deployment
OPENCLAW_GATEWAY_TOKEN falls back to ADMIN_TOKEN Bearer for /v1/responses and /v1/agents (the API contract)
ADMIN_TOKEN local-demo login token for /studio (change in production)
DEEPSEEK_API_KEY — provider key. Without it the engine still boots (/up green), but turns fail until it is configured (env or Studio → LLM providers) — cloud resilience
DEEPSEEK_MODEL deepseek-chat model
ACHEI_INTERNAL_URL — base URL for data-tools calling back a consumer’s internal API (see below)
INSIKA_EGRESS_HOSTS — outbound host allowlist (SSRF guard)
INSIKA_EGRESS_ALLOW_HTTP / _ALLOW_PRIVATE off for http/loopback callbacks only (never in cloud)
INSIKA_RELAY_TOKEN — mounts the relay channel at POST /channels/relay/events, and is the Bearer it requires. Empty = the route does not exist (404). See Channels
INSIKA_RELAY_DELIVER_URL — your callback; the engine POSTs each reply there. Goes through the egress guard
INSIKA_RELAY_DELIVER_TOKEN — Bearer the engine sends to your callback (optional)
INSIKA_WIDGET_ORIGINS — exact-match origins allowed to embed the web widget, comma-separated. No wildcards. Half the switch: with INSIKA_WIDGET_AGENTS unset, nothing is mounted (404)
INSIKA_WIDGET_AGENTS — agent ids a widget visitor may address, comma-separated. The other half of the switch. A chat rate limit is also required or the widget answers 503
LITESTREAM_REPLICA_URL — enables Litestream (backup/DR). Empty = disabled (default). See below
LITESTREAM_ENDPOINT — S3-compatible endpoint (R2/MinIO). Empty = AWS S3
LITESTREAM_REGION — bucket region (AWS: us-east-1; R2: auto)
LITESTREAM_ACCESS_KEY_ID / LITESTREAM_SECRET_ACCESS_KEY — bucket credentials (read natively by Litestream)

Renamed from HARNESS_* → INSIKA_*. Every engine variable now uses the INSIKA_ prefix. The old HARNESS_* names are still honored as deprecated aliases — set either one and the engine reads it, logging a one-line deprecation notice at boot (insika doctor reports it too). Migrate at your convenience; the legacy names will be dropped in a future release.

The database filename changed too — the image now defaults to INSIKA_DB=/data/insika.db (it was /data/harness.db). Existing volumes are adopted automatically: the container entrypoint renames the old file — with its -wal/-shm siblings, before anything opens it — when the configured path does not exist yet. Nothing to run by hand, no data lost, and a no-op from the second boot on. To keep the old filename instead, point INSIKA_DB at it: the variable is the knob, the image only picks a default. (The adoption lives in deploy/entrypoint.sh, not in the engine — it is deploy baggage, not a runtime behavior.)

Tokens & rotation (keep the two separate!)

There are two secrets with distinct purposes — in production use different values (the API token falling back to ADMIN_TOKEN is a dev convenience only):

  • ADMIN_TOKEN — the /studio login (cookie auth). This is the operator surface (just you). Rotating it is safe and independent: change it, redeploy, log in with the new value. It does not affect any API consumer.
  • OPENCLAW_GATEWAY_TOKEN — the Bearer for /v1/responses and /v1/agents. This is the contract with your API consumers. Rotating it means changing both sides together (or the integration breaks): update the runtime var and each consumer’s token in the same step.

Generate a strong token: ruby -rsecurerandom -e 'puts SecureRandom.hex(24)'.

Strict config and insika doctor

Config discipline that rejects unknown keys — no silent schema tolerance. Two parts:

1. Boot gate. On boot, the engine validates the environment against a schema of known keys (Insika::EnvSchema): a wrong type (INSIKA_PORT=abc) or an unknown key in the INSIKA_ namespace (a typo like INSIKA_EGRES_ALLOW_HTTP the runtime would silently ignore). By default it only warns and boots anyway (last-known-good — a rotated key or a typo never takes the whole service down). To refuse boot on any finding, set INSIKA_CONFIG_STRICT=1. Unknown-key detection is scoped to the INSIKA_ prefix; the shared OPENCLAW_, LITESTREAM_, and OTEL_ namespaces are never flagged.

2. bin/insika doctor — on-demand diagnostics. Reads the same durable backend the server uses (INSIKA_DB) without booting the whole app (no provider, no seed) — safe to run against a production volume:

insika doctor            # colored report; exits != 0 on any error
insika doctor --json     # machine-readable (CI / monitoring)
insika doctor --fix      # applies the safe autofixes and re-diagnoses
insika env               # lists known keys + current values (secrets masked)

Checks: env (the schema above), settings schema version (a pending migration → --fix applies it), a missing platform default_model (--fix seeds it from DEEPSEEK_MODEL), durable vs ephemeral backend, LLM provider configured, ADMIN_TOKEN set, data-tool definitions still valid, and prompt files that hold text rather than a serialized object (a file whose content is a stringified Hash serves a mangled prompt on every turn while looking perfectly healthy — present, non-empty, and the agent still answers). Settings-schema migrations are explicit — no Studio save silently reinterprets old-shape data.

Data-tool callbacks to a backend — via a tunnel

Data-tools call back a consumer’s internal HTTP API. With the engine in the cloud and your backend on your machine (:3000), expose it over a public https tunnel and point the engine at it:

# in the tool/manifest: base_url = {{env.ACHEI_INTERNAL_URL}}
ACHEI_INTERNAL_URL=https://your-tunnel.example.dev
INSIKA_EGRESS_HOSTS=your-tunnel.example.dev

Because the tunnel is public https, the strict egress guard (the default) already allows it — you do not need ALLOW_HTTP/ALLOW_PRIVATE (those are only for a fully-local loop). Restricting INSIKA_EGRESS_HOSTS to the tunnel host is the secure posture. See Security.

Railway

railway.json already configures the Dockerfile builder, startCommand, the /up healthcheck, and a restart policy.

  1. Create the project/service from this repo (builder = Dockerfile).
  2. Volume: mount it at /data (the default INSIKA_DB points there) — without a volume, SQLite is ephemeral and recovery resumes nothing after a redeploy.
  3. Vars: DEEPSEEK_API_KEY, OPENCLAW_GATEWAY_TOKEN, ACHEI_INTERNAL_URL, INSIKA_EGRESS_HOSTS (and WEB_CONCURRENCY to match your plan/CPU).
  4. The healthcheck hits /up.
  5. Point your consumer at the service’s public URL, with a matching API token (see RUNNING-LOCAL.md).

Backup / DR — Litestream (opt-in, configurable)

A single volume is the one point of total loss between a pilot and production (disk corruption/loss = goodbye conversations + config). Litestream does continuous replication of insika.db (its WAL) to an S3/R2 bucket, without changing databases and without a line of Ruby.

It is off by default and turns on by env — a single-box ephemeral deploy pays nothing; a durable deploy enables it by pointing at a bucket. The trigger is one variable, LITESTREAM_REPLICA_URL:

  • empty (default): the entrypoint execs Falcon directly. The Litestream binary is never invoked — behavior identical to not having it.
  • set: on a fresh box the entrypoint restores insika.db from the replica before the app opens it (litestream restore -if-replica-exists; a no-op if the bucket is still empty), then supervises the app (litestream replicate -exec), replicating the WAL continuously and doing a final sync on shutdown (Railway’s SIGTERM).

Enable in production (Railway)

Add the vars (keep the volume at /data):

# AWS S3
LITESTREAM_REPLICA_URL=s3://my-bucket/insika
LITESTREAM_REGION=us-east-1
LITESTREAM_ACCESS_KEY_ID=AKIA...
LITESTREAM_SECRET_ACCESS_KEY=...

# Cloudflare R2 (S3-compatible): same, + endpoint and region=auto
LITESTREAM_REPLICA_URL=s3://my-bucket/insika
LITESTREAM_ENDPOINT=https://<accountid>.r2.cloudflarestorage.com
LITESTREAM_REGION=auto
LITESTREAM_ACCESS_KEY_ID=...
LITESTREAM_SECRET_ACCESS_KEY=...

Credentials are read natively by Litestream (they are not in deploy/litestream.yml, which only references URL/endpoint/region via ${VAR}).

Restore drill (the real “done” — an untested backup does not count)

The pilot→production gap only closes once a restore has been exercised. Two ways:

1. Local, automated (proves the mechanism, zero credentials): uses the real image + a file:// replica; boots → replicates → deletes the volume → boots a new box → restores → confirms the marker row survived and /up is green.

scripts/litestream-restore-drill.sh      # needs docker, sqlite3, curl
# → [drill] PASS — marker … restored from replica; /up green on the new box.

2. Production (the drill that counts for go-live): against the real bucket.

# a. with the service live and replicating, generate some config/conversation and
#    confirm the replica has generations:
litestream snapshots -config deploy/litestream.yml "$INSIKA_DB"

# b. boot a NEW box (empty volume) with the same LITESTREAM_* vars → the entrypoint
#    restores on boot. Verify manually in /studio that conversations and config came
#    back. Manual restore alternative:
litestream restore -config deploy/litestream.yml -o /tmp/restored.db "$INSIKA_DB"

Kubernetes (evolution)

SQLite does not share one file across nodes. Paths forward: a StatefulSet + a PVC per pod + sticky-by-agent routing (shard by tenant), or LiteFS, or an optional Postgres adapter. Litestream (above) for backup/DR from day one — orthogonal to topology.

Measuring performance / load

1. SQLite write ceiling (no provider) — bench_store.rb

Isolates “can SQLite take multi-process writes?” from LLM noise: N processes hammering writes on the same file (WAL + busy_timeout — the real config).

bundle exec ruby scripts/bench_store.rb 1,2,4,8 3000

Measured (mid-2026, laptop, ~481B payload):

procs writes/s (aggregate) p50 p95 max locked
1 ~29.6k 0.02ms 0.04ms 2.4ms 0
2 ~29.9k 0.02ms 0.04ms 59ms 0
4 ~23.7k 0.02ms 0.05ms 336ms 0
8 ~28.4k 0.03ms 0.05ms 539ms 0

Reading: aggregate throughput stays ~25–30k writes/s regardless of process count (the WAL’s one-writer-at-a-time ceiling), with zero “database is locked” (the busy_timeout absorbs contention into tail latency, not errors), and a microscopic p95. A real turn is provider-bound (seconds) and does a handful of writes → the workload sits ~100× under the ceiling. Empirically, SQLite is not the bottleneck on a single box.

2. End-to-end load (with provider) — loadtest.rb

Hits POST /v1/responses (SSE), the production path. Measures TTFB, total, tokens, cache hits, P50/P95, error rate. Runs against local or a remote deployment. See LOADTEST.md.

INSIKA_URL=http://localhost:9292 OPENCLAW_GATEWAY_TOKEN=xxx \
  bundle exec ruby scripts/loadtest.rb --agents assistant --concurrency 16 --iterations 3

3. Baseline vs multi-worker on one box — loadtest-local.sh

Boots Falcon with 1 worker, then N, over the same SQLite, and counts “database is locked” in the logs.

DEEPSEEK_API_KEY=sk-... ./scripts/loadtest-local.sh 4 24

See also

Back to top

Insika is MIT-licensed. Reading this as an agent? llms.txt indexes these docs as raw markdown.

This site uses Just the Docs, a documentation theme for Jekyll.