Operations Runbook¶

Health Check¶

GET /api/health returns the status of the database and Redis:

{
  "status": "ok",
  "version": "0.1.0",
  "uptime": 12345.67,
  "checks": {
    "database": { "status": "ok", "latencyMs": 2 },
    "redis": { "status": "ok", "latencyMs": 1 }
  }
}

Status	Meaning
`ok`	Both database and Redis are healthy
`degraded`	One or both dependencies are down

Use this endpoint for load balancer health checks. A degraded status means the server is running but some functionality (sessions, job scheduling) may be impaired.

The portal catalog Related articles typeahead depends on the published Floh docs site, not Floh API routes.

Check	What to verify
Index file	`GET https://<docs-host>/search/knowledge_index.json` returns 200 and only `user-guide/` + `portal/` sections
CORS	Response includes `Access-Control-Allow-Origin` (Pages `_headers` on `/search/knowledge_index.json`)
Cache	`Cache-Control: public, max-age=300` so repeat catalog opens do not re-download the index

If suggestions are empty but the catalog form otherwise works, check the browser network tab for a failed cross-origin fetch or an oversize/malformed JSON body. There is no Postgres or /api/knowledge fallback — restore the docs deploy or CORS headers.

Local portal dev: a 500 on /docs-search/knowledge_index.json with Vite logging ECONNREFUSED 127.0.0.1:8000 means MkDocs is not running. Start pnpm docs:serve in a second terminal. Confirm curl -sI http://127.0.0.1:8000/search/knowledge_index.json returns 200 before retrying the catalog form.

If curl -sI http://127.0.0.1:8000/search/search_index.json is 200 but knowledge_index.json is 404 (or missing under site/search/), the post-build filter did not run — restart pnpm docs:serve and look for Wrote .../knowledge_index.json in serve output or a knowledge_index hook error.

Content authoring: requester-facing articles live in the Floh docs tree (docs/user-guide/, docs/portal/). The portal typeahead only searches the filtered index built from those sections. There is no Postgres or /api/knowledge fallback. Merging a docs-only PR to main triggers a Cloudflare Pages rebuild via the Release workflow (no package changeset required); allow a few minutes before expecting new suggestions in the catalog form.

Legacy Postgres knowledge_article: this cutover stops reading the table. Do not drop it in this change. Authored rows stay in the database so they can be exported before the bake-period drop. Until that follow-up ships, SELECT from knowledge_article is the export path.

Stuck Run Recovery¶

Workflow runs that remain in running status without progress are automatically failed by a periodic job.

Setting	Env var	Default
Timeout	`STUCK_RUN_TIMEOUT_MINUTES`	30
Check interval	—	Every 15 minutes

A run is considered stuck if its updated_at timestamp is older than the timeout. The run is marked as failed with reason "Run timed out (no progress for N minutes)".

The timeout can also be configured via Admin > Security Settings in the web UI.

BullMQ Job Processing¶

All background jobs use a single BullMQ queue named workflow-scheduler.

Retry Policy¶

Jobs are retried up to 3 times with exponential backoff (5s, 10s, 20s). Failed jobs are retained for inspection (up to 5000).

Job Types¶

Job	Schedule	Purpose
`role-expiry-check`	Hourly	Revoke expired role assignments
`document-expiry-check`	Hourly	Revoke roles with expired documents
`entitlement-reconciliation`	Daily 02:00 UTC	Reconcile entitlements with connectors
`run-orphan-cleanup`	Daily 02:30 UTC	Clean up orphaned run artifacts
`stuck-run-recovery`	Every 15 minutes	Fail runs stuck in running state
`audit-checkpoint`	Configurable	Create signed audit checkpoint
`deliver-scheduled-report`	Per-schedule cron	Generate and email scheduled reports
`trigger-workflow`	Per-schedule cron	Start scheduled workflow runs
`escalation-reminder`	Delayed	Send approval reminders
`escalation-reassignment`	Delayed	Reassign overdue approvals

Monitoring Failed Jobs¶

Use the BullMQ dashboard or query Redis directly to inspect failed jobs:

redis-cli keys "bull:workflow-scheduler:failed:*"

Graceful Shutdown¶

The server handles SIGTERM and SIGINT signals:

Stops accepting new HTTP connections
Completes in-flight requests
Drains the BullMQ worker (waits for active jobs)
Closes Redis and database connections
Exits process

Docker and Kubernetes send SIGTERM on container stop. The default grace period should be at least 30 seconds.

Shutdown now emits warning logs when close operations fail (log service, scheduler, Redis) so partial cleanup failures are visible in production logs.

Database Pool Tuning¶

Env var	Default	Description
`DB_POOL_MAX`	10	Maximum number of connections
`DB_POOL_MIN`	2	Minimum idle connections
`DB_POOL_IDLE_TIMEOUT_MS`	30000	Close idle connections after this time
`DB_POOL_CONNECTION_TIMEOUT_MS`	5000	Fail if connection cannot be acquired

Monitor active connections with:

SELECT count(*) FROM pg_stat_activity WHERE datname = 'floh';

Log Retention¶

System logs are stored in the system_log table. A daily cleanup job purges entries older than LOG_RETENTION_DAYS (default: 30).

Manual purge is available via Admin > Logging > Purge Now in the web UI.

Audit Checkpoints¶

Signed audit checkpoints are created on a configurable schedule (default: every 6 hours). Checkpoints are stored according to AUDIT_CHECKPOINT_STORE:

Store	Env var	Description
`file`	`AUDIT_CHECKPOINT_PATH`	Local filesystem (default)
`s3`	—	S3-compatible storage (planned)
`siem`	—	SIEM integration (planned)

Connector Execution Audit Tags¶

Manual connector execution events are written to the audit log with distinct actions:

connector.executed for standard integrations
connector.mock_executed for built-in mock/test connectors

Use this distinction in audits and dashboards to separate simulation traffic from production integration activity.

Configuration Validation¶

The server validates all configuration at startup. Invalid values (e.g., PORT=abc, DB_POOL_MAX=0) produce clear error messages and prevent startup.

In production, the server also validates that required secrets are not using default/fallback values. See Security for the full list.