Skip to content

Deployment Guide (Non-Production)

Introduction

This guide deploys the full Floh stack to a single EC2 instance with Docker Compose, with Caddy providing automatic HTTPS. It covers the CI (dev) and QA environments, which share one Compose file and one deploy workflow. It does not cover production.

Authifi setup (OIDC clients, resource servers, channel audiences, the Floh roles catalog, and the console Authifi sync) is in Auth Configuration.

Contents

TLS by Tier

Tier TLS mode How configured
Edge (caddy) HTTPS on :443, certs from Let's Encrypt DEPLOY_DOMAIN, DEPLOY_PORTAL_DOMAIN, DEPLOY_FORM_BUILDER_DOMAIN
Admin/portal/form-builder browser traffic HTTPS at public domains handled by Caddy reverse proxy
Internal container traffic (server, web, portal-web, form-builder) HTTP on Docker network default compose networking (server:7070, etc.)
Caddy → portal-bff HTTPS on Docker network, verified PORTAL_BFF_TLS_CERT / PORTAL_BFF_TLS_KEY, SAN must cover DEPLOY_PORTAL_DOMAIN
Caddy → console-bff HTTPS on Docker network, verified CONSOLE_BFF_TLS_CERT / CONSOLE_BFF_TLS_KEY, SAN must cover DEPLOY_DOMAIN
OIDC redirect / post-logout URLs HTTPS public URLs public/ci.env (OIDC_REDIRECT_URI, FRONTEND_URL, PORTAL_FRONTEND_URL)

Notes:

  • Do not set TLS_CERT_FILE / TLS_KEY_FILE for deploy compose. API TLS terminates at Caddy.
  • The two BFF legs are the exception to "internal traffic is HTTP". Each Authifi gateway serves HTTPS on the Docker network — portal-bff on :7071 from PORTAL_BFF_TLS_CERT / PORTAL_BFF_TLS_KEY, console-bff on :7074 from CONSOLE_BFF_TLS_CERT / CONSOLE_BFF_TLS_KEY — and Caddy verifies each (no tls_insecure_skip_verify) against the matching *_CA_CERT rather than the container's system roots. The two leaves are not interchangeable: each certificate's SAN must cover the tls_server_name Caddy sends for that upstream, which is DEPLOY_PORTAL_DOMAIN for the portal and DEPLOY_DOMAIN for the console. See Generating the portal BFF certificate and Generating the console BFF certificate.
  • FLOH_INTERNAL_URL in public config should stay http://server:7070 for Docker deploy.
  • The Floh server runs a periodic TLS health check (every 15 min). If a domain's cert is missing or invalid, the server reloads Caddy via its admin API to trigger a fresh ACME challenge. Set CADDY_ADMIN_URL=http://caddy:2019 in ci.env to enable (already configured).

Architecture

Internet
  │
  ├─ :443 ──► Caddy (TLS termination, Let's Encrypt)
  │             ├─ floh-dev.example.com/api/*      ──► server:7070
  │             ├─ floh-dev.example.com/scim/v2/*  ──► server:7070 (inbound SCIM)
  │             ├─ floh-dev.example.com/*           ──► web:8080 (nginx)
  │             ├─ portal.example.com/bff/*         ──► portal-bff:7071 (HTTPS, OIDC login/logout/step-up)
  │             ├─ portal.example.com/api/*         ──► portal-bff:7071 (HTTPS, proxied to server)
  │             ├─ portal.example.com/*               ──► portal-web:8080 (nginx)
  │             └─ forms.example.com/*                ──► form-builder:8080 (nginx)
  │                  (CSP frame-ancestors pinned to floh-dev.example.com)
  │
  └─ :8025 ──► MailHog UI (restrict to your IP)

Inbound SCIM (Entra ID, Okta, etc.) uses the admin domain, not a separate API hostname. IdPs call https://<DEPLOY_DOMAIN>/scim/v2 (for example https://floh.authilize.com/scim/v2). Caddy must proxy /scim/v2/* to server:7070 before the catch-all rule that serves the admin SPA; otherwise provisioning clients receive the Angular index.html instead of SCIM JSON.

Verify after deploy:

curl -sS "https://<DEPLOY_DOMAIN>/scim/v2/ServiceProviderConfig" \
  -H "Authorization: Bearer <scim-token>"

Expect JSON with "patch":{"supported":true}. HTML containing <app-root> means the /scim/v2 route is missing or the deploy has not picked up an updated docker/Caddyfile. Token setup is documented in Entra setup for Floh and Okta setup for Floh.

The form-builder site is the standalone @floh/form-builder-app SPA, iframed by the Floh admin UI to power the Workflow Designer's Visual editor. It MUST live on an origin distinct from DEPLOY_DOMAIN — the host class rejects same-origin embeds at runtime. The deploy writes https://${DEPLOY_FORM_BUILDER_DOMAIN}/ into the web container's environment. It does not bake it into the image. The container writes /runtime-config.json at start so the Visual toggle is enabled.

Prerequisites

  • AWS account
  • A domain name with DNS you can control
  • GitHub repo: github.com/Authifi/floh

First-time bring-up checklist

The numbered steps below cover host setup. This checklist is the dependency order for a domain that has never been deployed, including the identity and certificate work that the host steps do not mention. Each item links to its section.

Skipping an item does not fail the deploy at that point — it fails later, often somewhere unrelated-looking. The right-hand column names what you get if you skip it.

Provisional: verified against the code, not yet run end to end

This ordering was derived by tracing the deploy workflow and the Authifi setup scripts, and each stated failure mode is taken from the actual error path rather than inferred. It has not been executed against a fresh tenant from step 1 to step 13.

The --identity-provider requirement in step 9 was found by tracing the code rather than by running the checklist, so treat a first real bring-up as the thing that validates this list, and please correct it where reality differs. Run step 9 with --dry-run first; it is the step with the most ways to fail.

# Do this Skip it and you get
1 Provision the host: Steps 1–6 —
2 Point DNS at the Elastic IP for all three domains (Step 4) Caddy cannot complete ACME; TLS verification fails
3 Set the deploy variables (DEPLOY_DOMAIN, DEPLOY_PORTAL_DOMAIN, AUTHIFI_ADMIN_TARGET, …) (Variables) Preflight fails naming the variable
4 Generate the portal BFF certificate and set its three secrets (recipe) Every /bff/* request 502s
5 Generate the console BFF certificate — separate leaf, SAN covers DEPLOY_DOMAIN (recipe) console-bff never becomes healthy; the whole console and API are down
6 Set the remaining secrets (Secrets) — all except OIDC_CLIENT_SECRET and PORTAL_OIDC_CLIENT_SECRET, which do not exist yet Preflight fails naming the secret
7 Decide the channel audience identifiers (FLOH_CONSOLE_AUDIENCE, FLOH_PORTAL_AUDIENCE, FLOH_MCP_AUDIENCE) and set them in the environment you run steps 8–9a from Steps 8–9a act on the default identifiers: step 8 creates resource servers the deployed stack never requests, step 9 binds the clients to those, and step 9a verifies that same wrong set. All three steps read all three identifiers
8 Create the three channel resource servers with pnpm run sync-authifi-rbac -- --apply. No client exists yet, so this run creates no client bindings Step 9 refuses to run because the missing channel resource server cannot be resolved
9 Register the OIDC clients and their resource-server bindings, passing the deployed origins and an identity provider explicitly (details) Without the origins, login fails with invalid_redirect_uri; without --identity-provider, creating a client aborts outright
9a Rerun pnpm run sync-authifi-rbac -- --apply to verify the assignment state step 9 created against the shared planner Step 9's bindings are never checked against desired state, so a partial step 9 or later drift goes unnoticed until a login fails
10 Capture the minted client secrets into OIDC_CLIENT_SECRET and PORTAL_OIDC_CLIENT_SECRET — the two deferred from step 6 Login fails with invalid_client; the value cannot be re-read later
11 Merge config/public/ci.env changes (URLs, origins, and the same audience identifiers chosen in step 7) Wrong redirect URLs or CORS rejections; an audience mismatch with step 8 fails login
12 Run the deploy workflow (Deploying) —
13 Smoke-test login, not just the home page (below) A broken auth path reports as a green deploy

Four ordering constraints are easy to get wrong:

  • Certificates before the first deploy. The health gate blocks on console-bff and portal-bff, so a missing or malformed leaf fails the whole deploy rather than degrading one route.
  • Resource servers before client registration, then the sync again. On a new tenant, setup-authifi-oidc-clients resolves all three channel resource servers before its first write and refuses to proceed if any is missing, so the RBAC sync has to come first. The first sync run therefore happens when no Floh OIDC client exists yet: it detects that, plans the whole RBAC catalog without any client-to-resource-server assignment, and prints the next step. The division of labour is worth stating plainly: the sync owns resource servers and the RBAC catalog and never creates a client, while setup-authifi-oidc-clients owns the clients and their bindings — registering each one and linking it only to its own channel resource server. Step 9a is therefore a verification pass, not the step that creates the links. It still matters: it is the only operation that checks step 9's result against the shared planner's desired state. It repairs a missing managed link after an interrupted setup, a hand edit, or later drift before that problem surfaces as a failed login. It is idempotent, so on a clean run it reports no changes. A tenant where only some of the three clients exist, where one is duplicated, or where a client row carries no usable id fails closed before any mutation instead of bootstrapping over the inconsistency.
  • Audience identifiers before either Authifi command. Both commands read the identifiers from the environment you invoke them in, and neither reads config/public/ci.env. Every sync-authifi-rbac invocation reads all three (FLOH_CONSOLE_AUDIENCE, FLOH_PORTAL_AUDIENCE, FLOH_MCP_AUDIENCE) and fails closed if any is missing or if two collide — that applies to the step 8 bootstrap and the step 9a verification alike, so the rerun needs the same environment as the first run rather than inheriting anything from it. setup-authifi-oidc-clients reads the same three, because it registers floh-mcp-client alongside the browser clients. So if the deployment needs non-default identifiers, choosing them after those commands leaves the tenant holding resource servers under the old names while the deployed API and BFFs request the new ones. Set the same values in both places.

Legacy transitional configuration

Ordinary client setup requires only the console, portal, and MCP channel resource servers. It does not inspect, add, or remove pre-existing transitional client links. Those legacy relationships remain untouched until the explicit, previewable cleanup owned by LSA-10182.

Setup also ignores and preserves existing FLOH_RESOURCE_ID and OIDC_AUDIENCE assignments even when the pair conflicts. The separate sync-authifi-rbac command still validates that canonical/deprecated alias pair as part of its operator bootstrap and may fail before planning until the operator corrects the values or removes the stale assignments.

  • Client registration before capturing secrets. Authifi cannot re-read an existing client secret. If you register clients and do not capture the minted values, the only recovery is rotation.

Step 9 has a trap worth spelling out. Run bare, setup-authifi-oidc-clients takes its origin lists from FRONTEND_URL and PORTAL_FRONTEND_URL in your local environment, which on a fresh checkout from .env.example are https://localhost:7072 and https://localhost:7073. The deployed values live in config/public/ci.env, which this checklist does not touch until step 11 and which the script does not read in any case. So the bare command registers localhost callbacks, the first deployed login fails with invalid_redirect_uri, and — because the run reconciles rather than appends — any deployed origins already registered are removed. Always pass --console-origin and --portal-origin explicitly and preview with --dry-run first; the full command, including the localhost origins to keep when the tenant is shared with local development, is in Authifi client registration.

Step 9 has a second trap on a genuinely new tenant: creating the console client requires an identity provider, and the script refuses to guess one. With none supplied it stops before any write, including under --dry-run:

Creating floh-client and floh-mcp-client requires --identity-provider (or AUTHIFI_IDENTITY_PROVIDERS).
Duplicate of an existing console client copies providers automatically.

The message names whichever clients the run would create. floh-mcp-client appears because its initial refresh token comes from an authorization-code flow, so it needs a provider too; the vault client does not (it authenticates with private_key_jwt).

Pass --identity-provider <name> (repeatable) or set AUTHIFI_IDENTITY_PROVIDERS=nih,google. .env.example leaves the variable commented out, so a fresh checkout supplies nothing. The value is the Authifi identity-provider name your tenant federates to — ask whoever owns the tenant rather than guessing, since a wrong provider produces a client nobody can log in through. This constraint only applies when the client does not yet exist; reconciling an existing client copies its providers.

For where each value lives and what overrides what, see the Configuration Map.

Post-deploy login smoke test

The deploy's own TLS check probes https://<domain>/, which Caddy serves from the SPA containers. It therefore passes while the entire authentication path is broken. After every deploy to a new domain, confirm:

  1. https://<DEPLOY_DOMAIN>/ loads the console.
  2. Clicking sign-in reaches the IdP (not an invalid_redirect_uri page).
  3. Completing login returns you to the console authenticated — this is the step that catches a stale OIDC_CLIENT_SECRET.
  4. https://<DEPLOY_PORTAL_DOMAIN>/bff/login redirects rather than 502s.
  5. Complete a portal login and land on an authenticated portal page. Step 4 is not sufficient on its own: the portal client secret is not used until the authorization-code exchange on the callback, so /bff/login redirects perfectly well with a stale PORTAL_OIDC_CLIENT_SECRET and every login then fails with invalid_client. This is the portal equivalent of step 3.

Any failure here maps to an entry in the Failure reference.

Step 1: Launch EC2 Instance

  • Instance type: t3.medium (2 vCPU, 4 GB RAM)
  • AMI: Ubuntu 24.04 LTS
  • Storage: 30 GB gp3 EBS
  • Key pair: Create or select one (save the .pem file)

Step 2: Security Group

Allow inbound:

Port Protocol Source Purpose
22 TCP Your IP SSH
80 TCP Anywhere HTTP (ACME challenges + redirect)
443 TCP Anywhere HTTPS
8025 TCP Your IP MailHog UI (optional)

Step 3: Elastic IP

Allocate an Elastic IP and associate it with your instance. This gives a stable address that survives reboots.

Step 4: DNS

Create A records pointing to the Elastic IP:

  • floh-dev.example.com → <ELASTIC_IP>
  • portal.floh-dev.example.com → <ELASTIC_IP>
  • forms.floh-dev.example.com → <ELASTIC_IP> (form-builder SPA — must be a distinct origin from the admin domain)

Caddy will auto-provision Let's Encrypt certificates once DNS resolves.

Step 5: Install Docker and Dependencies

ssh -i your-key.pem ubuntu@<ELASTIC_IP>

curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker ubuntu
newgrp docker
sudo apt-get install -y jq

docker --version
docker compose version
jq --version

Step 6: Register the Self-Hosted Runner

The deploy workflow runs on a self-hosted runner with the deploy label. Register the EC2 instance as a GitHub Actions runner:

  1. Go to Settings → Actions → Runners
  2. Click New self-hosted runner and follow the install steps for Linux x64
  3. Configure the runner with the label deploy
  4. Install and start the runner as a service so it survives reboots:
sudo ./svc.sh install
sudo ./svc.sh start

The workflow authenticates to GHCR automatically via docker/login-action — no manual PAT login required.

Step 7: Configure the GitHub "dev" Environment

The deploy workflow pulls all secrets and variables from the GitHub dev environment. Non-sensitive runtime config lives in checked-in public config files (config/public/base.env and config/public/ci.env) and is copied to ~/floh/public/ during each deploy, alongside the portal BFF's docker/bff/portal.json in ~/floh/bff/.

In Settings → Environments, create an environment named dev and add:

Secrets

Secret Value
DB_PASSWORD Strong database password
REDIS_PASSWORD Redis password (leave empty if no auth)
SMTP_USER SMTP username (empty for MailHog)
SMTP_PASS SMTP password (empty for MailHog)
JWT_SECRET 64-char hex key (pnpm run generate-key)
SESSION_SECRET 64-char hex key (pnpm run generate-key)
SESSION_ENCRYPTION_KEY 64-char hex key (pnpm run generate-key)
AUDIT_CHECKPOINT_KEY 64-char hex key (pnpm run generate-key)
OIDC_CLIENT_SECRET Console (floh-client) OIDC client secret — written to ~/floh/env/console.env
PORTAL_OIDC_CLIENT_SECRET Portal (floh-portal-client) OIDC client secret — written to ~/floh/env/portal.env
PORTAL_BFF_COOKIE_ENCRYPTION_SECRET Portal BFF session-cookie encryption key — any non-empty string; use pnpm run generate-key. Written to ~/floh/env/portal.env
PORTAL_BFF_TLS_CERT PEM certificate the portal BFF serves on :7071 — written to ~/floh/env/portal.env
PORTAL_BFF_TLS_KEY PEM private key for the above — written to ~/floh/env/portal.env
PORTAL_BFF_CA_CERT PEM CA certificate that issued the pair above — written to ~/floh/bff-ca.crt
CONSOLE_BFF_COOKIE_ENCRYPTION_SECRET Console BFF session-cookie encryption key — written to ~/floh/env/console.env
CONSOLE_BFF_TLS_CERT PEM certificate the console BFF serves on :7074 — written to ~/floh/env/console.env
CONSOLE_BFF_TLS_KEY PEM private key for the above — written to ~/floh/env/console.env
CONSOLE_BFF_CA_CERT PEM CA certificate that issued the pair above — written to ~/floh/console-bff-ca.crt
CONNECTOR_ENCRYPTION_KEY 64-char hex key (pnpm run generate-key)
CONNECTOR_ENCRYPTION_KEY_PREVIOUS Previous key during rotation (empty when not rotating)
AUDIT_CHECKPOINT_KEY_PREVIOUS Previous key during rotation (empty when not rotating)
AUTHIFI_VAULT_PRIVATE_KEY PEM private key for the vault client — only required when SECRETS_BACKEND=authifi (see Authifi tenant-secrets vault)

The deploy fails closed if any of DB_PASSWORD, JWT_SECRET, SESSION_SECRET, SESSION_ENCRYPTION_KEY, AUDIT_CHECKPOINT_KEY, CONNECTOR_ENCRYPTION_KEY, OIDC_CLIENT_SECRET, PORTAL_OIDC_CLIENT_SECRET, PORTAL_BFF_COOKIE_ENCRYPTION_SECRET, PORTAL_BFF_TLS_CERT, PORTAL_BFF_TLS_KEY, PORTAL_BFF_CA_CERT, CONSOLE_BFF_COOKIE_ENCRYPTION_SECRET, CONSOLE_BFF_TLS_CERT, CONSOLE_BFF_TLS_KEY, or CONSOLE_BFF_CA_CERT is unset. The BFF TLS PEMs are multiline and are written to ~/floh/env/console.env and ~/floh/env/portal.env single-quoted so the newlines survive; PORTAL_BFF_CA_CERT and CONSOLE_BFF_CA_CERT are public, so they are written to ~/floh/bff-ca.crt and ~/floh/console-bff-ca.crt and bind-mounted instead — into Caddy, which verifies each gateway's certificate, and into the matching BFF, whose healthcheck probes /health over the same TLS hop and would otherwise fail to verify the chain against Node's bundled roots.

Rotating an existing secret

Most of these can be replaced freely. Rotating JWT_SECRET, SESSION_SECRET, SESSION_ENCRYPTION_KEY, or PORTAL_BFF_COOKIE_ENCRYPTION_SECRET costs nothing worse than logging everyone out, and the three portal BFF PEMs can be reissued as a set at any time because nothing outside this deployment trusts them. Three are not free, and none of the three fails at deploy time:

  • DB_PASSWORD cannot be rotated here alone. Compose passes it as POSTGRES_PASSWORD, which Postgres honors only when initializing an empty data directory. The volume persists across deploys, so changing the secret changes what the server sends without changing what the database expects, and the next deploy fails to authenticate. See Rotating DB_PASSWORD.
  • CONNECTOR_ENCRYPTION_KEY is one-way. Replacing it leaves stored connector credentials undecryptable. Carry the outgoing value into CONNECTOR_ENCRYPTION_KEY_PREVIOUS and follow the connector key rotation runbook, which re-encrypts every stored secret under the new key and ends by clearing _PREVIOUS.
  • AUDIT_CHECKPOINT_KEY is one-way and, unlike the connector key, has no re-signing step. Re-signing historical checkpoints would defeat the tamper-evidence they exist to provide, so there is no equivalent of rotate-keys here. New checkpoints are signed with the current key, and GET /api/audit-logs/verify-integrity verifies each one against the current key plus AUDIT_CHECKPOINT_KEY_PREVIOUS. That means _PREVIOUS is not a transitional value to clear afterwards — it must stay populated for as long as checkpoints signed with the old key are still in scope for verification, and only one generation back is supported.

Both _PREVIOUS values must be exactly 64 hex characters. A value of any other length is silently ignored rather than rejected, so a truncated or whitespace-padded paste presents as connector secrets or historical checkpoints failing with nothing in the logs pointing at the key.

PORTAL_OIDC_CLIENT_SECRET and OIDC_CLIENT_SECRET have counterparts in Authifi, which cannot re-read an existing client secret. A lost value can only be replaced, not recovered: pnpm run setup-authifi-oidc-clients -- --rotate-secrets mints a new one and captures it.

When SECRETS_BACKEND=authifi, DB_PASSWORD is the one secret that lives in two stores at once — see the dual-source hazard — because the server reads it from the vault while Compose interpolates the GitHub copy into POSTGRES_PASSWORD. Even updating both is not sufficient; the live Postgres role has to be altered too, which the sequence immediately below covers.

The OIDC client secrets behave differently, and the vault copy does no work today. Neither OIDC_CLIENT_SECRET nor PORTAL_OIDC_CLIENT_SECRET is read by the API server — PORTAL_OIDC_CLIENT_SECRET appears nowhere in packages/server/src, and only the non-secret PORTAL_OIDC_CLIENT_ID is looked up. The consumer is the BFF, which Compose feeds from ~/floh/env/portal.env as AUTH_CLIENT_SECRET, and the deploy writes that file from the GitHub secret on both backends. Rotating the GitHub secret is therefore what actually takes effect.

They are still listed as required tenant secrets below, because both runbooks are pinned to SECRET_KEYS by test/unit/architecture/deploy-vault-bootstrap.test.ts and SECRET_KEYS is the set the deployment manages, not the narrower set the server resolves. Populating them keeps the vault self-consistent and costs a redundant copy of a live credential; whether the list should be split is #1192.

Rotating DB_PASSWORD

POSTGRES_PASSWORD is only an initialization convenience; the password is ordinary role state once the cluster exists, so change it in place:

  1. Set the new value as the DB_PASSWORD secret in the dev environment. This has no effect until the next deploy.
  2. If the stack runs SECRETS_BACKEND=authifi, set it in the vault too, as the secret named FLOH_DB_PASSWORD (the provider strips the AUTHIFI_VAULT_SECRET_PREFIX, default FLOH_). The vault is where the server actually reads this value on that backend, so a GitHub-only update leaves the restarted server sending the old password to a role that has already changed — the same lockout this runbook exists to prevent. Keep the two in sync: the GitHub secret still feeds Compose's POSTGRES_PASSWORD interpolation.
  3. Apply it to the running database:
cd ~/floh
docker compose -p floh -f docker-compose.deploy.yml \
  --env-file .env --env-file env/console.env --env-file env/portal.env \
  exec postgres psql -U floh -d floh -c '\password floh'

\password prompts for the value and sends it as a parameter, so the credential never reaches your shell history or the psql process arguments, and it needs no hand-quoting — an apostrophe in the password would otherwise terminate the SQL literal and reject a perfectly valid credential. Avoid -c "ALTER ROLE floh PASSWORD '…';" for exactly those two reasons.

  1. Re-run the deploy.

The role, database, and user name are all floh because DB_USER and DB_NAME are unset in this deployment and fall through to the compose defaults; if either is ever set, substitute accordingly.

Connections already open are authenticated and survive, so the only exposure is new connections between step 3 and the server restarting.

Do not rotate this by rebuilding the volume. Getting POSTGRES_PASSWORD to apply again means deleting pgdata, which turns a one-statement change into a dump, a destroy, and a restore. Worse, the obvious backup command hides a trap: pg_dumpall includes globals, and the globals dump carries the role's password hash as ALTER ROLE floh WITH PASSWORD 'SCRAM-SHA-256$...'. Restoring it re-applies the old password and silently undoes the rotation, presenting as the same authentication failure that prompted the rotation in the first place.

Generating the portal BFF certificate

Caddy proxies to https://portal-bff:7071 and verifies the certificate, so the BFF needs a certificate for <DEPLOY_PORTAL_DOMAIN> that chains to an anchor Caddy trusts. Generate a small private CA once, issue a long-lived leaf from it, and keep ca.key offline — only ca.crt is deployed:

DOMAIN=myfloh.authilize.com

openssl req -x509 -newkey rsa:4096 -sha256 -days 3650 -nodes \
  -keyout ca.key -out ca.crt \
  -subj "/CN=Floh Portal BFF Internal CA" \
  -addext "basicConstraints=critical,CA:TRUE,pathlen:0" \
  -addext "keyUsage=critical,keyCertSign,cRLSign"

openssl req -newkey rsa:2048 -nodes -keyout bff.key -out bff.csr \
  -subj "/CN=${DOMAIN}"

openssl x509 -req -in bff.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
  -out bff.crt -days 3650 -sha256 \
  -extfile <(printf "subjectAltName=DNS:%s\nextendedKeyUsage=serverAuth\nbasicConstraints=critical,CA:FALSE" "${DOMAIN}")

gh secret set PORTAL_BFF_TLS_CERT --repo Authifi/floh --env dev < bff.crt
gh secret set PORTAL_BFF_TLS_KEY  --repo Authifi/floh --env dev < bff.key
gh secret set PORTAL_BFF_CA_CERT  --repo Authifi/floh --env dev < ca.crt

The SAN must match the tls_server_name Caddy sends, which is DEPLOY_PORTAL_DOMAIN. A mismatch, or a bare self-signed certificate with no matching PORTAL_BFF_CA_CERT, fails the handshake and makes every /bff/* request return 502. The deploy's TLS verification step will not catch this, because it probes the portal root — which Caddy serves from portal-web, not the BFF. Test https://<DEPLOY_PORTAL_DOMAIN>/bff/login explicitly after a deploy.

These certificates are deliberately long-lived and are not renewed by ACME. Re-run the leaf steps before the 10-year expiry, or whenever DEPLOY_PORTAL_DOMAIN changes.

Generating the console BFF certificate

The console BFF needs the same treatment as the portal BFF, but its own leaf with a different SAN. Caddy sends tls_server_name {$DEPLOY_DOMAIN} on this hop (docker/Caddyfile), so the console leaf must cover DEPLOY_DOMAIN — reusing the portal leaf fails the handshake even though both certificates are valid and share a CA.

The blast radius is wider here than on the portal. Caddy routes /api/*, /bff/*, and /authifi/* through console-bff, so a bad certificate takes down the entire admin console and its API, not just login.

You can reuse the CA from the portal section (keep ca.key offline) and issue a second leaf, which is what the dev tier does:

DOMAIN=floh.authilize.com  # DEPLOY_DOMAIN, *not* DEPLOY_PORTAL_DOMAIN

openssl req -newkey rsa:2048 -nodes -keyout console.key -out console.csr \
  -subj "/CN=${DOMAIN}"

openssl x509 -req -in console.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
  -out console.crt -days 3650 -sha256 \
  -extfile <(printf "subjectAltName=DNS:%s\nextendedKeyUsage=serverAuth\nbasicConstraints=critical,CA:FALSE" "${DOMAIN}")

gh secret set CONSOLE_BFF_TLS_CERT --repo Authifi/floh --env dev < console.crt
gh secret set CONSOLE_BFF_TLS_KEY  --repo Authifi/floh --env dev < console.key
gh secret set CONSOLE_BFF_CA_CERT  --repo Authifi/floh --env dev < ca.crt

CONSOLE_BFF_CA_CERT is a separate secret from PORTAL_BFF_CA_CERT even when both hold the same CA — they are mounted to different files (~/floh/console-bff-ca.crt and ~/floh/bff-ca.crt). Set both.

Pipe the file with <. It is the one capture path that cannot reshape the value. Every malformed-PEM incident on this tier has come from copying the text through something else first — a browser paste into the GitHub secrets UI, an editor that escaped the newlines, a value round-tripped through JSON — leaving either literal \n escapes or a base64 blob where the PEM should be. The PEM is then unparseable, and Node fails with ERR_OSSL_PEM_NO_START_LINE inside configSecureContext at BFF startup. (--body "$(cat console.crt)" preserves interior newlines correctly, so it is not itself a corruption source; stdin redirection is still preferred because it keeps the secret out of your shell history and argument list.) The deploy's preflight now catches this shape before writing any host file; see Deploy preflight for TLS secrets.

As with the portal, the deploy's "Verify TLS for all domains" step does not exercise this leg — it probes https://<DEPLOY_DOMAIN>/, which Caddy serves from web. What does catch a broken console BFF is the health gate in "Start all services", which fails the deploy after ~120s with console-bff did not become healthy after startup. Confirm explicitly after a deploy by loading the console and completing a login.

Deploy preflight for TLS secrets

Before writing any environment file, the deploy runs scripts/validate-bff-tls-secrets.mjs against all seven PEM secrets. It fails the job in seconds — naming the variable, the problem, and the fix — for:

  • PEM that is unparseable because newlines became literal \n escapes, the value was base64-encoded, or the value arrived wrapped in quote characters
  • a private key that does not pair with its certificate
  • a leaf that was not issued by the CA in its *_CA_CERT secret
  • a leaf whose SAN does not cover the domain Caddy requests (the reused-portal-leaf mistake)
  • a certificate that is expired or not yet valid

It never prints secret values. Run the same check locally against files before setting secrets:

DEPLOY_DOMAIN=floh.authilize.com DEPLOY_PORTAL_DOMAIN=myfloh.authilize.com \
CONSOLE_BFF_TLS_CERT="$(cat console.crt)" \
CONSOLE_BFF_TLS_KEY="$(cat console.key)" \
CONSOLE_BFF_CA_CERT="$(cat ca.crt)" \
PORTAL_BFF_TLS_CERT="$(cat bff.crt)" \
PORTAL_BFF_TLS_KEY="$(cat bff.key)" \
PORTAL_BFF_CA_CERT="$(cat ca.crt)" \
node scripts/validate-bff-tls-secrets.mjs && echo "TLS secrets OK"

Variables

Variable Value
DEPLOY_DOMAIN floh.authilize.com
DEPLOY_PORTAL_DOMAIN myfloh.authilize.com
ACME_EMAIL Email for Let's Encrypt expiry warnings (optional)
SECRETS_BACKEND env (default when unset) or authifi. See Authifi tenant-secrets vault.
AUTHIFI_ADMIN_TARGET Fixed tenant-scoped <authifi-base>/auth/admin/tenants/<positive-tenant-id> target for the console BFF.
FILE_SCREENING_MODE disabled (server default) or clamd; the committed CI overlay selects clamd. See File screening.

Deploy-day note for FILE_SCREENING_MODE. clamd turns on the opt-in clamav service and points the API at it, unless the FILE_SCREENING_CLAMD_HOST variable names another clamd; any value other than disabled or clamd fails the deploy before a host file is written. Before the first enabled deploy, confirm the host has about 3 GiB of memory to spare for clamav, and expect that deploy to take several extra minutes while ClamAV downloads its signatures into the persistent clamav-db volume. Once enabled, ticket attachment uploads are screened before they are stored: a detected threat returns 422, and uploads return 503 while clamd is unreachable. Rollback is setting the FILE_SCREENING_MODE variable back to disabled and re-running Deploy, which also removes the container. Removing or unsetting the Actions variable does not disable screening because the committed CI overlay still selects clamd. The full runbook, including verification commands, is File screening.

The workflow writes three host files on every deploy: ~/floh/.env (API + infra), ~/floh/env/console.env, and ~/floh/env/portal.env. Manual edits to those files are overwritten. Rollback is this PR's three-file layout together with matching compose --env-file flags — a leftover single .env from before LSA-9829 is not a compatible host layout.

The console BFF's token slots, and the token-exchange grant they need on the console OIDC client, are described in Auth Configuration.

Repository-level variables

In addition to the dev environment vars above, the deploy workflow reads one repository-level variable (Settings → Variables → Actions, not the dev environment). Repository-level scope keeps the matrix build job out of the dev environment so it does not write dev deployment events on every push or inherit any future dev protection rules.

Variable Value
DEPLOY_FORM_BUILDER_DOMAIN forms.authilize.com — required. Must differ from DEPLOY_DOMAIN. The web container reads it at start.

config/public/ci.env contains non-secret URL and OIDC settings. Note that it outranks ~/floh/.env for every non-secret, so changing a non-secret in the GitHub environment usually has no effect — see the Configuration Map.

  • FRONTEND_URL, PORTAL_FRONTEND_URL, ALLOWED_ORIGINS, ALLOWED_PORTAL_ORIGINS
  • OIDC_ISSUER, OIDC_CLIENT_ID, OIDC_REDIRECT_URI, PORTAL_OIDC_CLIENT_ID
  • OIDC_AUDIENCE — Authifi catalog API resource id (FLOH_RESOURCE_ID locally). Runtime and operator commands other than OIDC client setup still use it. The API does not accept it as JWT aud.
  • FLOH_CONSOLE_AUDIENCE, FLOH_PORTAL_AUDIENCE, FLOH_MCP_AUDIENCE — channel catalog RSIDs. Console BFF AUTH_RESOURCE is FLOH_CONSOLE_AUDIENCE (also written to env/console.env); portal BFF uses FLOH_PORTAL_AUDIENCE (and env/portal.env). The API verifies JWT aud against these identifiers. The deploy workflow copies them from ci.env into the host env files. An unset or blank value defaults to https://console.floh.api, https://portal.floh.api, or https://mcp.floh.api, in the API, the BFFs, and the operator scripts alike. The three channel values remain pairwise distinct, and none may equal the AUTHIFI_ADMIN_RESOURCE derived from the admin token; the API refuses to start when two of them match or when one equals OIDC_ISSUER. Operator commands that still consume OIDC_AUDIENCE also validate it separately.
  • OIDC_JWKS_URI — JWKS from the issuer's discovery document (jwks_uri). Used for BFF step-up and the console Authifi admin token slot. Optional when OIDC_ISSUER is set: Compose and the host-Node launchers derive {OIDC_ISSUER}/.well-known/jwks.json. PORTAL_OIDC_JWKS_URI remains a one-release alias. Deploy fetches discovery and fails if the committed value disagrees.
  • OIDC_CLAIM_UPSTREAM_ISSUER, OIDC_CLAIM_UPSTREAM_ID — claim names the IdP uses for upstream (federated) identity. For Authifi these are identityIssuer and email.
  • CADDY_ADMIN_URL — enables automated TLS cert recovery via Caddy's admin API

Why some ci.env keys are also copied into ~/floh/.env: the server container mounts config/public and reads ci.env directly, but the BFF services have no env_file — they interpolate from host --env-file flags (.env, env/console.env, env/portal.env) plus their compose environment: block. The workflow therefore resolves OIDC_ISSUER, PORTAL_OIDC_CLIENT_ID, OIDC_AUDIENCE, FLOH_CONSOLE_AUDIENCE, FLOH_PORTAL_AUDIENCE, FLOH_MCP_AUDIENCE, and OIDC_JWKS_URI out of ci.env and writes them into ~/floh/.env as well. Adding a new ${VAR} reference to a BFF service without extending that list (or the matching process file) yields an empty value, because Compose substitutes unset variables silently. packages/portal-bff/test/config-invariants.test.ts and console-config-invariants.test.ts fail the build when the lists drift.

Config precedence: When public config files are loaded (APP_ENV is set and PUBLIC_CONFIG_DIR points to the config directory), the server reads non-secret settings only from base.env / ci.env. Values in process.env (including those from Docker Compose env_file) are not consulted for non-secret keys unless ALLOW_LEGACY_ENV_NON_SECRET=true is set, which re-enables process.env as a fallback for migration or back-compat. See readNonSecret in packages/server/src/config/index.ts for the implementation. In normal operation, all non-secret OIDC and runtime settings should go in the public config files.

TRUST_PROXY (in config/public/base.env) should only be enabled when the proxy layer strips/overwrites untrusted X-Forwarded-* headers. If misconfigured, clients can spoof source IPs and impact rate-limiting and audit/access-log attribution.

GitHub Actions Node Version

CI and release automation resolve Node from the repository root .nvmrc via actions/setup-node node-version-file, so local nvm use, PR CI, and release tagging all use the same toolchain selection. The package compatibility floor remains node >=24.0.0; do not introduce Node 26-only package APIs just because GitHub Actions currently runs Node 26.

Rollback. If Node 26 exposes a tooling regression, restore .nvmrc to the previous Node 24 value and rerun CI or release. No workflow YAML edit is needed for that rollback path because the workflows already read .nvmrc.

Authifi client registration

Registering the OIDC clients and their resource servers, upgrading or migrating an existing tenant, the platform access rollout, the Floh roles catalog, and the console Authifi sync are described in Auth Configuration.

Authifi tenant-secrets vault

By default the server reads its secrets from the environment, which means every value in the Secrets table lives in GitHub. Setting the SECRETS_BACKEND variable to authifi switches the server to fetching them from the Authifi tenant-secrets vault at startup instead. See secrets management for the provider design.

Three properties of this switch drive everything below:

  • It is a total replacement, not a fallback. AuthifiSecretProvider serves only what the vault returns, and loadConfig resolves secrets through the provider alone — it does not consult process.env behind it. This matters because the deploy still writes the GitHub secret copies into ~/floh/.env: without that guarantee, a vault missing a key would boot on the stale environment value, cutover verification would falsely succeed, and a later rotation in the vault alone would keep using the old credential. A missing required secret instead resolves to its default, which validateProductionSecrets rejects under NODE_ENV=production, so a partial migration fails the boot rather than degrading quietly.
  • It covers the server only. postgres, redis, and portal-bff read ~/floh/.env directly and have no vault client, so their secrets stay in GitHub regardless of this setting.
  • It is a runtime switch, not a build-time one. Flipping the variable and re-running the deploy is the whole cutover, and flipping it back is the whole rollback.

Cutover

  1. Create the tenant secrets. Because the vault is the only source the server consults, every secret it reads must exist there before the switch, prefixed with FLOH_. The full set is SECRET_KEYS in packages/shared; copy each value from the existing GitHub secret rather than generating a new one.
Tenant secret Required
FLOH_DB_PASSWORD yes
FLOH_JWT_SECRET yes
FLOH_SESSION_SECRET yes
FLOH_SESSION_ENCRYPTION_KEY yes
FLOH_AUDIT_CHECKPOINT_KEY yes
FLOH_CONNECTOR_ENCRYPTION_KEY yes
FLOH_OIDC_CLIENT_SECRET yes
FLOH_PORTAL_OIDC_CLIENT_SECRET yes
FLOH_REDIS_PASSWORD when Redis auth is enabled
FLOH_SMTP_USER when SMTP auth is in use
FLOH_SMTP_PASS when SMTP auth is in use
FLOH_CONNECTOR_ENCRYPTION_KEY_PREVIOUS whenever the GitHub copy is set
FLOH_AUDIT_CHECKPOINT_KEY_PREVIOUS whenever the GitHub copy is set

Re-generating rather than copying CONNECTOR_ENCRYPTION_KEY leaves stored connector credentials undecryptable, and re-generating AUDIT_CHECKPOINT_KEY breaks audit-chain verification.

The two _PREVIOUS keys are the easiest to miss and the quietest to get wrong. They exist only during a key rotation, they are optional, and nothing validates them at startup — so a cutover performed mid-rotation that omits them boots cleanly and then cannot decrypt connector credentials written under the old key, or verify historical audit checkpoints. If either GitHub secret is currently set, carry it across before flipping the switch.

  1. Register the vault client. It authenticates with private_key_jwt, so it needs a key pair rather than a client secret. The setup script handles the whole exchange — do not generate the key by hand:

This run also reconciles the console and portal clients, whether or not you are touching them. Reconciliation is additive: every registered callback and post-logout URI is preserved, including the developer localhost callbacks on a tenant shared with local development, except a callback whose parsed pathname is exactly /api/auth/callback or /api/auth/callback/. A bare --vault therefore cannot silently drop the localhost callbacks and break local login. Preview with --dry-run first — it lists any required additions separately from that one known-deprecated removal class, before the first write:

pnpm run setup-authifi-oidc-clients -- --vault --dry-run \
  --console-origin https://floh.authilize.com \
  --portal-origin https://myfloh.authilize.com

# same command without --dry-run once the diff shows no unexpected removals

This registers floh-vault-client with private_key_jwt and client_credentials, has Authifi mint the JWK and register the public half, and writes the private key to ~/.floh/vault-key.pem at mode 0600 (override with --vault-key-file; the script refuses any path inside the repo).

Two things that look like they should be steps here and are not. --vault is required — without it the script plans no vault client at all. And --print-commands returns before the key exchange, so it can show you the client-creation recipe but can never register a JWK; use it to preview, not to execute. The SECRETS_MANAGER.LIST and SECRETS_MANAGER.PLAIN_SECRET scopes are not granted on the client either — they are requested at token time and already default correctly in vault-config.ts.

  1. Store the private key, and keep the local copy:
gh secret set AUTHIFI_VAULT_PRIVATE_KEY --repo Authifi/floh --env dev < ~/.floh/vault-key.pem

Do not delete ~/.floh/vault-key.pem afterwards. Its existence is the script's entire rotation guard: ensureVaultKey decides whether a run would be a rotation by checking whether that path exists, and POST .../jwk replaces the registered key immediately. Delete the file and the next --vault run mints a replacement without warning, while GitHub still holds the superseded key — the server then cannot authenticate to the vault on its next restart. Keep it in a password manager or another encrypted store, and treat a deliberate rotation as "move the file aside, re-run, upload the new key" in one sitting.

  1. Flip the switch. Set the SECRETS_BACKEND variable to authifi and re-run the deploy.

  2. Confirm the source. The server logs the backend it resolved as its first startup line, and a vault failure aborts the boot with Fatal: Floh server failed to start rather than falling back:

docker compose -p floh -f docker-compose.deploy.yml \
  --env-file .env --env-file env/console.env --env-file env/portal.env \
  logs server | grep -i 'Loading secrets from'

The worker entrypoint reports the same line, but docker-compose.deploy.yml defines no worker service (only the dev docker-compose.yml does), so there is nothing to query for it here.

The deploy rejects SECRETS_BACKEND=authifi when AUTHIFI_VAULT_PRIVATE_KEY is unset, and rejects any value other than env or authifi — an unrecognized value would otherwise fall back to env at runtime and leave the vault quietly disabled. The non-secret locators (AUTHIFI_VAULT_URL, _TENANT, _TENANT_ID, _CLIENT_ID, _KEY_FILE) live in config/public/ci.env; the private key is written to ~/floh/vault-key.pem at mode 0600 and bind-mounted, never into .env.

DB_PASSWORD must be kept in both places — and neither is where the database actually keeps it. The postgres container gets its password from ~/floh/.env, so FLOH_DB_PASSWORD in the vault and DB_PASSWORD in GitHub have to hold the same value. Rotating one without the other leaves the server unable to authenticate against a database it can still see. Rotating both is still not enough on an existing volume, because POSTGRES_PASSWORD is honored only at initialization and the live floh role keeps its original password; follow Rotating DB_PASSWORD, which adds the ALTER ROLE. PORTAL_OIDC_CLIENT_SECRET is not a second instance of this rule: the API server never reads it, so the vault copy is inert. It reaches the portal BFF from the GitHub secret via ~/floh/env/portal.env as AUTH_CLIENT_SECRET, and rotating the GitHub secret is what takes effect. Nothing detects a DB_PASSWORD divergence at deploy time — it surfaces as an authentication failure at startup.

Rollback

Set SECRETS_BACKEND back to env (or unset it) and re-run the deploy. This works only while the GitHub secrets are still populated, which is why the cutover copies values into the vault rather than moving them. Retiring the GitHub copies is a separate decision to make after the vault path has held.

Deploying

  1. Go to Actions → Deploy
  2. Click Run workflow
  3. Choose the target. dev is the existing CI host. qa is the separate QA host.
  4. Optionally specify a branch or tag (defaults to main)
  5. The workflow publishes the 5 Floh images for that commit when :<sha> is not already in GHCR, then the self-hosted runner for that target pulls :<sha>, syncs config, and starts services. A ref that does not contain docker/render-runtime-config.sh is not rebuilt. CI pulls that ref's :<sha>. QA pulls that ref's :<sha>-qa. The origin preflight is skipped for that ref.

The build matrix produces:

Image Role
ghcr.io/.../floh/server Floh API (Node).
ghcr.io/.../floh/web Floh admin SPA (nginx). Serves /runtime-config.json. The digest does not contain a hostname.
ghcr.io/.../floh/portal-web Self-service portal SPA (nginx). Serves /runtime-config.json for the console origin. The digest does not contain a hostname.
ghcr.io/.../floh/form-builder-app Standalone Form Builder SPA (nginx). Iframed by the admin SPA from DEPLOY_FORM_BUILDER_DOMAIN.
ghcr.io/.../floh/mcp Floh MCP server (stdio). Pull-only artifact — operators / MCP clients run it via docker run --rm -i …. Not registered as a service in docker-compose.deploy.yml.

The portal BFF is not a Floh-built image. Since LSA-9825 it is the upstream Authifi gateway, pinned in docker/docker-compose.deploy.yml:

Image Role
ghcr.io/authifi/idbroker-tools/bff-gateway:3.10.1 Portal OIDC relying party and API proxy. Configured by docker/bff/portal.json (mounted at /app/config/local.json) plus compose env vars.

The deploy runner therefore needs pull access to the Authifi/idbroker-tools package scope, not just Authifi/floh. The version pin is deliberate and is asserted by packages/portal-bff/test/config-invariants.test.ts; read the comment above BFF_VERSION there before bumping it.

First deploy takes ~5 minutes (image pulls). Subsequent deploys are faster due to layer caching.

Running the MCP server

The MCP server speaks stdio, so MCP clients invoke it on demand rather than pinning it to a port. After the deploy publishes the image, an MCP-aware client (e.g. Claude Desktop, Cursor) can pull and run it directly. Deploy derives the GHCR prefix from GITHUB_REPOSITORY (lowercased); the compose default and the example below match the current Authifi org:

docker run --rm -i \
  -e FLOH_API_URL="https://floh.authilize.com/api" \
  -e OIDC_ISSUER="..." -e OIDC_CLIENT_ID="floh-mcp-client" \
  -e OIDC_CLIENT_SECRET="..." \
  -e FLOH_MCP_AUDIENCE="..." \
  -e FLOH_REFRESH_TOKEN="..." \
  ghcr.io/authifi/floh/mcp:latest

floh-mcp-client is registered with client_secret_post, so the secret is not optional: without it Authifi rejects the refresh request before any tool runs. Use the value setup captured into env/mcp.env as MCP_OIDC_CLIENT_SECRET (either env key works in the MCP process; MCP_OIDC_CLIENT_SECRET wins).

The Deploy workflow writes IMAGE_PREFIX into ~/floh/.env so host pulls track the publishing org without editing compose. The host .env sets IMAGE_TAG to the deployed SHA. CI also moves :latest and, on main, :<version>, for clients that pull those aliases. The running stack does not. There is no per-environment image tag. /runtime-config.json is a JSON file the SPA fetches. It is not an inline script, and its path is not configurable. Knowledge-index URLs stay compiled into the portal bundle. The workflow target stays dev or qa. The same-origin check still runs in the deploy job after the image push. Local docker-compose.yml does not inject these origins; ng serve remains the local SPA.

MCP authentication is OIDC refresh-token exchange only (FLOH_REFRESH_TOKEN + OIDC_ISSUER + OIDC_CLIENT_ID=floh-mcp-client or MCP_OIDC_CLIENT_ID). Do not put the MCP secret in env/console.env. See packages/mcp/src/index.ts for the full env-var contract.

Operations

Running compose commands on the host

Every compose command on the deployed host needs the project name and all three env files. Define this helper once per shell session and use it for the commands below:

ssh -i your-key.pem ubuntu@<ELASTIC_IP>
cd ~/floh
compose() {
  docker compose -p floh -f docker-compose.deploy.yml \
    --env-file .env --env-file env/console.env --env-file env/portal.env "$@"
}

Omitting the --env-file flags is not cosmetic. Compose interpolates every unresolved ${VAR} to an empty string, so:

  • compose up without them recreates containers with empty configuration — including empty SERVER_CERT / SERVER_KEY on the BFFs, which reproduces the ERR_OSSL_PEM_NO_START_LINE crash loop on a stack that was healthy a moment earlier. This is the dangerous case, and it extends to any other command that creates or recreates containers (run, create).
  • restart and stop are safe but noisy. Neither recreates a container, so the empty interpolation never reaches the running config — they act on the containers exactly as up last created them. You still get the warnings below, which is why the habit is worth keeping.
  • Read-only commands (ps, logs) emit misleading warnings, e.g. The "AUTHIFI_ADMIN_TARGET" variable is not set. Defaulting to a blank string. Those warnings say nothing about the running containers, which were created with the values intact. This one wasted real debugging time; see the failure reference below.

Omitting -p floh is equally load-bearing: compose derives the project name from the directory otherwise, and commands then target a different (usually empty) set of containers than the deploy created.

Service healthchecks

The deploy polls every compose service that defines a HEALTHCHECK (GATED in .github/workflows/deploy.yml). form-builder — and any future nginxinc/nginx-unprivileged service — must probe http://127.0.0.1:8080/, not localhost. BusyBox wget (the alpine image's wget, BusyBox v1.37+) prefers IPv6 for localhost, and the nginx configs listen on IPv4 :8080 only, so a localhost probe stays unhealthy while Caddy is happily serving the SPA. See floh#1109.

View logs

ssh -i your-key.pem ubuntu@<ELASTIC_IP>
cd ~/floh
compose logs -f          # all services
compose logs -f server   # single service

Restart a service

compose restart server

Update secrets or deploy variables

Secrets and deploy variables are managed in the GitHub dev environment. Update values in Settings → Environments → dev and re-run the deploy workflow — the workflow writes ~/floh/.env, ~/floh/env/console.env, and ~/floh/env/portal.env from environment secrets/vars on every deploy.

That is the whole procedure for most secrets, but four of them break something that the deploy will still report as successful — DB_PASSWORD, CONNECTOR_ENCRYPTION_KEY, AUDIT_CHECKPOINT_KEY, and the OIDC client secrets. Check Rotating an existing secret before changing any of those.

Update non-sensitive runtime config

Edit config/public/ci.env (or base.env) in the repo, merge to the deploy branch, and re-run the deploy workflow. The workflow copies these files to ~/floh/public/, and docker/bff/portal.json to ~/floh/bff/.

Editing a key that a BFF service reads is a two-place change: add it to ci.env and to the read-env-values.mjs invocation (or the matching process file write) in .github/workflows/deploy.yml, since those containers do not mount config/public.

Non-secret keys (OIDC settings, URLs, feature flags) should be added to the public config files rather than GitHub environment variables or .env. When public config is loaded, process.env is only consulted as a fallback if ALLOW_LEGACY_ENV_NON_SECRET=true is set (see packages/server/src/config/index.ts). The NON_SECRET_KEYS list in that file defines which keys follow this rule.

Token revocation checks

The API can ask Authifi whether a verified access token has been revoked, so ending an Authifi session cuts off Floh API access within seconds rather than at the token's expiry. What is checked, and when, is described in Security § Token revocation. The settings are non-secret, so set them in config/public/ci.env and redeploy as in Update non-sensitive runtime config.

Key Default Accepted values
FLOH_TOKEN_REVOCATION_MODE off off, observe, enforce
FLOH_REVOCATION_WRITE_MAX_AGE_SECONDS 0 whole number, 0–60 (0 checks every write)
FLOH_REVOCATION_CHECK_TIMEOUT_MS 2000 whole number, 250–5000

With observe or enforce, OIDC_ISSUER must be https; the server refuses to start otherwise, because each check sends the caller's bearer token to Authifi.

A key that is set to anything else — Enforce, 60s, a value out of range — stops the server at startup with a message naming the key. An unset or blank key takes its default.

Roll out in two deploys:

  1. Set FLOH_TOKEN_REVOCATION_MODE=observe. The API makes the checks but never rejects. In the server logs, look for three warn reasons:
  2. would_reject — a revoked token was used; in enforce mode this request would have received 401.
  3. revocation_status_unavailable — Authifi or Redis did not answer, and detail says why (timeout, network, no_session, unexpected_status, cache_read_failed, cache_connection_failed, …). A steady stream of no_session from one azp means that channel's tokens have no bound Authifi session, so revocation is never enforced for it.
  4. token_without_jti — logged once per process if tokens carry no jti; their status is then checked on every request and never cached.
  5. Set FLOH_TOKEN_REVOCATION_MODE=enforce.

Alert on the rate of revocation_status_unavailable. The API allows a request when it cannot get a status, so while that warning persists, a revoked token that is not already cached as revoked keeps working until it expires.

To roll back, set FLOH_TOKEN_REVOCATION_MODE=off and redeploy. There is no migration; the Redis entries under floh:revocation:* expire on their own.

Connector key rotation runbook

Use this when changing CONNECTOR_ENCRYPTION_KEY without breaking existing connector secrets.

  1. Generate a new 64-char hex key:
openssl rand -hex 32
  1. In the GitHub dev environment secrets, set:
  2. CONNECTOR_ENCRYPTION_KEY → the new key

  3. CONNECTOR_ENCRYPTION_KEY_PREVIOUS → the old key

  4. Run the deploy workflow to apply the change.

  5. Rotate connector secrets:

  6. UI: Connectors -> Rotate Keys
  7. API:
curl -X POST https://<DEPLOY_DOMAIN>/api/connectors/rotate-keys \
  -H "Authorization: Bearer <admin-token>"

  1. Verify the summary response has failed: [].

  2. Clear CONNECTOR_ENCRYPTION_KEY_PREVIOUS from the dev environment secrets and re-deploy.

Check service status

compose ps

Access MailHog

Open http://<ELASTIC_IP>:8025 in your browser (if port 8025 is open in the security group).

Database access

compose exec postgres psql -U floh -d floh

Failure reference

Deploy and first-login failures, keyed on the error text you actually see. Search this page for the literal string.

Error text Where it appears Cause
ERR_OSSL_PEM_NO_START_LINE BFF container logs Malformed TLS PEM secret
<service> did not become healthy after startup Deploy job Container crash-looping; the cause is in its logs
invalid_redirect_uri Browser, after login Callback URL not registered on the OIDC client
invalid_client (client authentication failed: invalid secret provided) Browser, after login OIDC_CLIENT_SECRET does not match the tenant
The "AUTHIFI_ADMIN_TARGET" variable is not set Deploy job Benign — a compose call missing --env-file
Stored session payload is incompatible with express-openid-connect BFF container logs Back-channel logout on a gateway older than 3.10.1

ERR_OSSL_PEM_NO_START_LINE

The BFF crash-loops at startup inside configSecureContext → setCerts. Node could not parse the PEM it was handed, which means the secret's content is wrong — not the file plumbing.

An empty value does not produce this error, so emptiness is ruled out when you see it. The three shapes that do produce it are a PEM whose newlines became literal \n escapes, a base64 encoding of the PEM, and a PEM wrapped in quote characters. All three come from capturing the secret with anything other than gh secret set <NAME> … < <file>.

Fix by re-setting the secret from the file and re-running the deploy:

gh secret set CONSOLE_BFF_TLS_CERT --repo Authifi/floh --env dev < console.crt
gh secret set CONSOLE_BFF_TLS_KEY  --repo Authifi/floh --env dev < console.key

The deploy preflight now rejects all three shapes before writing any host file, so this should surface as a named preflight error rather than a crash loop. See Deploy preflight for TLS secrets.

A second cause with the same signature: running compose up on the host without the --env-file flags, which recreates the container with an empty certificate. (restart cannot cause this — it does not recreate the container.) See Running compose commands on the host.

<service> did not become healthy after startup

The health gate polled the service for 120s and it never reported healthy or running. A crash-looping container sits in restarting and never reaches a terminal state, so this timeout — not a terminal-state error — is what reports it. The deploy prints the last 50 log lines for the failing service immediately after this message; the real cause is there, not in the gate message.

invalid_redirect_uri

redirect_uri did not match any of the client's registered redirect_uris. The browser reaches the IdP, so client ID and issuer are correct; only the callback URL is unregistered.

The console's RP is the console BFF, so the callback is https://<DEPLOY_DOMAIN>/bff/callback — not the SPA origin. The portal's is https://<DEPLOY_PORTAL_DOMAIN>/bff/callback. Re-register with the canonical client definitions rather than hand-editing:

pnpm run setup-authifi-oidc-clients

See Authifi client registration. Any domain change requires this step; it is easy to miss because nothing else fails.

invalid_client (client authentication failed: invalid secret provided)

The redirect worked and the BFF is exchanging the code, so this is the client secret alone: the value in OIDC_CLIENT_SECRET (or PORTAL_OIDC_CLIENT_SECRET) is not the secret the tenant holds for that client.

The usual cause is drift — the tenant's client was recreated or its secret rotated, while the GitHub environment kept the old value. Authifi cannot re-read an existing client secret, so there is no way to look up the current value: you either hold a copy, or you rotate and capture the new one.

# Check when the secret was last touched — a date long before the tenant's
# client was created is the tell.
env -u GITHUB_TOKEN gh secret list --repo Authifi/floh --env dev

# If you hold a verified copy (e.g. a working local env/console.env):
env -u GITHUB_TOKEN gh secret set OIDC_CLIENT_SECRET --repo Authifi/floh --env dev < secret.txt

# Otherwise rotate and capture the minted value:
pnpm run setup-authifi-oidc-clients -- --rotate-secrets

Rotating invalidates any other copy of that secret. See Rotating an existing secret.

The "AUTHIFI_ADMIN_TARGET" variable is not set

level=warning msg="The \"AUTHIFI_ADMIN_TARGET\" variable is not set. Defaulting to a blank string."

Benign. This is compose interpolation warning about its own invocation, not a statement about the running containers. It appears when a ps or logs call omits the --env-file flags that the up call used, so the variable really was set when the containers were created.

It is worth knowing because the warning repeats once per poll during a health gate, which makes it look like the cause of whatever is failing. It is not. The deploy workflow's read-only compose calls now pass the env files, so these warnings should no longer appear; if you see them from a manual command, add the flags.

Stored session payload is incompatible with express-openid-connect

Every request from one browser returns 500, including /bff/login and /bff/logout, so the user cannot sign in again until they clear that site's cookies. Other browsers are unaffected.

It follows a back-channel logout. When a user's Authifi session ends — signing in as a different account in the Authifi console is enough — Authifi sends a logout token to every client that used that session. The gateway records it as a marker in Redis, and gateways older than 3.10.1 then fail to read their own marker on every later request carrying that session cookie.

Fix by running bff-gateway:3.10.1 or later for both portal-bff and console-bff. From 3.10.1 the browser lands on the console's "Session Ended" screen and can sign in again. 3.10.1 also logs unreadable_session_payload when it discards a session record it cannot read, and backchannel_logout_check_failed when the logout lookup itself fails; both sign the browser out rather than returning 500.

3.10.1 also caps proxied WebSockets at 5 per session and closes each after an hour. The console opens one per run-execution view and reconnects with backoff, so a long-open view just reconnects, and a sixth concurrent view keeps retrying until another closes. To roll back, revert the image tags in docker/docker-compose.deploy.yml and redeploy; no config change goes with them.

Changing Domains Later

  1. Update DNS A records for the new domain
  2. Update DEPLOY_DOMAIN and/or DEPLOY_PORTAL_DOMAIN in the GitHub dev environment variables, and/or DEPLOY_FORM_BUILDER_DOMAIN in the repository-level variables (Settings → Variables → Actions)
  3. Update FRONTEND_URL, PORTAL_FRONTEND_URL, OIDC_REDIRECT_URI, ALLOWED_ORIGINS, and ALLOWED_PORTAL_ORIGINS in config/public/ci.env and merge
  4. Re-register the OIDC clients against the new origins (see Authifi client registration) — the console callback and the portal /bff/callback both carry the domain
  5. Run the deploy workflow
  6. Caddy auto-provisions new Let's Encrypt certificates

A domain change does not require a new image. Redeploy the same ref after the public env file and the GitHub variables change.

QA environment

QA is a second copy of the CI stack, not a change of platform. It uses the same Compose file (docker/docker-compose.deploy.yml): Caddy, server, web, portal-web, form-builder, portal-bff and console-bff (both bff-gateway:3.10.1), Postgres 16, Redis 7, and MailHog. ClamAV stays optional. The MCP image is published and is not a running service.

CI (GitHub environment dev) QA (GitHub environment qa)
Runner labels self-hosted, deploy self-hosted, deploy-qa
Public config config/public/ci.env (APP_ENV=ci) config/public/qa.env (APP_ENV=qa)
Image tags :<sha>. CI also moves :latest and, on main, :<version>. :<sha> of the same commit. QA does not move :latest.
Authifi tenant https://a-ci.ncats.io/_api/auth/ls https://a.qa.authifi.io/_api/auth/floh
Admin / portal / forms floh.authilize.com, myfloh.authilize.com, forms.authilize.com floh-qa.authifi.io, my.floh-qa.authifi.io, forms.floh-qa.authifi.io
Log group /floh/floh.authilize.com /floh/floh-qa.authifi.io

Dispatch Deploy with target qa. A default dispatch stays on CI.

Both hosts pull :<sha> for a commit that contains the runtime-config renderer. QA does not publish a second tag for that commit and does not move :latest. A QA deploy of a SHA that CI already published does not rebuild. Caddy and the BFFs still use qa.env hostnames (FRONTEND_URL, PORTAL_FRONTEND_URL, and DEPLOY_FORM_BUILDER_DOMAIN). They do not fall back to the CI repository variables. A ref without docker/render-runtime-config.sh keeps the tag that ref published: :<sha>-qa on QA and :<sha> on CI. The workflow does not build that ref when the tag is missing.

config/public/qa.env names the three hostnames and the Authifi QA issuer. Secrets are generated for the qa environment. Do not copy them from dev.

What has to exist before the first green QA run

  1. A second EC2 host, security group, and Elastic IP. Postgres, Redis, and ClamAV listen only on the private network. Public ingress is 80 and 443. MailHog's UI (port 8025) is restricted by IP. Register a runner with labels self-hosted and deploy-qa. The CI runner must not be given the deploy-qa label.
  2. DNS for floh-qa.authifi.io, my.floh-qa.authifi.io, and forms.floh-qa.authifi.io pointing at that Elastic IP.
  3. A GitHub environment named qa. Use the same secret and variable names as dev (see Secrets and Variables), except QA_PLATFORM_ACCESS_GROUP_IDS (item 6), QA_FILE_SCREENING_* (item 8), and QA_SECRETS_BACKEND. Setting SECRETS_BACKEND on qa leaves the server on env secrets. Set QA_SECRETS_BACKEND there only when QA should use the Authifi vault. The admin, portal, and forms hostnames for QA come from config/public/qa.env, not from DEPLOY_DOMAIN, DEPLOY_PORTAL_DOMAIN, or DEPLOY_FORM_BUILDER_DOMAIN.
  4. New private-CA leaves for portal-bff and console-bff. The portal SAN is my.floh-qa.authifi.io. The console SAN is floh-qa.authifi.io.
  5. Floh clients and resource servers on the Authifi QA Floh tenant, using the first-time bring-up checklist with config/public/qa.env in place of ci.env. The issuer is https://a.qa.authifi.io/_api/auth/floh. OIDC_CLIENT_SECRET and PORTAL_OIDC_CLIENT_SECRET are both required on the qa environment.
  6. The Floh roles map for the QA tenant, once groups exist there. Run pnpm run sync-authifi-rbac -- --emit-group-ids against that tenant and store the value as QA_PLATFORM_ACCESS_GROUP_IDS on the GitHub qa environment. Leave the repository variable PLATFORM_ACCESS_GROUP_IDS for CI. A QA deploy does not read it. Until the QA map is set, the overlay stays unset and the console catalog stays empty.
  7. AUTHIFI_VAULT_TENANT_ID in config/public/qa.env is 8, the Floh tenant on a.qa.authifi.io. The GitHub qa variable AUTHIFI_ADMIN_TARGET is https://a.qa.authifi.io/_api/auth/admin/tenants/8. CI keeps tenant 1 for slug ls on a-ci.ncats.io.
  8. QA file screening, only if QA should scan uploads. Set QA_FILE_SCREENING_MODE and the related QA_FILE_SCREENING_* variables on the qa environment. A QA deploy does not read the repository FILE_SCREENING_* variables, so it cannot point at CI's scanner. Unset leaves screening disabled.

Cost

Same shape as CI, so the same monthly cost again: about $33 for a t3.medium, $2.50 for 30 GB gp3, and $1–5 of data transfer. Roughly $37–41, on top of the CI host.

Rollback

Re-run Deploy with the previous ref. A ref that contains the runtime-config renderer pulls :<sha> and does not rebuild when that tag exists. A ref that does not contain it pulls :<sha>-qa and is not rebuilt. The Postgres volume on the QA host is kept across redeploys. Do not retarget the QA DNS names at the CI host, and do not dispatch target dev to undo a QA release. That workflow runs on the CI runner.

Cost Estimate

Resource Monthly Cost
EC2 t3.medium ~$33
EBS 30 GB gp3 ~$2.50
Elastic IP (attached) Free
Data transfer ~$1-5
GHCR Free (included in GitHub plan)
Total ~$37-41