# Upgrades, backups and scaling

> Upgrade InfraWard, back up and restore it, scale the worker, rotate credentials and harden OpenBao.

All commands on this page run from the installation directory (`/opt/infraward` in the [installation guide](/docs/installation.md)).

## Upgrade

1. Read the release notes of the new version and [back up](#back-up) first.

2. Update the deployment files from the new release (`compose.yml`, `compose.openbao.yml`, `init.sh`, `openbao/config.hcl`). Keep your `.env`, `data/` and `secrets/`. Compare `.env.example` with your `.env` and add any new variables.

3. Set the new version in `.env`:

   ```ini title=".env"
   INFRAWARD_VERSION=v0.6.0
   ```

4. Run `init.sh` again. It creates any key, secret, AppRole or directory that the new version needs and leaves everything else as it is. With OpenBao it needs `secrets/openbao/init.json`: if you moved it off the server, put it back for this step and remove it afterwards.

   ```bash
   ./init.sh
   ```

5. Pull the images and recreate the containers:

   ```bash
   docker compose pull
   docker compose up -d
   ```

   The API applies database migrations when it starts. The worker and the gateway wait until the API is healthy.

6. Check `docker compose ps` and sign in.

> **Caution**
>
> Skipping step 4 is the most common upgrade failure. Versions that add a Compose secret (for example the Redis password or the terminal AppRole) fail with an error like `TERMINAL_OPENBAO_ROLE_ID es obligatoria (la escribe init.sh)` until `init.sh` has run.

Rolling back to an earlier version after its migrations have run is not supported. Restore the backup taken before the upgrade instead.

## Back up

| What | Where | How often | Notes |
|---|---|---|---|
| Database | `data/postgres` | Daily | Use a logical dump (below) rather than copying files of a running database. |
| OpenBao storage | `data/openbao` | Daily | Secrets, SSH CA and AppRoles. Stop OpenBao or take a filesystem snapshot for a consistent copy. |
| Keys and secrets | `secrets/` | After `init.sh` and after each rotation | **Separately from the database backup.** |
| Configuration | `.env`, `compose*.yml`, `openbao/config.hcl` | After each change | |
| Redis | `data/redis` | Optional | Only queues and locks. Pending jobs are rescheduled. |
| Trivy cache | `data/worker` | Never | Downloaded again on start. |

Dump the database:

```bash
docker compose exec -T postgres sh -c 'pg_dump -U "$POSTGRES_USER" -d "$POSTGRES_DB" -Fc' > infraward-$(date +%F).dump
```

> **Keep keys and data apart**
>
> `secrets/master.key` and `secrets/vault.master.key` decrypt the secrets stored by the local provider and the local vault. A backup that holds both the database and these keys gives away every stored secret. A database backup without the keys cannot be decrypted. Store them in different places with different access.

## Restore

1. Install the same InfraWard version on the new host following the [installation guide](/docs/installation.md), but **do not** run `init.sh` or start the stack yet.
2. Copy `.env`, `secrets/` and `data/openbao` from the backup.
3. Run `./init.sh`. It creates `data/postgres`, `data/redis` and `data/worker` and fixes ownership of everything.
4. Start PostgreSQL alone and load the dump:

   ```bash
   docker compose up -d postgres
   docker compose exec -T postgres sh -c 'pg_restore -U "$POSTGRES_USER" -d "$POSTGRES_DB" --clean --if-exists' < infraward-2026-10-27.dump
   ```

5. Start everything with `docker compose up -d`.

## Scale the worker

Run several worker replicas to scan more images in parallel:

```bash
docker compose up -d --scale worker=3
```

Or set `WORKER_REPLICAS=3` in `.env` so the number survives the next `up`. Replicas:

- share the queues without duplicating jobs;
- respect per-server limits across all replicas: `HOST_MAX_JOBS` (2) jobs at a time against one server, of which `HOST_MAX_SCAN_JOBS` (1) scans; a CT counts against its Proxmox node;
- share `data/worker`, so the Trivy database is downloaded once;
- elect one leader through Redis to run scheduled tasks. **Actividad** shows the connected replicas and which one runs each job.

When a replica stops, it waits up to 90 seconds (`WORKER_SHUTDOWN_GRACE`) for running jobs and returns the rest to the queue. Another replica picks them up.

Per-replica concurrency is set with `WORKER_CONCURRENCY`, `COLLECT_CONCURRENCY`, `DISCOVERY_CONCURRENCY`, `IMAGE_SCAN_CONCURRENCY` and the other variables in [Configuration variables](/docs/configuration.md#worker). Scanning is memory-hungry: add memory before raising `IMAGE_SCAN_CONCURRENCY`.

## Rotate credentials

| Credential | How to rotate |
|---|---|
| OpenBao AppRole `secret_id` (app, vault and terminal) | `./init.sh --rotate-secret-id`, then `docker compose restart api worker gateway`. |
| Administrator password | **Ajustes → Perfil → Seguridad**. Closes your other sessions. |
| SSH CA keys | After changing the CA issuer in OpenBao (`infraward-ssh/`), InfraWard flags every server whose installed CA keys are out of date. Use **Regenerar el comando** on each one and run the new command. |
| Cloudflare token | **Ajustes → Conectores → Reemplazar token**. |
| API tokens | Revoke and create new ones in **Ajustes → Tokens de API**. |

## Harden OpenBao

The example deployment unseals OpenBao automatically with `secrets/openbao/unseal.key`, which lives on the same host as the data. Anyone who can read the disk can read the secrets. For stronger guarantees, edit `openbao/config.hcl`:

- **Manual unseal**: remove the `seal "static"` block. After every restart, run `docker compose exec openbao bao operator unseal` with the unseal key.
- **External auto-unseal**: replace the block with a cloud KMS or the `transit` engine of another OpenBao or Vault.

OpenBao listens without TLS on the internal network only. To use an external OpenBao instead, remove `compose.openbao.yml` from `COMPOSE_FILE`, configure the AppRoles and the SSH engine as `init.sh` does, and set `OPENBAO_ADDR` and the AppRole variables for `api` and `worker` in an override file.

## Migrate from named volumes

Early deployments stored data in named volumes (`postgres-data`, `redis-data`, `worker-data`, `openbao-data`). To move to `data/`:

```bash
docker compose down
./init.sh
for v in postgres redis openbao; do
  docker run --rm -v infraward-stack_$v-data:/from:ro -v "$PWD/data/$v:/to" \
    docker.io/library/alpine sh -c 'cp -a /from/. /to/'
done
./init.sh
docker compose up -d
```

The second `init.sh` fixes ownership after the copy. Check that everything works before deleting the old volumes with `docker volume rm`. The Trivy cache does not need copying.

---

HTML page: https://infraward.net/docs/operations/