Cicd Plan

CI/CD — dev and prod environments

Status: implemented, awaiting the manual setup steps below. The workflows and Terraform are committed but nothing has run yet: the dev environment does not exist until someone applies Terraform, and the deploy workflows will fail until their AWS and Apple secrets exist.

Environment model

Flow: feature branch → PR into dev (CI must pass) → merge deploys the dev server → PR dev → main → merge deploys production and uploads to TestFlight. No buttons, no tags — the merge is the release. workflow_dispatch exists on both deploy workflows purely to re-run a deploy without a code change.

Decisions behind this shape

1. Dev borrows production's RDS instance with a separate steelnotes_dev database rather than provisioning a second one (~$15/month saved). create_db_instance = false in the dev workspace.

2. No approval gates. Environment protection rules need GitHub Pro/Team on a private repo, and a button to click is still manual work. Automatic-on-merge with branch protection in front is both cheaper and less friction.

3. Tests gate every deploy — not just PRs. The suites live in reusable workflows that both CI and the deploy workflows call, so a direct push that never saw a PR still cannot deploy a red build.

4. iOS never runs in CI. macOS runners bill at a 10x multiplier and were the bulk of the spend. The simulator suite is a pre-push manual step, and no automated check gates iOS code — not on PRs, not on the TestFlight release. _test-ios.yml is dispatch-only, for when a build looks suspect.

5. Terraform stays manual. CI only swaps Lambda code.

What exists

Notable implementation details, because they are not obvious from the files:

Manual steps — nothing works until these are done

M1. Branch protection (do this first)

Settings → Branches → add a rule for main, and optionally dev:

Since merge-to-main deploys straight to production, this rule is what stands between a broken change and prod.

M2. Dev database

On the existing RDS instance:

```sql

CREATE DATABASE steelnotes_dev;

CREATE USER steelnotes_dev WITH PASSWORD '<generate one>';

GRANT ALL PRIVILEGES ON DATABASE steelnotes_dev TO steelnotes_dev;

```

Migrations run from the binary on first boot, so there is no schema step.

M3. Dev Terraform workspace

```bash

cd server/deploy/terraform

cp dev.tfvars.example dev.tfvars

fill in: database_url, and fresh secrets from openssl rand -hex 32⧉

jwt_secret, path_hmac_key, path_enc_key — never reuse production's

terraform workspace new dev

terraform plan -var-file=dev.tfvars

```

Review the plan, then apply. This creates the dev Lambdas, S3 bucket, SQS/SNS, log groups, API Gateway, and the ACM certificate.

**Incident, 2026-08-02.** The first attempt at this step ran without -var-file=dev.tfvars. Every variable fell back to its default, environment defaulted to "production", and Terraform planned to build production inside the dev workspace's empty state. Most resources collided and errored, but SNS CreateTopic and SQS CreateQueue are idempotent — they returned the **existing production** topic and queues, which Terraform then recorded in dev's state as if it owned them. The follow-up apply *with* the var file saw those resources needed -dev names, and since renaming an SNS topic or SQS queue forces replacement, it destroyed production's live topic and both queues.

Recovery was a terraform apply in the default workspace: SNS and SQS ARNs are derived from account, region, and name, so recreating them under the same names restored byte-identical ARNs and queue URLs and no Lambda configuration had to change.

The fix is structural, not procedural: environment-shaped values now derive from terraform.workspace, so a forgotten -var-file cannot resolve to production names. Two lessons worth keeping — a partial, *failed* apply can still adopt live resources into state, and idempotent create APIs are what make that silent.

M4. DNS for api-dev.steelnotes.app

steelnotes.app is on Cloudflare, so both records below must be DNS only (grey cloud), not proxied. A proxied record makes Cloudflare answer with its own IPs instead of the value you entered: ACM never sees its validation token, and API Gateway never sees a matching SNI. Production's api.steelnotes.app is already configured this way — match it.

Two records, in order:

1. ACM validation, a CNAME from terraform output acm_validation_records. Add it, then apply — aws_acm_certificate_validation.api waits up to 30 minutes for ACM to see it, so the rest of the apply proceeds in the same pass.

2. The domain itself, a CNAME from terraform output api_domain_target, available only after the API Gateway domain resource exists. Point api-dev.steelnotes.app at that d-xxxx.execute-api.us-east-1.amazonaws.com value.

Check a record is live before waiting on it:

```bash

dig +short CNAME <the-record-name> @8.8.8.8

```

Empty output means it is not published or not propagated. If it returns Cloudflare IPs rather than the value you entered, the proxy is on — switch that record to DNS only.

M5. GitHub Actions deploy roles

The roles are defined in github-oidc.tf but only exist once production is applied:

```bash

terraform workspace select default

terraform plan -var-file=terraform.tfvars # expect: 2 IAM roles, 2 role policies

terraform apply -var-file=terraform.tfvars

terraform output github_deploy_role_arns

```

This assumes the account already has the GitHub OIDC provider — the plan-site workflow uses it, so it should. The config looks it up rather than creating it; if the lookup fails, create the provider for https://token.actions.githubusercontent.com with audience sts.amazonaws.com first.

Then add repository secrets (Settings → Secrets and variables → Actions):

M6. App Store Connect — deferred

Nothing else depends on this; the server pipeline works without it. The wider set

of launch requirements this belongs to is tracked as Phase 9 in

PLAN.md — this section is just the CI-credential slice.

**Not App Manager.** App Manager can upload builds but cannot manage signing assets, and cloud signing has to *create and download* the distribution certificate and profiles. An App Manager key archives fine and then fails at export with Cloud signing permission error followed by No profiles for '<bundle id>' were found for all three bundle IDs — the "no profiles" line is the symptom, not the cause. Confirmed the hard way on 2026-08-18 (run 32088036309).

This is also why a successful *local* export proves nothing about CI: locally xcodebuild authenticates as the Xcode-signed-in Apple ID (Account Holder), while CI authenticates as the API key. Two identities, two permission sets.

```bash

cd iosApp && xcodebuild archive -project SteelNotes.xcodeproj -scheme SteelNotes \

-configuration Release -destination 'generic/platform=iOS' \

-archivePath build/SteelNotes.xcarchive -allowProvisioningUpdates

```

M7. Point the dev app at the dev backend

Nothing to do — Debug builds already resolve to https://api-dev.steelnotes.app once M3/M4 land. Until then a Debug build will fail to reach a backend, which is the one behavior change developers will notice. To temporarily override, change API_BASE_URL for the Debug configuration in iosApp/project.yml and re-run xcodegen.

Remaining backlog

T1.3 — Make Rust lint blocking. cargo fmt --check⧉ currently reports ~220 diffs across 33 files and clippy a handful of warnings, so the lint job in ci.yml is continue-on-error: true. Run cargo fmt⧉, fix or #[allow] the clippy findings (mostly dead code in tests/common/mod.rs and doc-list indentation in src/bin/), then drop continue-on-error. Worth doing as its own PR — it is a large mechanical diff that would bury a review.

T2.5 — Verify the first dev deploy end to end. After M1–M5, push a trivial server change to dev and confirm both dev Lambdas update and api-dev.steelnotes.app answers /health.

T4.4 — Confirm extension signing. ✅ Resolved 2026-08-17. -allowProvisioningUpdates does cover the share and widget extensions; no pre-created profiles and no fastlane match. A local archive plus -exportArchive produced App Store profiles for all three bundle IDs and signed all three binaries with one Cloud Managed Apple Distribution certificate, sharing build number 1 and carrying beta-reports-active. The app group survived re-signing on both extensions.

Worth knowing for anyone debugging signing later: the archive itself is signed with an Apple Development identity — distribution re-signing happens during export, not archive, so an archive that says "Apple Development" is not a failure. And the cloud-managed certificate never appears in security find-identity; its private key stays on Apple's servers and is fetched transiently at export.

T4.5 — Stop the development-certificate churn. ✅ Resolved 2026-08-27. The asymmetry above has a cost that took ten deploys to surface. The cloud-managed distribution certificate is free to reuse because Apple holds its private key, but the development certificate the archive signs with is generated on the runner — and a fresh runner has an empty keychain, so every run asked Apple for a new one. They accumulated until the account hit its cap and every archive failed with Choose a certificate to revoke. Your account has reached the maximum number of certificates, followed by No profiles for '<bundle id>' were found for all three bundle IDs (again the symptom, not the cause — same misleading pairing as the App Manager key above).

deploy-ios.yml now imports one fixed development identity from the APPLE_DEVELOPMENT_CERT_P12 / APPLE_DEVELOPMENT_CERT_PASSWORD secrets into a throwaway keychain before archiving; the export path is unchanged and still cloud-signs. Profiles remain Apple's job, so adding a capability still costs nothing. The secrets are optional — absent them the job warns and reverts to the old churning behavior — so the workflow is safe to run before they are set. Rotation is annual, when the certificate expires; the workflow header has the export steps.

Possible later work, in rough priority order: run terraform plan in CI on PRs that touch server/deploy/terraform/**; add a smoke test against api-dev after the dev deploy; and, if macOS minutes become a problem, demote the per-PR iOS job to nightly-on-dev.

Sources