Sync Build Guide

Sync Build Guide — Actionable Implementation Plan

This document is the step-by-step build plan for cloud vault sync. Each section is scoped to be assignable to a single agent session. Read SYNC_ARCHITECTURE.md for the full design spec — this document tells you **what to build and in what order**.

Prerequisites

Before starting any sync work, understand these fundamentals:

1. .md files are the source of truth — SQLite (index.db) is never synced. It's rebuilt locally after pulling changes.

2. Sync state lives in .steel/sync.db — a separate SQLite database that tracks what's been synced, versions, and hashes.

3. File bytes never touch the API — clients upload/download directly to S3 via presigned URLs. The API is a thin coordinator.

4. The existing KMP shared module already has Ktor 3.0.3 — use it for the sync HTTP client.

How Local File Edits Get Synced

This is the most important flow to understand. When a user edits a .md file:

On iOS (app controls all writes)

1. User edits via the app → VaultManager.updateNote() writes .md to disk

2. After write, SyncManager.onLocalFileChanged(path) is called directly

3. 2-second debounce (user may still be editing)

4. ChangeDetector computes SHA-256 hash, compares against sync_state table

5. File marked pending_push in sync_state

6. SyncManager runs push: POST /vault/push → presigned S3 URL → upload bytes → POST /vault/push/confirm

7. Server updates file_state in Postgres, sends silent APNs push to other devices

8. Other devices receive push → SyncManager.syncNow() → pull the changed file

On macOS (user may edit files externally)

Same as iOS, plus:

On CLI

Full Sync Cycle (syncNow())

This is the catch-all. It detects ANY changes that individual triggers might have missed:

1. Scan all local .md files → compute hashes

2. Compare against sync_state table → find local changes

3. GET /vault/state?since={last_sync_time} → find remote changes

4. Classify each file: push-only, pull-only, conflict, or delete

5. Execute: upload/download via S3, delete propagation

6. Update sync_state for every resolved file

7. Trigger VaultIndexer for any changed/added/deleted files

8. Record last_sync_time

Build Order

Step 1: Rust Backend Skeleton

Scope: Get a Rust/Axum server that compiles, runs locally, and has a health endpoint.

Create these files:

```

server/

├── Cargo.toml

├── src/

│ ├── main.rs # Axum router, local server entry

│ ├── config.rs # Env-based config struct

│ ├── error.rs # AppError enum → HTTP response

│ └── routes/

│ ├── mod.rs

│ └── health.rs # GET /health → 200 OK

```

Cargo.toml dependencies:

Config (env vars):

```

DATABASE_URL=postgres://user:pass@localhost:5432/steelnotes

S3_BUCKET=steel-notes-vaults

S3_REGION=us-east-1

JWT_SECRET=<random-256-bit-key>

APPLE_TEAM_ID=<from-apple-dev-portal>

APPLE_CLIENT_ID=<bundle-id>

```

Acceptance criteria:

Step 2: Postgres Schema & Database Layer

Scope: Create the database tables and typed query functions.

Create migrations:

```

server/src/db/

├── mod.rs

├── models.rs # User, Device, FileState structs

├── queries.rs # Typed query functions

└── migrations/

├── 001_users.sql

├── 002_devices.sql

└── 003_file_state.sql

```

SQL (from SYNC_ARCHITECTURE.md):

001_users.sql:

```sql

CREATE TABLE users (

id UUID PRIMARY KEY DEFAULT gen_random_uuid(),

email TEXT UNIQUE,

provider TEXT NOT NULL, -- 'apple' | 'google'

provider_id TEXT NOT NULL,

created_at TIMESTAMPTZ NOT NULL DEFAULT now(),

vault_size BIGINT NOT NULL DEFAULT 0,

UNIQUE(provider, provider_id)

);

```

002_devices.sql:

```sql

CREATE TABLE devices (

id UUID PRIMARY KEY DEFAULT gen_random_uuid(),

user_id UUID NOT NULL REFERENCES users(id),

device_name TEXT NOT NULL,

platform TEXT NOT NULL, -- 'ios' | 'macos' | 'android'

push_token TEXT,

last_seen_at TIMESTAMPTZ NOT NULL DEFAULT now(),

created_at TIMESTAMPTZ NOT NULL DEFAULT now()

);

```

003_file_state.sql:

```sql

CREATE TABLE file_state (

user_id UUID NOT NULL REFERENCES users(id),

path TEXT NOT NULL,

version INTEGER NOT NULL DEFAULT 1,

content_hash TEXT NOT NULL,

size_bytes BIGINT NOT NULL,

updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),

updated_by UUID REFERENCES devices(id),

is_deleted BOOLEAN NOT NULL DEFAULT false,

deleted_at TIMESTAMPTZ,

PRIMARY KEY (user_id, path)

);

CREATE INDEX idx_file_state_deleted ON file_state(user_id, is_deleted, deleted_at);

CREATE INDEX idx_file_state_updated ON file_state(user_id, updated_at);

```

models.rs structs:

```rust

struct User { id: Uuid, email: Option<String>, provider: String, provider_id: String, created_at: DateTime<Utc>, vault_size: i64 }

struct Device { id: Uuid, user_id: Uuid, device_name: String, platform: String, push_token: Option<String>, last_seen_at: DateTime<Utc>, created_at: DateTime<Utc> }

struct FileState { user_id: Uuid, path: String, version: i32, content_hash: String, size_bytes: i64, updated_at: DateTime<Utc>, updated_by: Option<Uuid>, is_deleted: bool, deleted_at: Option<DateTime<Utc>> }

```

queries.rs functions needed:

Acceptance criteria:

Step 3: Auth Routes

Scope: Implement authentication endpoints and JWT middleware.

Create:

```

server/src/

├── routes/auth.rs # POST /auth/login, POST /auth/refresh

├── services/

│ ├── mod.rs

│ ├── apple_auth.rs # Validate Apple identity tokens (fetch JWKS, verify JWT)

│ └── tokens.rs # JWT creation (access + refresh), validation

├── middleware/

│ └── auth.rs # Tower layer: extract Bearer token, validate, inject UserId

```

Auth flow:

1. Client sends { "provider": "apple", "identity_token": "..." } to POST /auth/login

2. Server fetches Apple's JWKS from https://appleid.apple.com/auth/keys

3. Validates the identity token signature and claims (aud, iss, exp)

4. Extracts sub (provider_id) and email

5. Calls find_or_create_user("apple", sub, email)

6. Generates JWT access token (1hr TTL) with { user_id, device_id } claims

7. Generates opaque refresh token (stored in DB or signed, 30-day TTL)

8. Returns { access_token, refresh_token, user_id }

JWT middleware:

Acceptance criteria:

Step 4: S3 Presigned URLs & Vault Endpoints

Scope: Implement the core sync API endpoints.

Create:

```

server/src/

├── routes/vault.rs # All vault endpoints

├── routes/devices.rs # POST /devices/register

└── services/s3.rs # Presigned URL generation

```

Endpoints to implement (see SYNC_ARCHITECTURE.md for request/response schemas):

1. POST /devices/register — Register device + push token

2. GET /vault/state?since={ISO} — Return all files (or changed since timestamp) + deletions

3. POST /vault/push — For each file: check base_version against DB. If match → generate presigned PUT URL + increment version. If mismatch → return in conflicts[] with presigned GET URL for server version.

4. POST /vault/push/confirm — Update file_state rows, update vault_size, send push notifications to other devices

5. POST /vault/pull — Generate presigned GET URLs for requested file paths

6. POST /vault/delete — Version-check, then mark is_deleted=true + deleted_at=now()

S3 presigned URL generation:

```rust

// PUT (upload) — 5 min expiry

fn presigned_put(bucket: &str, key: &str) -> String

// GET (download) — 5 min expiry

fn presigned_get(bucket: &str, key: &str) -> String

```

S3 key format: {user_id}/{relative_path} (e.g., 550e8400.../sources/beyond-good-and-evil.md)

Acceptance criteria:

Step 5: Client — Auth & Sync Client (KMP)

Scope: Build the Kotlin client-side sync infrastructure in the shared module.

Create in shared/src/commonMain/kotlin/com/steelnotes/:

```

auth/

├── AuthManager.kt # Login flow, token refresh, logout

├── AuthModels.kt # AuthTokens, User data classes

└── TokenStore.kt # expect class — secure token storage

sync/

├── SyncClient.kt # Ktor HTTP wrapper for all API endpoints

├── SyncModels.kt # Data classes for API request/response DTOs

├── SyncStateDatabase.sq # SQLDelight schema for .steel/sync.db

├── ChangeDetector.kt # Hash local files, compare against sync_state

├── SyncManager.kt # Orchestrator — full sync cycle

└── SyncStatus.kt # SyncStatus enum (IDLE, SYNCING, OFFLINE, etc.)

```

Create in shared/src/appleMain/kotlin/com/steelnotes/:

```

auth/

└── AppleTokenStore.kt # Keychain-based secure token storage (actual)

```

SyncClient methods (thin Ktor wrappers):

```kotlin

class SyncClient(private val httpClient: HttpClient, private val baseUrl: String) {

suspend fun login(provider: String, identityToken: String): AuthTokens

suspend fun refreshToken(refreshToken: String): AuthTokens

suspend fun registerDevice(name: String, platform: String, pushToken: String?)

suspend fun getVaultState(since: Instant?): VaultStateResponse

suspend fun requestPush(files: List<PushFileRequest>): PushResponse

suspend fun confirmPush(confirmed: List<FileVersion>)

suspend fun requestPull(paths: List<String>): PullResponse

suspend fun deleteFiles(files: List<DeleteFileRequest>): DeleteResponse

suspend fun uploadToS3(presignedUrl: String, content: ByteArray)

suspend fun downloadFromS3(presignedUrl: String): ByteArray

}

```

ChangeDetector:

```kotlin

class ChangeDetector(private val vaultPath: String, private val fileSystem: PlatformFileSystem, private val syncDb: SyncStateDatabase) {

fun detectLocalChanges(): ChangeSet // scan all .md files, hash, compare against sync_state

fun hashFile(path: String): String // SHA-256

}

data class ChangeSet(

val added: List<String>, // new files not in sync_state

val modified: List<String>, // hash differs from sync_state

val deleted: List<String>, // in sync_state but not on disk

)

```

SyncManager — the core orchestrator:

```kotlin

class SyncManager(

private val syncClient: SyncClient,

private val changeDetector: ChangeDetector,

private val vaultManager: VaultManager,

private val vaultIndexer: VaultIndexer,

private val syncDb: SyncStateDatabase,

private val fileSystem: PlatformFileSystem,

) {

val status: StateFlow<SyncStatus>

val pendingConflicts: StateFlow<List<SyncConflict>>

suspend fun syncNow() // full 5-phase sync cycle

suspend fun onLocalFileChanged(path: String) // debounce + push single file

suspend fun onPushNotificationReceived(changedPaths: List<String>) // pull specific files

}

```

sync_state SQLDelight schema (.steel/sync.db):

```sql

CREATE TABLE sync_state (

relative_path TEXT PRIMARY KEY,

content_hash TEXT NOT NULL,

server_version INTEGER NOT NULL DEFAULT 0,

sync_status TEXT NOT NULL, -- 'synced' | 'pending_push' | 'pending_pull' | 'conflict'

last_synced_at TEXT

);

CREATE TABLE sync_meta (

key TEXT PRIMARY KEY,

value TEXT NOT NULL

);

CREATE TABLE sync_log (

id INTEGER PRIMARY KEY AUTOINCREMENT,

timestamp TEXT NOT NULL,

event_type TEXT NOT NULL,

relative_path TEXT NOT NULL,

detail TEXT

);

```

Acceptance criteria:

Step 6: iOS — Sign In & Sync UI

Scope: Wire up auth and sync in the iOS app.

Create:

```

iosApp/SteelNotes/

├── Services/

│ ├── AuthService.swift # Sign in with Apple flow

│ └── SyncService.swift # Swift wrapper around SyncManager

├── Views/

│ ├── Auth/

│ │ └── SignInView.swift # Sign in with Apple button + onboarding

│ ├── Sync/

│ │ ├── SyncStatusView.swift # Toolbar indicator (synced/syncing/offline/error)

│ │ └── ConflictResolutionView.swift # (Phase 2D)

│ └── Settings/

│ └── SyncSettingsView.swift # Account, storage, device list

└── ViewModels/

├── AuthViewModel.swift

└── SyncViewModel.swift

```

Key integration points:

Acceptance criteria:

Step 7: Infrastructure & Deployment

Scope: Deploy the backend to AWS.

Create:

```

server/deploy/

├── terraform/

│ ├── main.tf # Provider, backend state

│ ├── lambda.tf # Lambda function + API Gateway

│ ├── rds.tf # Postgres (t4g.nano)

│ ├── s3.tf # Vault storage bucket

│ ├── iam.tf # Lambda execution role, S3 access policy

│ └── variables.tf # Environment config

├── Dockerfile # Multi-stage: build Rust, package for Lambda

└── scripts/

└── deploy.sh # Build + terraform apply

```

Target architecture:

Acceptance criteria:

Testing Strategy

Key Decisions for the Implementer

1. Start with local dev, not Lambda. Build and test with cargo run⧉ locally + local Postgres + LocalStack S3 (or MinIO). Deploy to Lambda at the end.

2. SyncStateDatabase needs its own DatabaseDriverFactory. The existing one creates index.db. Sync needs a second SQLite database at .steel/sync.db. Either parameterize the factory or create SyncDatabaseDriverFactory.

3. SHA-256 hashing in KMP. The ChangeDetector needs SHA-256. Options:

4. Token storage. TokenStore is expect/actual. Apple actual uses Keychain via Security.framework (SecItemAdd, SecItemCopyMatching). This is straightforward cinterop.

5. SKIE is already in the build. Use it for exposing StateFlow<SyncStatus> to Swift as AsyncSequence. This avoids manual Flow-to-Combine bridging.

6. Don't sync index.db or sync.db. Both are in .steel/ which should be excluded from sync. The ChangeDetector should only scan .md files in sources/, thoughts/, questions/, syntheses/.