30 KiB

Raw Blame History

MotorSignal OPS-4 Production Deployment & Database Migration Runbook

(Hardened Final Execution Guide)

NOTICE: DOCUMENTATION & PLANNING ONLY
This runbook is a procedural guide for human execution during an authorized maintenance window on the MotorSignal production VPS.
DO NOT EXECUTE AUTOMATICALLY. Every destructive or state-changing step requires human review, validation, and confirmation.

Document Metadata & Verified Baseline

Target Checkpoint Commit: 8681e6f25972cea768106e534b11418b58374d28
Target Branch: fix/restore-clean-build
Production Host: api.motorsignal.app (89.31.239.10)
Operating System: Ubuntu 22.04.5 LTS (Jammy Jellyfish)
Database Engine: PostgreSQL 14.24 (postgresql@14-main.service on 127.0.0.1:5432)
Database Name / User: database=motorsignal, user=motorsignal_app (Size: ~9.2 MB)
Pre-Migration Alembic Revision: e0f5a7123b46 (link_firebase_identity)
Target Alembic Revision: f1a2b3c4d5e6 (add_ops_schema_and_incidents)
Authoritative Migration Path:
e0f5a7123b46 \to f1a6b8234c57 \to f1b7c8345d68 \to f1a2b3c4d5e6
Deployment User / Service User: motorsignal-admin (deployer) / motorsignal:motorsignal (service account)
Service Ingress & Ports:
- API: motorsignal-api.service (127.0.0.1:8080, WorkingDirectory=/opt/motorsignal/current/app)
- Website: motorsignal-website.service (127.0.0.1:3001, Next.js standalone server.js)
- Ingress: nginx.service (proxies api.motorsignal.app \to 8080, motorsignal.app \to 3001)
Authoritative Backup Directory: /var/backups/motorsignal (DO NOT use /opt/motorsignal/backups)
Host Security Baseline: Fail2ban active; UFW inactive (DO NOT alter firewall during deployment).

Core Operating Principles

Investigate First & Stop on Error: If any step produces an unexpected result, error, or revision mismatch, STOP IMMEDIATELY. Never "continue anyway" or retry failing commands blindly.
Zero-Drift & Safe Secret Handling: Follow the exact commands specified. Never expand secrets into command-line arguments or echo passwords. Pass environment variables in-memory using dedicated Python runners.
Controlled Quiescence: Application writers (motorsignal-api.service and motorsignal-website.service) must be stopped before database migration to eliminate version-skew writes.
Immutable Releases & Truly Atomic Cutover: Deploy via timestamped release directories (/opt/motorsignal/releases/<release-id>) and atomic filesystem renames (current.next \to current).
Rollback Discipline: Never run blind database downgrades on production after writes have occurred. Emergency disaster recovery uses the verified pre-migration backup.

PHASE 0 — Scope & Safety

0.1 Deployment Scope Summary

Backend API: FastAPI application update supporting OPS-4 operational incidents workflow, CSRF origin verification on mutations, audit-event rollback safety, and rate-limiting enhancements.
Operations Frontend: Next.js Operations console (/admin, /admin/incidents, /admin/incidents/[id], /admin/incidents/create), session management, and CSRF-protected proxy routes.
Database Schema: PostgreSQL ops schema creation and evolution:
- ops.operators: Operations staff accounts with authority_version and immutable identity trigger.
- ops.sessions: Operator sessions tied to __Host-motorsignal_ops cookies.
- ops.audit_events: Tamper-evident operational audit trail.
- ops.incidents: Incident management records with case references (OPS-YYYY-NNNNNN).
- ops.incident_notes: Internal operator notes.
- ops.incident_case_seq: Sequence for case reference generation.
- ops.guard_operator_identity(): Trigger function enforcing identity immutability.

PHASE 1 — Preflight & Host Baseline Checks

1.1 VPS Connection & Identity Verification

Precondition: SSH access configured with deployment key C:\Users\Bill\.ssh\motorsignal_prod.

Command:

ssh -i ~/.ssh/motorsignal_prod motorsignal-admin@89.31.239.10
hostname
whoami
id

Expected Result: Hostname is api.motorsignal.app; user is motorsignal-admin (uid=1000, member of sudo and motorsignal groups).
Failure Action: STOP. Do not proceed on an unverified host.

1.2 Host Storage & Resources Inspection

Command:

# Storage check (require >= 10 GB free on root /)
df -h /

# Memory and load check (require >= 1 GB available RAM)
free -m
uptime

Expected Result: Free disk \ge 10 GB (discovery showed ~272 GB free); available RAM \ge 1 GB; system load normal.
Failure Action: STOP. Resolve resource constraints before proceeding.

1.3 Service Baseline Status

Command:

systemctl status motorsignal-api.service --no-pager
systemctl status motorsignal-website.service --no-pager
systemctl status nginx.service --no-pager
systemctl status postgresql@14-main.service --no-pager

Expected Result: All services reporting active (running).
Failure Action: STOP. Investigate pre-existing service failures.

1.4 PostgreSQL 14.24 Engine & Exact Alembic Revision Gate

Command:

# Check PostgreSQL version
sudo -u postgres psql -d motorsignal -c "SELECT version();"

# Query exact current Alembic revision
sudo -u postgres psql -d motorsignal -t -A -c "SELECT version_num FROM alembic_version;"

Expected Result: Version string indicates PostgreSQL 14.24; revision is EXACTLY e0f5a7123b46.
Failure Action: STOP IMMEDIATELY. If the revision is not e0f5a7123b46, abort deployment.

1.5 Baseline Health Probes

Command:

# Record currently active release
PREV_RELEASE=$(readlink -f /opt/motorsignal/current)
echo "CURRENT_ACTIVE_RELEASE=${PREV_RELEASE}"

# Probe API and website baseline
curl -fsS http://127.0.0.1:8080/api/v1/health
curl -fsS http://127.0.0.1:8080/api/v1/ready
curl -fsS -I http://127.0.0.1:3001/

Expected Result: API returns {"status":"ok","service":"motorsignal-api"} and {"status":"ready","database":"ok"}; website returns HTTP 200.
Failure Action: STOP. Do not deploy on a degraded baseline.

1.6 Security & Firewall Notice

Command:
```
systemctl is-active fail2ban
```
Expected Result: active.
Policy Notice: Discovery confirmed UFW is inactive on this host. DO NOT modify UFW or iptables during OPS-4 deployment.

PHASE 2 — Release Staging & Artifact Preflight

Execute this phase while production is STILL ONLINE and serving traffic from ${PREV_RELEASE}.

2.1 Release Directory Setup & Namespace Isolation

To satisfy motorsignal-api.service (WorkingDirectory=/opt/motorsignal/current/app) and motorsignal-website.service (ExecStart=/usr/bin/node /opt/motorsignal/current/website/server.js) without directory collision:

Command:

RELEASE_ID="20260910-ops4"
NEW_RELEASE="/opt/motorsignal/releases/${RELEASE_ID}"

# Ensure release directory structure exists
sudo install -d -o motorsignal -g motorsignal "${NEW_RELEASE}"
sudo install -d -o motorsignal -g motorsignal "${NEW_RELEASE}/backend"
sudo install -d -o motorsignal -g motorsignal "${NEW_RELEASE}/website"

2.2 Unpack Release Artifacts & Compatibility Symlinks

Command:

# Extract uploaded artifacts
sudo -u motorsignal unzip -q /tmp/motorsignal-backend-${RELEASE_ID}.zip -d "${NEW_RELEASE}/backend"
sudo -u motorsignal unzip -q /tmp/motorsignal-website-${RELEASE_ID}.zip -d "${NEW_RELEASE}/website"

# Establish release-internal compatibility symlinks for systemd and Alembic
sudo -u motorsignal ln -sfn backend/app "${NEW_RELEASE}/app"
sudo -u motorsignal ln -sfn backend/alembic "${NEW_RELEASE}/alembic"
sudo -u motorsignal ln -sfn backend/alembic.ini "${NEW_RELEASE}/alembic.ini"
sudo -u motorsignal ln -sfn backend/requirements.lock "${NEW_RELEASE}/requirements.lock"

sudo chown -R motorsignal:motorsignal "${NEW_RELEASE}"

2.3 Virtual Environment Setup

Command:

# Create dedicated virtual environment for this release
sudo -u motorsignal python3 -m venv "${NEW_RELEASE}/venv"

# Install locked dependencies strictly from requirements.lock
sudo -u motorsignal "${NEW_RELEASE}/venv/bin/pip" install --no-cache-dir -r "${NEW_RELEASE}/requirements.lock"

Expected Result: Virtualenv created cleanly with all locked dependencies installed.

2.4 Artifact Preflight & Manifest Verification (MANDATORY GATE)

Command:

# 1. Verify release manifest source commit
grep "source_commit=8681e6f25972cea768106e534b11418b58374d28" "${NEW_RELEASE}/backend/release_manifest.json" || \
grep "8681e6f25972cea768106e534b11418b58374d28" /tmp/release_manifest_${RELEASE_ID}.txt

# 2. Verify all runtime files and migration files exist and resolve
test -f "${NEW_RELEASE}/app/main.py" && echo "BACKEND_MAIN_OK"
test -f "${NEW_RELEASE}/alembic.ini" && echo "ALEMBIC_INI_OK"
test -f "${NEW_RELEASE}/alembic/env.py" && echo "ALEMBIC_ENV_OK"
test -f "${NEW_RELEASE}/alembic/versions/e0f5a7123b46_link_firebase_identity.py" && echo "MIGRATION_e0f5_OK"
test -f "${NEW_RELEASE}/alembic/versions/f1a6b8234c57_add_ops_foundation.py" && echo "MIGRATION_f1a6_OK"
test -f "${NEW_RELEASE}/alembic/versions/f1b7c8345d68_evolve_ops_runtime_schema.py" && echo "MIGRATION_f1b7_OK"
test -f "${NEW_RELEASE}/alembic/versions/f1a2b3c4d5e6_add_ops_schema_and_incidents.py" && echo "MIGRATION_f1a2_OK"
test -f "${NEW_RELEASE}/requirements.lock" && echo "LOCK_OK"
test -x "${NEW_RELEASE}/venv/bin/python" && echo "VENV_PYTHON_OK"
test -x "${NEW_RELEASE}/venv/bin/alembic" && echo "VENV_ALEMBIC_OK"
test -f "${NEW_RELEASE}/website/server.js" && echo "WEBSITE_SERVER_OK"

# 3. Check Python engine version
"${NEW_RELEASE}/venv/bin/python" --version

# 4. Verify Alembic heads in new release (Must be exactly f1a2b3c4d5e6, single head)
"${NEW_RELEASE}/venv/bin/alembic" -c "${NEW_RELEASE}/alembic.ini" heads

Expected Result: All artifact checks print OK; Python version is 3.10.x; Alembic heads shows f1a2b3c4d5e6 (head).
Failure Action: STOP BEFORE MAINTENANCE. If any artifact or migration file is missing, abort.

2.5 Pre-Maintenance Authorization Gate

NEW_RELEASE directory created at /opt/motorsignal/releases/20260910-ops4
Source commit confirmed as 8681e6f25972cea768106e534b11418b58374d28
Backend runtime files verified (app/main.py, alembic.ini, alembic/env.py)
Website runtime file verified (website/server.js)
Dedicated venv created and populated from locked requirements
All 4 migration files present in chain (e0f5, f1a6, f1b7, f1a2)
Alembic head confirmed as f1a2b3c4d5e6 (single head)
Active release path recorded in PREV_RELEASE
Production currently healthy (/health == 200, /ready == 200)

GATE DECISION: If ALL items checked, proceed to Phase 3. Otherwise, DO NOT STOP SERVICES.

PHASE 3 — Pre-Maintenance Backup & Disposable Restore Drill

3.1 Safe Pre-Migration Logical Backup

Command:

# Ensure backup directory exists
sudo install -d -m 0750 -o postgres -g postgres /var/backups/motorsignal
BACKUP_TS=$(date +%Y%m%d_%H%M%S)
BACKUP_FILE="/var/backups/motorsignal/motorsignal_pre_ops4_${BACKUP_TS}.dump"
export BACKUP_FILE

# Execute pg_dump securely via Python runner (No passwords in argv or stdout)
sudo -u motorsignal BACKUP_FILE="${BACKUP_FILE}" "${NEW_RELEASE}/venv/bin/python" - <<'PY'
import os, subprocess, sys
from dotenv import dotenv_values

env_vars = dotenv_values("/etc/motorsignal/production.env")
pg_env = os.environ.copy()
pg_env["PGHOST"] = env_vars.get("DATABASE_HOST", "127.0.0.1")
pg_env["PGPORT"] = str(env_vars.get("DATABASE_PORT", "5432"))
pg_env["PGDATABASE"] = env_vars.get("DATABASE_NAME", "motorsignal")
pg_env["PGUSER"] = env_vars.get("DATABASE_USER", "motorsignal_app")
if "DATABASE_PASSWORD" in env_vars:
    pg_env["PGPASSWORD"] = env_vars["DATABASE_PASSWORD"]

backup_file = os.environ["BACKUP_FILE"]
res = subprocess.run(["pg_dump", "-Fc", "-v", "-f", backup_file, pg_env["PGDATABASE"]], env=pg_env)
sys.exit(res.returncode)
PY

# Verify backup file size, checksum, and table-of-contents readability
test -s "${BACKUP_FILE}" && echo "BACKUP_SIZE_OK"
sha256sum "${BACKUP_FILE}" | sudo tee "${BACKUP_FILE}.sha256"
sudo -u postgres pg_restore --list "${BACKUP_FILE}" | head -n 25

Expected Result: Backup created with non-zero size (~9 MB); checksum recorded; pg_restore --list displays valid TOC.

3.2 Disposable Restore Drill (Mandatory Gate)

Command:

DRILL_DB="motorsignal_restore_drill_${BACKUP_TS}"

# 1. Create temporary drill database
sudo -u postgres psql -c "CREATE DATABASE ${DRILL_DB};"

# 2. Restore backup into drill database
sudo -u postgres pg_restore -v --no-owner --no-privileges -d "${DRILL_DB}" "${BACKUP_FILE}"

# 3. Verify Alembic revision in drill database
sudo -u postgres psql -d "${DRILL_DB}" -t -A -c "SELECT version_num FROM alembic_version;"
# Expected: e0f5a7123b46

# 4. Verify core table counts match
sudo -u postgres psql -d "${DRILL_DB}" -c "SELECT 'users' AS tbl, count(*) FROM users UNION ALL SELECT 'rides', count(*) FROM rides;"

# 5. Clean up drill database
sudo -u postgres psql -c "DROP DATABASE ${DRILL_DB};"

Expected Result: Drill DB restores cleanly; alembic_version is e0f5a7123b46; counts match; drill DB dropped.
Failure Action: STOP DEPLOYMENT. If the drill fails, do not proceed with migration.

PHASE 4 — Maintenance Quiesce

4.1 Quiesce Application Writers

Command:

# Record pre-quiesce service states
sudo systemctl status motorsignal-api.service motorsignal-website.service --no-pager

# Stop application services
sudo systemctl stop motorsignal-api.service
sudo systemctl stop motorsignal-website.service

# Confirm services are inactive
sudo systemctl is-active motorsignal-api.service || echo "API_STOPPED"
sudo systemctl is-active motorsignal-website.service || echo "WEBSITE_STOPPED"

# Confirm PostgreSQL remains active
sudo systemctl is-active postgresql@14-main.service

Expected Result: Application services report inactive; PostgreSQL remains active (running).
Note: Nginx remains running. Public traffic temporarily receives 502 Bad Gateway during the ~1–2 minute migration window.

PHASE 5 — Final Quiesced Database Backup

5.1 Final Quiesced Snapshot

To guarantee an absolute zero-skew recovery point after writers are confirmed stopped:

Command:

FINAL_BACKUP_FILE="/var/backups/motorsignal/motorsignal_final_quiesced_${BACKUP_TS}.dump"
export FINAL_BACKUP_FILE

sudo -u motorsignal BACKUP_FILE="${FINAL_BACKUP_FILE}" "${NEW_RELEASE}/venv/bin/python" - <<'PY'
import os, subprocess, sys
from dotenv import dotenv_values

env_vars = dotenv_values("/etc/motorsignal/production.env")
pg_env = os.environ.copy()
pg_env["PGHOST"] = env_vars.get("DATABASE_HOST", "127.0.0.1")
pg_env["PGPORT"] = str(env_vars.get("DATABASE_PORT", "5432"))
pg_env["PGDATABASE"] = env_vars.get("DATABASE_NAME", "motorsignal")
pg_env["PGUSER"] = env_vars.get("DATABASE_USER", "motorsignal_app")
if "DATABASE_PASSWORD" in env_vars:
    pg_env["PGPASSWORD"] = env_vars["DATABASE_PASSWORD"]

backup_file = os.environ["BACKUP_FILE"]
res = subprocess.run(["pg_dump", "-Fc", "-v", "-f", backup_file, pg_env["PGDATABASE"]], env=pg_env)
sys.exit(res.returncode)
PY

test -s "${FINAL_BACKUP_FILE}" && sha256sum "${FINAL_BACKUP_FILE}" | sudo tee "${FINAL_BACKUP_FILE}.sha256"

Expected Result: Final quiesced backup created and checksum recorded.

PHASE 6 — Database Migration

CRITICAL EXECUTION GATE:
Executes migration against production database. Run once. DO NOT retry blindly on error.

6.1 Pre-Migration State Gate

Command:

# Verify current database revision is exactly e0f5a7123b46
sudo -u postgres psql -d motorsignal -t -A -c "SELECT version_num FROM alembic_version;"
# Must output: e0f5a7123b46

6.2 Execute Migration via Secure Python Runner

Command:

sudo -u motorsignal NEW_RELEASE="${NEW_RELEASE}" "${NEW_RELEASE}/venv/bin/python" - <<'PY'
import os, subprocess, sys
from dotenv import dotenv_values

release = os.environ["NEW_RELEASE"]
env_vars = dotenv_values("/etc/motorsignal/production.env")
run_env = os.environ.copy()
for k, v in env_vars.items():
    if v is not None:
        run_env[k] = v

res = subprocess.run(
    [
        f"{release}/venv/bin/alembic",
        "-c",
        f"{release}/alembic.ini",
        "upgrade",
        "f1a2b3c4d5e6",
    ],
    cwd=release,
    env=run_env,
)
sys.exit(res.returncode)
PY

Expected Result: Alembic logs execution through:
- e0f5a7123b46 -> f1a6b8234c57 (add_ops_foundation)
- f1a6b8234c57 -> f1b7c8345d68 (evolve_ops_runtime_schema)
- f1b7c8345d68 -> f1a2b3c4d5e6 (add_ops_schema_and_incidents)
  Python process exits with code 0.
Failure Action: If non-zero exit code: STOP IMMEDIATELY. Do not retry. Proceed to PHASE 12 (Rollback - Case C).

PHASE 7 — Post-Migration Database Verification

Perform read-only PostgreSQL catalog inspection before cutover.

7.1 Verify Alembic Version

Command:

sudo -u postgres psql -d motorsignal -t -A -c "SELECT version_num FROM alembic_version;"

Expected Result: EXACTLY f1a2b3c4d5e6.

7.2 Verify `ops` Schema & Tables

Command:

sudo -u postgres psql -d motorsignal -c "
SELECT table_schema, table_name 
FROM information_schema.tables 
WHERE table_schema = 'ops' 
ORDER BY table_name;"

Expected Result: 5 tables listed:
- ops.audit_events
- ops.incident_notes
- ops.incidents
- ops.operators
- ops.sessions

7.3 Verify Sequence, Functions & Triggers

Command:

# Sequence check
sudo -u postgres psql -d motorsignal -c "
SELECT sequence_name FROM information_schema.sequences WHERE sequence_schema = 'ops';"

# Function check
sudo -u postgres psql -d motorsignal -c "
SELECT routine_name FROM information_schema.routines WHERE routine_schema = 'ops';"

# Trigger check
sudo -u postgres psql -d motorsignal -c "
SELECT trigger_name, event_manipulation, event_object_table 
FROM information_schema.triggers 
WHERE trigger_schema = 'ops';"

Expected Result: ops.incident_case_seq exists; guard_operator_identity function and trigger on ops.operators exist.

7.4 Verify Permissions & Constraints

Command:

# Check PUBLIC schema usage revoked
sudo -u postgres psql -d motorsignal -t -A -c "SELECT has_schema_privilege('public', 'ops', 'usage');"
# Expected: f

Expected Result: Returns f (false).

PHASE 8 — Atomic Symlink Cutover & Service Startup

8.1 Truly Atomic Symlink Cutover

Command:

# 1. Verify temporary target does NOT already exist
if [ -e /opt/motorsignal/current.next ] || [ -L /opt/motorsignal/current.next ]; then
    echo "STOP: /opt/motorsignal/current.next already exists! Investigate before proceeding."
    exit 1
fi

# 2. Create temporary symlink on the same filesystem
sudo -u motorsignal ln -s "${NEW_RELEASE}" /opt/motorsignal/current.next

# 3. Verify temporary symlink resolves to NEW_RELEASE
test "$(readlink -f /opt/motorsignal/current.next)" = "${NEW_RELEASE}" || { echo "STOP: current.next target mismatch!"; exit 1; }

# 4. Atomically replace current symlink via Linux rename semantics
sudo mv -Tf /opt/motorsignal/current.next /opt/motorsignal/current

# 5. Verify active release
test "$(readlink -f /opt/motorsignal/current)" = "${NEW_RELEASE}" && echo "CUTOVER_SUCCESSFUL"

Expected Result: CUTOVER_SUCCESSFUL printed; /opt/motorsignal/current resolves to ${NEW_RELEASE}.

8.2 Start Services

Command:

sudo systemctl start motorsignal-api.service
sudo systemctl start motorsignal-website.service

# Check service statuses
sudo systemctl status motorsignal-api.service --no-pager
sudo systemctl status motorsignal-website.service --no-pager

Expected Result: Both services report active (running).

8.3 Inspect Startup Journals

Command:

sudo journalctl -u motorsignal-api.service -n 50 --no-pager
sudo journalctl -u motorsignal-website.service -n 50 --no-pager

Expected Result: Uvicorn listening on 127.0.0.1:8080, Next.js standalone listening on 127.0.0.1:3001, zero startup errors.

PHASE 9 — Local Service Health Verification

9.1 Local Loopback Health Probes

Command:

# Core API liveness probe
curl -fsS -v http://127.0.0.1:8080/api/v1/health

# Database readiness probe
curl -fsS -v http://127.0.0.1:8080/api/v1/ready

# Operations subsystem health probe
curl -fsS -v http://127.0.0.1:8080/api/v1/ops/health

# Next.js website loopback probe
curl -fsS -I http://127.0.0.1:3001/

Expected Results:
- /api/v1/health \to HTTP 200 {"status":"ok","service":"motorsignal-api"}
- /api/v1/ready \to HTTP 200 {"status":"ready","database":"ok"}
- /api/v1/ops/health \to HTTP 200 {"status":"available",...}
- http://127.0.0.1:3001/ \to HTTP 200 OK
Failure Action: STOP. Do not consider deployment successful if local probes fail.

PHASE 10 — Public HTTPS Health Verification

10.1 Public Ingress Probes

Command:

# Public API health
curl -fsS -i https://api.motorsignal.app/api/v1/health
curl -fsS -i https://api.motorsignal.app/api/v1/ready

# Public Website routes
curl -fsS -I https://motorsignal.app/
curl -fsS -I https://motorsignal.app/features
curl -fsS -I https://motorsignal.app/pricing

Expected Result: HTTP 200 OK across public endpoints.

10.2 Operations Security Probes (Safe Unauthenticated Checks)

Command:

# Probe /ops/me unauthenticated (Must return 401, NOT 404)
curl -s -o /dev/null -w "%{http_code}\n" https://api.motorsignal.app/api/v1/ops/me
# Expected: 401

# Probe /ops/incidents unauthenticated (Must return 401, NOT 404)
curl -s -o /dev/null -w "%{http_code}\n" https://api.motorsignal.app/api/v1/ops/incidents
# Expected: 401

# Check Operations web route redirect / response
curl -s -I https://motorsignal.app/admin | head -n 5

Expected Result: /ops/me and /ops/incidents return HTTP 401 Unauthorized (confirming OPS-4 routes are active and secured); no 404s.

PHASE 11 — Observation Window

11.1 30-Minute Intensive Monitoring Checklist

Service Restart Counts:

systemctl show motorsignal-api.service -p NRestarts
systemctl show motorsignal-website.service -p NRestarts
# Must remain 0

Nginx Error Logs:

sudo tail -n 100 /var/log/nginx/error.log

PostgreSQL Connection & Activity:

sudo -u postgres psql -c "SELECT count(*), state FROM pg_stat_activity GROUP BY state;"

Live Journal Monitoring:

sudo journalctl -u motorsignal-api.service -f --since "10 minutes ago"

11.2 Rollback Criteria During Observation

Sustained HTTP 5xx error rate > 1%.
Database deadlocks, lock contention, or connection exhaustion.
Uncaught exceptions or crash loops in API service.
Auth or session corruption affecting consumer riders or operators.

PHASE 12 — Rollback & Recovery Procedures

12.1 Rollback Philosophy & Decision Matrix

Scenario	Nature of Failure	Action
CASE A	Application fails immediately post-cutover; ZERO new production writes.	1. Stop services. 2. Atomically repoint `/opt/motorsignal/current` to `${PREV_RELEASE}`. 3. Restart previous services. 4. Re-verify health. 5. Do NOT downgrade database unless old code is incompatible with expanded schema.
CASE B	Failure occurs after production traffic and NEW WRITES have occurred.	1. DO NOT run blind alembic downgrade (risks data loss). 2. Deploy forward-fix patch, OR 3. If catastrophic, execute Disaster Recovery Restore from pre-migration backup.
CASE C	Alembic migration command fails during Phase 6.	1. Keep application stopped. 2. Inspect PostgreSQL transaction state (Alembic runs in transaction). 3. Fix migration issue or restart old application against unchanged `e0f5a7123b46` schema.

12.2 Procedure: Application Rollback (Case A)

Precondition: Stored ${PREV_RELEASE} path (e.g. /opt/motorsignal/releases/20260908-15).

Command:

# 1. Stop application services
sudo systemctl stop motorsignal-api.service
sudo systemctl stop motorsignal-website.service

# 2. Atomically repoint current symlink to previous release
sudo -u motorsignal ln -s "${PREV_RELEASE}" /opt/motorsignal/current.prev
sudo mv -Tf /opt/motorsignal/current.prev /opt/motorsignal/current

# 3. Restart previous services
sudo systemctl start motorsignal-api.service
sudo systemctl start motorsignal-website.service

# 4. Verify local health
curl -fsS http://127.0.0.1:8080/api/v1/health
curl -fsS http://127.0.0.1:8080/api/v1/ready
curl -fsS -I http://127.0.0.1:3001/

12.3 Procedure: Disaster Recovery Database Restore (Emergency Only)

DANGER: RESTORATIVE DISASTER RECOVERY.
Requires explicit operator authorization. Replaces database contents with pre-migration backup.

Command:

# 1. Stop application services
sudo systemctl stop motorsignal-api.service
sudo systemctl stop motorsignal-website.service

# 2. Terminate active database connections
sudo -u postgres psql -c "
SELECT pg_terminate_backend(pid) 
FROM pg_stat_activity 
WHERE datname = 'motorsignal' AND pid <> pg_backend_pid();"

# 3. Rename degraded database for forensic preservation (DO NOT DROP)
TS=$(date +%Y%m%d_%H%M%S)
sudo -u postgres psql -c "ALTER DATABASE motorsignal RENAME TO motorsignal_failed_${TS};"

# 4. Create fresh database
sudo -u postgres psql -c "CREATE DATABASE motorsignal OWNER motorsignal_app;"

# 5. Restore from verified pre-migration backup
sudo -u postgres pg_restore -v --no-owner --no-privileges -d motorsignal "${FINAL_BACKUP_FILE}"

# 6. Verify restored database revision
sudo -u postgres psql -d motorsignal -t -A -c "SELECT version_num FROM alembic_version;"
# Expected: e0f5a7123b46

# 7. Start previous release
sudo -u motorsignal ln -s "${PREV_RELEASE}" /opt/motorsignal/current.prev
sudo mv -Tf /opt/motorsignal/current.prev /opt/motorsignal/current
sudo systemctl start motorsignal-api.service
sudo systemctl start motorsignal-website.service

# 8. Re-verify health
curl -fsS http://127.0.0.1:8080/api/v1/health
curl -fsS http://127.0.0.1:8080/api/v1/ready

PHASE 13 — Post-Deployment Execution Record

Complete and archive this record upon deployment execution.

================================================================================
MOTORSIGNAL PRODUCTION DEPLOYMENT EXECUTION RECORD
================================================================================
Deployment Date/Time (UTC)    : ________________________________________________
Operator Name / ID            : ________________________________________________
Approved Checkpoint Commit    : 8681e6f25972cea768106e534b11418b58374d28
Deployed Release Directory    : /opt/motorsignal/releases/______________________
Previous Release Directory    : /opt/motorsignal/releases/______________________

DATABASE STATE
--------------------------------------------------------------------------------
Pre-Migration Alembic Rev     : e0f5a7123b46
Target Alembic Revision       : f1a2b3c4d5e6
Post-Migration Alembic Rev    : ________________________________________________
PostgreSQL Engine Version     : PostgreSQL 14.24
Backup Filename               : /var/backups/motorsignal/_______________________
Backup SHA-256 Checksum       : ________________________________________________
Restore Drill Result          : [ ] PASS (Verified e0f5a7123b46 in drill DB)

CHANGE WINDOW TIMINGS
--------------------------------------------------------------------------------
Application Quiesced          : _____:_____:_____ UTC
Migration Start Time          : _____:_____:_____ UTC
Migration End Time            : _____:_____:_____ UTC
Migration Exit Code           : ________________
Services Restarted            : _____:_____:_____ UTC
Total Change Window Duration  : _____ minutes

VERIFICATION & HEALTH
--------------------------------------------------------------------------------
Local API Health (200 OK)     : [ ] PASS  [ ] FAIL
Local API Ready (200 OK)      : [ ] PASS  [ ] FAIL
Local Ops Health (200 OK)     : [ ] PASS  [ ] FAIL
Local Website (200 OK)        : [ ] PASS  [ ] FAIL
Public HTTPS API Health       : [ ] PASS  [ ] FAIL
Public HTTPS Ready Check      : [ ] PASS  [ ] FAIL
Public Website Routes Check   : [ ] PASS  [ ] FAIL
Ops Unauth Probe /ops/me      : [ ] PASS (401)  [ ] FAIL
Ops Unauth Probe /ops/incidents: [ ] PASS (401)  [ ] FAIL
Observation Window (30 min)   : [ ] PASS  [ ] FAIL

INCIDENTS & ROLLBACK
--------------------------------------------------------------------------------
Rollback Required?            : [ ] YES  [ ] NO
Rollback Scenario (if any)    : [ ] CASE A  [ ] CASE B  [ ] CASE C
Incidents Encountered         : ________________________________________________
Corrective Notes              : ________________________________________________

Sign-off Operator Signature   : ________________________________________________
================================================================================

30 KiB Raw Blame History Unescape Escape

MotorSignal OPS-4 Production Deployment & Database Migration Runbook

Document Metadata & Verified Baseline

Core Operating Principles

PHASE 0 — Scope & Safety

0.1 Deployment Scope Summary

PHASE 1 — Preflight & Host Baseline Checks

1.1 VPS Connection & Identity Verification

1.2 Host Storage & Resources Inspection

1.3 Service Baseline Status

1.4 PostgreSQL 14.24 Engine & Exact Alembic Revision Gate

1.5 Baseline Health Probes

1.6 Security & Firewall Notice

PHASE 2 — Release Staging & Artifact Preflight

2.1 Release Directory Setup & Namespace Isolation

2.2 Unpack Release Artifacts & Compatibility Symlinks

2.3 Virtual Environment Setup

2.4 Artifact Preflight & Manifest Verification (MANDATORY GATE)

2.5 Pre-Maintenance Authorization Gate

PHASE 3 — Pre-Maintenance Backup & Disposable Restore Drill

3.1 Safe Pre-Migration Logical Backup

3.2 Disposable Restore Drill (Mandatory Gate)

PHASE 4 — Maintenance Quiesce

4.1 Quiesce Application Writers

PHASE 5 — Final Quiesced Database Backup

5.1 Final Quiesced Snapshot

PHASE 6 — Database Migration

6.1 Pre-Migration State Gate

6.2 Execute Migration via Secure Python Runner

PHASE 7 — Post-Migration Database Verification

7.1 Verify Alembic Version

7.2 Verify ops Schema & Tables

7.3 Verify Sequence, Functions & Triggers

7.4 Verify Permissions & Constraints

PHASE 8 — Atomic Symlink Cutover & Service Startup

8.1 Truly Atomic Symlink Cutover

8.2 Start Services

8.3 Inspect Startup Journals

PHASE 9 — Local Service Health Verification

9.1 Local Loopback Health Probes

PHASE 10 — Public HTTPS Health Verification

10.1 Public Ingress Probes

10.2 Operations Security Probes (Safe Unauthenticated Checks)

PHASE 11 — Observation Window

11.1 30-Minute Intensive Monitoring Checklist

11.2 Rollback Criteria During Observation

PHASE 12 — Rollback & Recovery Procedures

12.1 Rollback Philosophy & Decision Matrix

12.2 Procedure: Application Rollback (Case A)

12.3 Procedure: Disaster Recovery Database Restore (Emergency Only)

PHASE 13 — Post-Deployment Execution Record

30 KiB

Raw Blame History

7.2 Verify `ops` Schema & Tables