blackroad-operating-system/sop/workflows/release-pipeline.md

Release Pipeline Workflow

Owner: Engineering + DevOps Systems: GitHub → Railway/Cloudflare → Salesforce → Asana Status: Active Last Updated: 2025-11-17

---

Overview

This workflow automates the entire release and deployment lifecycle, from code commit to production deployment to updating all stakeholders automatically.

Key Principle: Merge to main should feel like magic. Everything else happens automatically.

---

The Pipeline

Developer pushes code
    ↓
GitHub Actions: CI pipeline (test + lint + build)
    ↓
    PASS → Ready for PR
    ↓
PR created + reviewed + approved
    ↓
PR merged to `main`
    ↓
GitHub Actions: Deploy pipeline
    ↓
    ├─→ Deploy to Railway (backend)
    ├─→ Deploy to Cloudflare Pages (frontend)
    └─→ Update configs (ops)
    ↓
Deploy succeeds
    ↓
GitHub Actions: Notify stakeholders
    ↓
    ├─→ Update Salesforce Project (deploy metadata)
    ├─→ Complete Asana tasks (deployment checklist)
    └─→ Post to Slack #deploys

---

Stage 1: Continuous Integration (CI)

Trigger:

• Every push to any branch
• Every pull request opened/updated

GitHub Action: .github/workflows/ci.yml

Jobs:

1.1 Test

test:
  runs-on: ubuntu-latest
  strategy:
    matrix:
      python-version: [3.11, 3.12]
  steps:
    - uses: actions/checkout@v4
    - name: Set up Python
      uses: actions/setup-python@v5
      with:
        python-version: ${{ matrix.python-version }}
    - name: Install dependencies
      run: |
        pip install -r requirements.txt
        pip install pytest pytest-cov
    - name: Run tests
      run: pytest --cov=. --cov-report=xml
    - name: Upload coverage
      uses: codecov/codecov-action@v4
      if: matrix.python-version == '3.12'

1.2 Lint

lint:
  runs-on: ubuntu-latest
  steps:
    - uses: actions/checkout@v4
    - name: Set up Python
      uses: actions/setup-python@v5
      with:
        python-version: '3.12'
    - name: Install linters
      run: pip install ruff black isort mypy
    - name: Run ruff
      run: ruff check .
    - name: Check formatting
      run: black --check .
    - name: Check import order
      run: isort --check-only .
    - name: Type check
      run: mypy . --ignore-missing-imports

1.3 Build

build:
  runs-on: ubuntu-latest
  steps:
    - uses: actions/checkout@v4
    - name: Build Docker image
      run: docker build -t test-build .
    - name: Test container starts
      run: |
        docker run -d --name test-container test-build
        sleep 5
        docker logs test-container
        docker stop test-container

Outputs: ✅ All checks pass → PR can be merged ❌ Any check fails → PR blocked until fixed

---

Stage 2: Pull Request Flow

Process:

1. Developer creates PR with standardized template:

   ## What
   Brief description of changes
   
   ## Why
   Business/technical justification
   
   ## How
   Implementation approach
   
   ## Testing
   - [ ] Unit tests added/updated
   - [ ] Integration tests pass
   - [ ] Manual testing completed
   
   ## Screenshots/Logs
   (if applicable)
   
   ## Related
   - Asana Task: [link]
   - Salesforce Project: [link]
   

2. Automated checks run:

   • CI pipeline (test + lint + build)
   • CODEOWNERS review assignment
   • Label auto-applied based on files changed

3. Human review:

   • At least 1 approval required
   • No unresolved conversations
   • All checks green

4. Merge:

   • Squash and merge (clean history)
   • Auto-delete branch after merge

---

Stage 3: Continuous Deployment (CD)

Trigger:

• Push to main branch
• GitHub Release created (for versioned deploys)

GitHub Action: .github/workflows/deploy.yml

3.1 Deploy Backend (Railway)

deploy-backend:
  runs-on: ubuntu-latest
  environment: production
  steps:
    - uses: actions/checkout@v4

    - name: Install Railway CLI
      run: |
        curl -fsSL https://railway.app/install.sh | sh

    - name: Deploy to Railway
      env:
        RAILWAY_TOKEN: ${{ secrets.RAILWAY_TOKEN }}
      run: |
        railway up --service backend --environment production

    - name: Wait for deployment
      run: |
        railway status --service backend --environment production --wait

    - name: Run post-deploy health check
      run: |
        BACKEND_URL=$(railway variables get BACKEND_URL --service backend --environment production)
        curl -f "$BACKEND_URL/health" || exit 1

    - name: Save deploy metadata
      run: |
        echo "DEPLOY_TIME=$(date -u +%Y-%m-%dT%H:%M:%SZ)" >> $GITHUB_ENV
        echo "DEPLOY_SHA=${{ github.sha }}" >> $GITHUB_ENV
        echo "DEPLOY_BRANCH=${{ github.ref_name }}" >> $GITHUB_ENV

3.2 Deploy Frontend (Cloudflare Pages)

deploy-frontend:
  runs-on: ubuntu-latest
  environment: production
  steps:
    - uses: actions/checkout@v4

    - name: Setup Node.js
      uses: actions/setup-node@v4
      with:
        node-version: '20'

    - name: Install dependencies
      run: npm ci

    - name: Build frontend
      env:
        VITE_API_URL: ${{ secrets.PROD_API_URL }}
        VITE_ENV: production
      run: npm run build

    - name: Deploy to Cloudflare Pages
      uses: cloudflare/pages-action@v1
      with:
        apiToken: ${{ secrets.CLOUDFLARE_API_TOKEN }}
        accountId: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }}
        projectName: blackroad-frontend
        directory: dist
        gitHubToken: ${{ secrets.GITHUB_TOKEN }}

    - name: Verify deployment
      run: |
        sleep 30  # Wait for propagation
        curl -f "https://blackroad.app" || exit 1

3.3 Update Infrastructure (if ops repo)

update-infra:
  runs-on: ubuntu-latest
  if: contains(github.repository, '-ops')
  steps:
    - uses: actions/checkout@v4

    - name: Apply Terraform changes
      env:
        TF_VAR_railway_token: ${{ secrets.RAILWAY_TOKEN }}
        TF_VAR_cloudflare_token: ${{ secrets.CLOUDFLARE_API_TOKEN }}
      run: |
        cd terraform
        terraform init
        terraform plan -out=tfplan
        terraform apply -auto-approve tfplan

    - name: Update Cloudflare DNS
      if: contains(github.event.head_commit.message, '[dns]')
      run: |
        # Apply DNS changes from dns-config.json
        python scripts/update_cloudflare_dns.py

---

Stage 4: Stakeholder Notification

Trigger: Deploy jobs complete successfully

GitHub Action: .github/workflows/notify-stakeholders.yml

4.1 Update Salesforce

update-salesforce:
  needs: [deploy-backend, deploy-frontend]
  runs-on: ubuntu-latest
  steps:
    - name: Get Project Key from repo
      run: |
        # Extract from repo name: blackroad-ACME-X7K9-backend → ACME-X7K9
        REPO_NAME="${{ github.repository }}"
        PROJECT_KEY=$(echo "$REPO_NAME" | sed -n 's/.*blackroad-\([A-Z0-9-]*\)-.*/\1/p')
        echo "PROJECT_KEY=$PROJECT_KEY" >> $GITHUB_ENV

    - name: Update Salesforce Project record
      env:
        SF_INSTANCE_URL: ${{ secrets.SALESFORCE_INSTANCE_URL }}
        SF_ACCESS_TOKEN: ${{ secrets.SALESFORCE_ACCESS_TOKEN }}
      run: |
        curl -X PATCH \
          "$SF_INSTANCE_URL/services/data/v58.0/sobjects/Project__c/Project_Key__c/$PROJECT_KEY" \
          -H "Authorization: Bearer $SF_ACCESS_TOKEN" \
          -H "Content-Type: application/json" \
          -d '{
            "Last_Deploy_At__c": "'"$(date -u +%Y-%m-%dT%H:%M:%SZ)"'",
            "Last_Deploy_SHA__c": "${{ github.sha }}",
            "Last_Deploy_Branch__c": "${{ github.ref_name }}",
            "Last_Deploy_Actor__c": "${{ github.actor }}",
            "Deploy_Status__c": "Success",
            "Environment__c": "Production",
            "Release_Notes_URL__c": "https://github.com/${{ github.repository }}/commit/${{ github.sha }}"
          }'

    - name: Create Salesforce deployment record
      run: |
        curl -X POST \
          "$SF_INSTANCE_URL/services/data/v58.0/sobjects/Deployment__c" \
          -H "Authorization: Bearer $SF_ACCESS_TOKEN" \
          -H "Content-Type: application/json" \
          -d '{
            "Project__c": "'"$PROJECT_KEY"'",
            "Deployed_At__c": "'"$(date -u +%Y-%m-%dT%H:%M:%SZ)"'",
            "Git_SHA__c": "${{ github.sha }}",
            "Git_Branch__c": "${{ github.ref_name }}",
            "Deployed_By__c": "${{ github.actor }}",
            "Status__c": "Success",
            "Repository__c": "${{ github.repository }}",
            "Commit_URL__c": "https://github.com/${{ github.repository }}/commit/${{ github.sha }}"
          }'

4.2 Update Asana

update-asana:
  needs: [deploy-backend, deploy-frontend]
  runs-on: ubuntu-latest
  steps:
    - name: Get Asana project and task IDs
      env:
        ASANA_PAT: ${{ secrets.ASANA_PAT }}
        PROJECT_KEY: ${{ env.PROJECT_KEY }}
      run: |
        # Find project by name containing PROJECT_KEY
        PROJECT_GID=$(curl -s "https://app.asana.com/api/1.0/projects?workspace=${{ secrets.ASANA_WORKSPACE_GID }}&opt_fields=name,gid" \
          -H "Authorization: Bearer $ASANA_PAT" | \
          jq -r '.data[] | select(.name | contains("'$PROJECT_KEY'")) | .gid')

        echo "ASANA_PROJECT_GID=$PROJECT_GID" >> $GITHUB_ENV

    - name: Find and complete deploy task
      run: |
        # Find "Deploy to production" task
        TASK_GID=$(curl -s "https://app.asana.com/api/1.0/projects/$ASANA_PROJECT_GID/tasks?opt_fields=name,gid,completed" \
          -H "Authorization: Bearer $ASANA_PAT" | \
          jq -r '.data[] | select(.name | contains("Deploy") and (.completed == false)) | .gid' | head -n 1)

        if [ -n "$TASK_GID" ]; then
          # Mark as complete
          curl -X PUT "https://app.asana.com/api/1.0/tasks/$TASK_GID" \
            -H "Authorization: Bearer $ASANA_PAT" \
            -H "Content-Type: application/json" \
            -d '{"data": {"completed": true}}'

          # Add comment with deploy details
          curl -X POST "https://app.asana.com/api/1.0/tasks/$TASK_GID/stories" \
            -H "Authorization: Bearer $ASANA_PAT" \
            -H "Content-Type: application/json" \
            -d '{
              "data": {
                "text": "✅ Deployed to production\n\n**Commit:** ${{ github.sha }}\n**By:** ${{ github.actor }}\n**Time:** '"$(date -u)"'\n**Link:** https://github.com/${{ github.repository }}/commit/${{ github.sha }}"
              }
            }'
        fi

4.3 Notify Slack

notify-slack:
  needs: [update-salesforce, update-asana]
  runs-on: ubuntu-latest
  if: always()
  steps:
    - name: Post to #deploys channel
      env:
        SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_DEPLOYS }}
      run: |
        STATUS_EMOJI="${{ job.status == 'success' && '✅' || '❌' }}"
        curl -X POST "$SLACK_WEBHOOK_URL" \
          -H "Content-Type: application/json" \
          -d '{
            "blocks": [
              {
                "type": "section",
                "text": {
                  "type": "mrkdwn",
                  "text": "'"$STATUS_EMOJI"' *Deploy to Production*\n\n*Project:* '"$PROJECT_KEY"'\n*Repo:* `${{ github.repository }}`\n*Commit:* <https://github.com/${{ github.repository }}/commit/${{ github.sha }}|'"${GITHUB_SHA:0:7}"'>\n*By:* ${{ github.actor }}\n*Status:* ${{ job.status }}"
                }
              }
            ]
          }'

---

Stage 5: Rollback (if needed)

Trigger: Manual action or automated health check failure

GitHub Action: .github/workflows/rollback.yml

name: Rollback Deployment

on:
  workflow_dispatch:
    inputs:
      target_sha:
        description: 'Git SHA to rollback to'
        required: true
      reason:
        description: 'Reason for rollback'
        required: true

jobs:
  rollback:
    runs-on: ubuntu-latest
    environment: production
    steps:
      - uses: actions/checkout@v4
        with:
          ref: ${{ github.event.inputs.target_sha }}

      - name: Rollback backend
        run: |
          railway up --service backend --environment production

      - name: Rollback frontend
        run: |
          npm ci
          npm run build
          # Deploy to Cloudflare...

      - name: Update Salesforce
        run: |
          curl -X POST "$SF_INSTANCE_URL/services/data/v58.0/sobjects/Deployment__c" \
            -H "Authorization: Bearer $SF_ACCESS_TOKEN" \
            -d '{
              "Project__c": "'"$PROJECT_KEY"'",
              "Status__c": "Rollback",
              "Git_SHA__c": "${{ github.event.inputs.target_sha }}",
              "Rollback_Reason__c": "${{ github.event.inputs.reason }}"
            }'

      - name: Notify team
        run: |
          curl -X POST "$SLACK_WEBHOOK_URL" \
            -d '{
              "text": "⚠️ *ROLLBACK PERFORMED*\n\nProject: '"$PROJECT_KEY"'\nRolled back to: '"${{ github.event.inputs.target_sha }}"'\nReason: '"${{ github.event.inputs.reason }}"'\nBy: '"${{ github.actor }}"'"
            }'

---

Human Touch Points

What Developers Do:

1. Write code in feature branch
2. Create PR when ready
3. Address review feedback
4. Merge PR (or approve auto-merge)
5. That's it. Everything else is automatic.

What Ops/Brenda Sees:

1. Slack notification: "✅ Deploy to production: ACME-X7K9"
2. Asana task auto-completes
3. Salesforce shows updated "Last Deploy" timestamp
4. No manual status updates needed

When to Intervene:

• Deploy fails (red X in GitHub Actions)
• Health check fails post-deploy
• Customer reports issue immediately after deploy
• → Use rollback workflow

---

Metrics Dashboard

Track these in Salesforce or a monitoring tool:

Metric	Target	Current
Deploy Frequency	> 5/week per project	-
Lead Time (commit → production)	< 30 minutes	-
Change Failure Rate	< 15%	-
MTTR (Mean Time to Recovery)	< 1 hour	-
Deploy Success Rate	> 95%	-

---

Validation Checklist

After each deploy, verify:

• CI pipeline passed all checks
• Backend health check returns 200
• Frontend loads without errors
• Database migrations applied (if any)
• Salesforce Project record updated
• Asana task marked complete
• Slack notification sent
• No alerts in monitoring

---

Troubleshooting

Symptom	Likely Cause	Fix
CI fails on test	Code issue	Fix tests, push again
Deploy fails on Railway	Token expired	Refresh RAILWAY_TOKEN secret
Salesforce not updating	Wrong project key	Verify repo name matches blackroad-{PROJECT_KEY}-*
Asana task not completing	Task name doesn't match	Ensure task name contains "Deploy"
Health check fails	Backend not fully started	Increase sleep time in workflow

---

Related Docs

• GitHub Actions: CI Workflow
• GitHub Actions: Deploy Workflow
• GitHub Actions: Notify Stakeholders
• Integration: GitHub → Salesforce
• New Client Kickoff Workflow

---

Philosophy

"Deploy should be boring."

The goal is to make deployments so reliable, automated, and well-monitored that they become non-events.

Every commit to main is a potential release. Every release updates all stakeholders automatically. No human should ever ask "Did this deploy?" or "What version is in production?"

The system should always know.