blackroad-docs/operations/BLACKROAD_PRODUCTION_PLAYBOOK.md

BlackRoad Production-Ready Automation Playbook

Version: 1.0.0 Last Updated: 2025-12-27 Goal: All BlackRoad-OS repos are production-ready, automated, and boringly reliable

---

🎯 Overview

This playbook contains everything needed to achieve 100% reliable, automated deployments across all BlackRoad infrastructure:

• 82+ GitHub repositories
• 15+ GitHub organizations
• Cloudflare Pages (16 zones, 8 Pages projects)
• Railway (12+ projects)
• Raspberry Pi devices (lucidia, blackroad-pi)
• Port 8080 services

---

📦 Components

1. E2E Testing Suite

Location: ~/blackroad-e2e-testing-suite.sh

Purpose: Comprehensive testing of all infrastructure components

Features:

• GitHub connectivity tests
• Cloudflare authentication tests
• Pi device health checks
• Port 8080 service verification
• Memory system validation
• Git workflow tests
• Node.js toolchain tests
• Docker environment tests

Usage:

# Run all tests once
~/blackroad-e2e-testing-suite.sh

# View test logs
cat ~/.blackroad/e2e-tests/e2e-test-*.log

Test Results:

• ✅ GitHub: Authenticated, 4 orgs, 82 repos
• ✅ Cloudflare: Connected
• ⚠️ Pi Devices: 2/3 online (lucidia ✅, blackroad-pi ✅, alt ❌)
• ❌ Port 8080: Services offline

---

2. Real-Time Monitoring Dashboard

Location: ~/blackroad-realtime-monitor.sh

Purpose: Live monitoring of all infrastructure

Features:

• Real-time status display
• JSON state export
• Historical tracking
• Component-specific checks
• 30-second refresh (configurable)

Usage:

# Start continuous monitoring
~/blackroad-realtime-monitor.sh monitor 30

# Single status check
~/blackroad-realtime-monitor.sh once

# Export current state as JSON
~/blackroad-realtime-monitor.sh export

# Check specific component
~/blackroad-realtime-monitor.sh check github
~/blackroad-realtime-monitor.sh check pi-lucidia

# View history
~/blackroad-realtime-monitor.sh history 10

State Storage:

• Current state: ~/.blackroad/monitor/state.json
• History: ~/.blackroad/monitor/history.jsonl

---

3. Automated Health Checks

Location: ~/blackroad-health-checks.sh

Purpose: Continuous health monitoring with alerting

Features:

• Configurable health checks
• Alert generation on failures
• Email/Slack/Discord notifications (configurable)
• Deployment verification for Git → Cloudflare → Pi flows
• Auto-retry with backoff

Usage:

# Initialize health check system
~/blackroad-health-checks.sh init

# Run checks once
~/blackroad-health-checks.sh check

# Continuous monitoring
~/blackroad-health-checks.sh monitor 60

# Verify deployments
~/blackroad-health-checks.sh verify-cloudflare BlackRoad-OS/repo abc1234
~/blackroad-health-checks.sh verify-pi 192.168.4.38 service-name

# View recent alerts
~/blackroad-health-checks.sh alerts 10

Configuration: Edit ~/.blackroad/health/config.json to configure:

• Check intervals
• Critical vs non-critical checks
• Alerting channels
• Deployment pipeline settings

---

4. Deployment Verification & Rollback Suite

Location: ~/blackroad-deployment-verifier.sh

Purpose: Track, verify, and rollback deployments

Features:

• Deployment tracking across platforms
• Automated verification
• One-click rollback to previous working state
• Deployment history
• Platform-agnostic design

Usage:

# Record a deployment
~/blackroad-deployment-verifier.sh record github "BlackRoad-OS/repo" abc1234 success

# Verify deployment
~/blackroad-deployment-verifier.sh verify deploy-1234567-8901

# Rollback deployment
~/blackroad-deployment-verifier.sh rollback deploy-1234567-8901

# List recent deployments
~/blackroad-deployment-verifier.sh list 10

# Deploy and verify in one step
~/blackroad-deployment-verifier.sh deploy cloudflare my-project abc1234 true

Supported Platforms:

• github - GitHub repositories
• cloudflare - Cloudflare Pages
• pi - Raspberry Pi devices
• railway - Railway projects

State Storage:

• Deployment history: ~/.blackroad/deployments/history/
• Rollback records: ~/.blackroad/deployments/rollbacks/

---

5. Master Deployment Orchestrator

Location: ~/blackroad-deploy-orchestrator.sh

Purpose: Unified interface for all deployment operations

Features:

• One command to deploy anywhere
• Integrated testing and verification
• Real-time monitoring
• Health check integration
• Rollback capabilities

Usage:

# Install all automation scripts
~/blackroad-deploy-orchestrator.sh install

# Run E2E tests
~/blackroad-deploy-orchestrator.sh test

# Run health checks
~/blackroad-deploy-orchestrator.sh health

# Deploy to specific platform
~/blackroad-deploy-orchestrator.sh deploy cloudflare my-project
~/blackroad-deploy-orchestrator.sh deploy github BlackRoad-OS/repo
~/blackroad-deploy-orchestrator.sh deploy pi lucidia
~/blackroad-deploy-orchestrator.sh deploy railway my-service

# Deploy everywhere
~/blackroad-deploy-orchestrator.sh deploy-all my-project

# Start monitoring
~/blackroad-deploy-orchestrator.sh monitor

# Show infrastructure status
~/blackroad-deploy-orchestrator.sh status

# Verify deployment
~/blackroad-deploy-orchestrator.sh verify deploy-1234567-8901

# Rollback deployment
~/blackroad-deploy-orchestrator.sh rollback deploy-1234567-8901

---

🔄 CI/CD Templates

Cloudflare Pages

Location: /tmp/ci-cd-templates/cloudflare-pages.yml

Copy to: .github/workflows/deploy-cloudflare.yml

Features:

• Quality checks (lint, type-check, tests)
• Build and artifact management
• Cloudflare Pages deployment
• Post-deployment verification
• Health checks
• Failure notifications

Required Secrets:

• CLOUDFLARE_API_TOKEN
• CLOUDFLARE_ACCOUNT_ID

---

Railway Deployment

Location: /tmp/ci-cd-templates/railway-deploy.yml

Copy to: .github/workflows/deploy-railway.yml

Features:

• Pre-deployment testing
• Railway CLI deployment
• Deployment URL extraction
• Health check verification

Required Secrets:

• RAILWAY_TOKEN
• RAILWAY_PROJECT_ID

---

Raspberry Pi Deployment

Location: /tmp/ci-cd-templates/pi-deployment.yml

Copy to: .github/workflows/deploy-pi.yml

Features:

• Multi-target deployment (lucidia, blackroad-pi, or all)
• SSH-based deployment
• Rsync file sync
• Systemd service restart
• Deployment verification

Required Secrets:

• PI_SSH_KEY

---

🚀 Quick Start Guide

1. Initial Setup (One-Time)

# Install all automation scripts
cd /tmp
chmod +x blackroad-deploy-orchestrator.sh
./blackroad-deploy-orchestrator.sh install

# Initialize health monitoring
~/blackroad-health-checks.sh init

# Run initial E2E tests
~/blackroad-e2e-testing-suite.sh

2. Daily Operations

# Morning: Check infrastructure health
~/blackroad-deploy-orchestrator.sh status

# Deploy changes
cd your-repo
git add .
git commit -m "feat: new feature"
git push

# Verify deployment
~/blackroad-deploy-orchestrator.sh verify <deployment-id>

# Monitor in real-time (optional)
~/blackroad-deploy-orchestrator.sh monitor

3. Emergency Response

# Deployment failed? Rollback immediately
~/blackroad-deploy-orchestrator.sh rollback <deployment-id>

# Check what's failing
~/blackroad-health-checks.sh check

# View recent alerts
~/blackroad-health-checks.sh alerts

---

📊 Production Standards

All BlackRoad-OS Repos Must Have:

1. CI/CD Pipeline

   • GitHub Actions workflow (use templates above)
   • Automated testing on PR
   • Automated deployment on merge to main

2. Health Checks

   • /health endpoint (for services)
   • Automated monitoring
   • Alerting on failures

3. Deployment Verification

   • Post-deployment tests
   • Rollback capability
   • Deployment tracking

4. Documentation

   • README with setup instructions
   • DEPLOYMENT.md with deployment guide
   • Architecture documentation

5. Testing

   • Unit tests (>70% coverage)
   • Integration tests
   • E2E tests for critical paths

---

🔧 Platform-Specific Setup

GitHub Repositories

Required Secrets:

• None (uses GITHUB_TOKEN automatically)

Setup:

1. Copy workflow template to .github/workflows/
2. Ensure tests pass locally
3. Push to trigger workflow

---

Cloudflare Pages

Required Secrets:

• CLOUDFLARE_API_TOKEN - Get from Cloudflare dashboard
• CLOUDFLARE_ACCOUNT_ID - Found in Cloudflare dashboard

Setup:

1. Create Cloudflare Pages project
2. Add secrets to GitHub repo
3. Copy workflow template
4. Configure build settings in workflow

---

Railway

Required Secrets:

• RAILWAY_TOKEN - Get from Railway dashboard
• RAILWAY_PROJECT_ID - Railway project ID

Setup:

1. Create Railway project
2. Link project: railway link
3. Get project ID: railway status
4. Add secrets to GitHub repo

---

Raspberry Pi

Required Secrets:

• PI_SSH_KEY - SSH private key for Pi access

Setup:

1. Generate SSH key: ssh-keygen -t ed25519 -C "github-actions"
2. Add public key to Pi: ~/.ssh/authorized_keys
3. Add private key to GitHub secrets
4. Test SSH: ssh -i key pi@192.168.4.38

---

📈 Monitoring & Alerting

Current Infrastructure Status

# Real-time dashboard
~/blackroad-realtime-monitor.sh monitor 30

Monitored Components:

• ✅ GitHub (4 orgs, 82 repos)
• ✅ Cloudflare (authenticated)
• ✅ Memory System (482 entries)
• ✅ Pi Lucidia (192.168.4.38)
• ✅ Pi BlackRoad (192.168.4.64)
• ❌ Pi Alt (192.168.4.99) - offline
• ❌ Port 8080 services - offline

Alert Channels

Configure in ~/.blackroad/health/config.json:

{
  "alerting": {
    "enabled": true,
    "email": "blackroad.systems@gmail.com",
    "slack_webhook": "",
    "discord_webhook": ""
  }
}

---

🎓 Best Practices

1. Always Test Before Deploy

# Run E2E tests
~/blackroad-e2e-testing-suite.sh

# Run health checks
~/blackroad-health-checks.sh check

2. Verify Every Deployment

# Deploy with verification
~/blackroad-deploy-orchestrator.sh deploy cloudflare my-project

# Manual verification
~/blackroad-deploy-orchestrator.sh verify <deployment-id>

3. Monitor Continuously

# Terminal 1: Monitoring
~/blackroad-realtime-monitor.sh monitor 30

# Terminal 2: Health checks
~/blackroad-health-checks.sh monitor 60

4. Keep Rollback Ready

# Every deployment is tracked
~/blackroad-deployment-verifier.sh list

# Rollback anytime
~/blackroad-deploy-orchestrator.sh rollback <deployment-id>

5. Log Everything

All operations are logged:

• E2E tests: ~/.blackroad/e2e-tests/
• Monitoring: ~/.blackroad/monitor/
• Health checks: ~/.blackroad/health/
• Deployments: ~/.blackroad/deployments/

---

🔐 Security Considerations

Secrets Management

• Never commit secrets to git
• Use GitHub Secrets for CI/CD
• Rotate tokens regularly
• Audit access logs

SSH Keys

• Use ed25519 keys
• One key per purpose
• Restrict key permissions (600)
• Monitor authorized_keys

API Tokens

• Minimum required scopes
• Time-limited when possible
• Monitor API usage
• Revoke unused tokens

---

📝 Notion Integration (Coming Soon)

Track deployment status in Notion:

Planned Features:

• Deployment dashboard
• Real-time status sync
• Alert notifications
• Deployment history
• Rollback tracking

Database Structure:

Deployments Database:
- ID
- Platform
- Target
- Commit SHA
- Status
- Timestamp
- Verification Result
- Rollback Info

---

🚦 Traffic Light System

Visual status indicators for all components:

• 🟢 Green: Healthy, no action needed
• 🟡 Yellow: Warning, monitor closely
• 🔴 Red: Critical, immediate attention required
• ⚫ Black: Offline, not reachable

---

📞 Support & Troubleshooting

Common Issues

1. GitHub Authentication Failed

# Re-authenticate
gh auth login

# Check status
gh auth status

2. Cloudflare Deployment Failed

# Re-authenticate
wrangler login

# Check status
wrangler whoami

3. Pi Device Unreachable

# Check network
ping 192.168.4.38

# Check SSH
ssh pi@192.168.4.38

# View logs
~/blackroad-health-checks.sh alerts

4. Deployment Verification Failed

# Check deployment logs
cat ~/.blackroad/deployments/history/*.json

# Manual verification
~/blackroad-deploy-orchestrator.sh status

# Rollback if needed
~/blackroad-deploy-orchestrator.sh rollback <deployment-id>

Getting Help

• Memory System: ~/memory-system.sh summary
• Issues: blackroad.systems@gmail.com
• Logs: ~/.blackroad/

---

🎯 Roadmap

Phase 1: Foundation ✅

• E2E testing suite
• Real-time monitoring
• Health checks
• Deployment verification
• CI/CD templates

Phase 2: Automation (In Progress)

• Notion integration
• Automated alerting (Slack/Discord)
• Auto-rollback on failures
• Performance monitoring
• Cost tracking

Phase 3: Intelligence (Planned)

• Predictive failure detection
• Automatic optimization
• AI-powered troubleshooting
• Self-healing infrastructure
• Capacity planning

---

📜 License

All BlackRoad automation scripts are proprietary to BlackRoad Systems.

Contact: blackroad.systems@gmail.com Documentation: This playbook Version: 1.0.0