blackroad-os/blackroad-operating-system
9
0
Fork 0
mirror of https://github.com/blackboxprogramming/BlackRoad-Operating-System.git synced 2026-03-18 02:33:59 -05:00
Code Issues Packages Projects Releases Wiki Activity

Add comprehensive BlackRoad ecosystem implementation plans

Browse Source 
This commit adds detailed implementation plans mapping all 23 BlackRoad
repositories to the 7-layer architecture defined in MASTER_ORCHESTRATION_PLAN.md.

New Documentation:
- ORG_STRUCTURE.md: Complete repo architecture & responsibility map
- IMPLEMENTATION.md: Detailed plan for BlackRoad-Operating-System monolith
- CLOUDFLARE_DNS_BLUEPRINT.md: DNS configuration with repo ownership map

Implementation Plans (in implementation-plans/):
- IMPLEMENTATION_blackroad-api.md: Standalone API gateway (Phase 2)
- IMPLEMENTATION_blackroad-operator.md: Agent orchestration & workflows (Phase 2)
- IMPLEMENTATION_blackroad-prism-console.md: Admin dashboard (Phase 2)
- IMPLEMENTATION_blackroad-io.md: Corporate marketing site (Phase 1)
- IMPLEMENTATION_lucidia.md: Multi-model AI orchestration (Phase 1-2)
- IMPLEMENTATION_blackroad.md: Investigation template for unknown repo

Key Decisions:
- Monolith strategy for Phase 1 (months 0-12)
- Strategic split to microservices in Phase 2 (months 12-18)
- 4 core active repos in Phase 1, expand to 10+ in Phase 2-3
- Cloudflare DNS for all domains with clear repo ownership

Each implementation plan includes:
- Purpose & final role in architecture
- Required GitHub Actions workflows
- Secrets & environment variables
- Cloudflare DNS configuration
- Migration notes from monolith
- Phase-specific milestones
- Success criteria

Ready for Phase 1 execution starting with Week 1 infrastructure tasks.
This commit is contained in:
Claude
2025-11-18 02:29:57 +00:00
parent 11a4f1e73a
commit 0529a05942
9 changed files with 3724 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

513
CLOUDFLARE_DNS_BLUEPRINT.md Normal file
View File

@@ -0,0 +1,513 @@
# ☁️ CLOUDFLARE DNS BLUEPRINT
## Multi-Domain DNS Configuration & Repo Responsibility Map
**Version**: 1.0
**Date**: 2025-11-18
**Source**: Extracted from MASTER_ORCHESTRATION_PLAN.md + ORG_STRUCTURE.md
---
## EXECUTIVE SUMMARY
This document maps **10+ BlackRoad domains** to:
- Cloudflare DNS records
- Repository ownership
- Deployment targets
- SSL configuration
**DNS Strategy**: Cloudflare nameservers (migrated from GoDaddy) for all domains
**SSL Strategy**: Full (strict) with automatic SSL via Cloudflare + Railway/Vercel
---
## PART 1: DOMAIN INVENTORY
### Primary Domains (Phase 1)
| Domain | Purpose | Owner Repo | Status | Phase |
|--------|---------|------------|--------|-------|
| **blackroad.systems** | Corporate site | blackroad.io | 🎯 Primary | 1 |
| **blackroad.ai** | Alias to OS | BlackRoad-Operating-System | Active | 1 |
| **blackroad.network** | Developer docs | BlackRoad-Operating-System | Planned | 1 |
| **blackroad.me** | Personal identity | BlackRoad-Operating-System | Planned | 1 |
### Secondary Domains (Phase 2)
| Domain | Purpose | Owner Repo | Status | Phase |
|--------|---------|------------|--------|-------|
| **aliceqi.com** | ALICE QI engine | lucidia / quantum-math-lab | Research | 2 |
| **blackroadqi.com** | Financial intelligence | blackroad-api (QI module) | Planned | 2 |
| **lucidia.earth** | Narrative experiences | lucidia | Development | 2 |
| **blackroadquantum.com** | Research hub | quantum-math-lab | Research | 2 |
### Tertiary Domains (Phase 3)
| Domain | Purpose | Owner Repo | Status | Phase |
|--------|---------|------------|--------|-------|
| **roadwallet.com** | Wallet service | BlackRoad-Operating-System | Alias | 3 |
| **aliceos.io** | Legacy alias | BlackRoad-Operating-System | Legacy | 3 |
| **blackroadquantum.net** | Quantum APIs | quantum-math-lab | Planned | 3 |
| **blackroadquantum.info** | Education hub | quantum-math-lab | Planned | 3 |
| **blackroadquantum.store** | Merch/courses | TBD (e-commerce repo) | Planned | 3 |
| **lucidia.studio** | Creative production | lucidia | Planned | 3 |
| **blackroad.store** | Community commerce | TBD (e-commerce repo) | Planned | 3 |
---
## PART 2: DNS RECORDS BY DOMAIN
### blackroad.systems (Primary Corporate Site)
**Zone ID**: `[Get from Cloudflare dashboard]`
**Registrar**: GoDaddy → **Migrate nameservers to Cloudflare**
**Owner Repo**: `blackboxprogramming/blackroad.io`
#### DNS Records
| Type | Name | Target | Proxy | TTL | Purpose | Responsible Repo |
|------|------|--------|-------|-----|---------|------------------|
| CNAME | @ | `cname.vercel-dns.com` | ✅ | Auto | Corporate site | blackroad.io |
| CNAME | www | `blackroad.systems` | ✅ | Auto | www redirect | blackroad.io |
| CNAME | os | `blackroad-os-production.up.railway.app` | ✅ | Auto | OS interface | BlackRoad-Operating-System |
| CNAME | api | `blackroad-api-production.up.railway.app` | ✅ | Auto | API gateway | blackroad-api (Phase 2) |
| CNAME | prism | `blackroad-prism-console.vercel.app` | ✅ | Auto | Prism Console | blackroad-prism-console |
| CNAME | operator | `blackroad-operator.up.railway.app` | ❌ | Auto | Operator (internal) | blackroad-operator |
| CNAME | lucidia | `lucidia-api.up.railway.app` | ✅ | Auto | Lucidia API | lucidia |
| CNAME | docs | `blackboxprogramming.github.io` | ✅ | Auto | Developer docs | BlackRoad-Operating-System |
| TXT | @ | `v=spf1 include:_spf.google.com ~all` | - | Auto | Email SPF | - |
| MX | @ | `1 aspmx.l.google.com` | - | Auto | Email MX | - |
**Cloudflare Settings**:
- SSL/TLS: **Full (strict)**
- Always Use HTTPS: **Enabled**
- Auto Minify: JavaScript, CSS, HTML
- Brotli: **Enabled**
- Cache Level: Standard
---
### blackroad.ai (OS Alias)
**Registrar**: GoDaddy
**Owner Repo**: `blackboxprogramming/BlackRoad-Operating-System`
#### DNS Records
| Type | Name | Target | Proxy | Purpose | Responsible Repo |
|------|------|--------|-------|---------|------------------|
| CNAME | @ | `os.blackroad.systems` | ✅ | Alias to OS | BlackRoad-Operating-System |
| CNAME | www | `blackroad.ai` | ✅ | www redirect | BlackRoad-Operating-System |
**Page Rule**:
```
blackroad.ai/*
→ Forwarding URL (301 - Permanent Redirect)
→ https://os.blackroad.systems/$1
```
---
### blackroad.network (Developer Portal)
**Registrar**: GoDaddy
**Owner Repo**: `blackboxprogramming/BlackRoad-Operating-System` (docs/ directory)
#### DNS Records
| Type | Name | Target | Proxy | Purpose | Responsible Repo |
|------|------|--------|-------|---------|------------------|
| CNAME | @ | `blackboxprogramming.github.io` | ✅ | Developer docs | BlackRoad-Operating-System/docs/ |
| CNAME | www | `blackroad.network` | ✅ | www redirect | BlackRoad-Operating-System/docs/ |
| CNAME | api | `blackroad-api-production.up.railway.app` | ✅ | API for developers | blackroad-api |
**GitHub Pages Setup** (in BlackRoad-Operating-System repo):
1. Enable GitHub Pages from `docs/` directory
2. Add custom domain: `blackroad.network`
3. Enforce HTTPS
4. Cloudflare DNS points to GitHub Pages
---
### blackroad.me (Personal Identity)
**Registrar**: GoDaddy
**Owner Repo**: `blackboxprogramming/BlackRoad-Operating-System`
#### DNS Records
| Type | Name | Target | Proxy | Purpose | Responsible Repo |
|------|------|--------|-------|---------|------------------|
| CNAME | @ | `os.blackroad.systems` | ✅ | Identity portal | BlackRoad-Operating-System |
| CNAME | www | `blackroad.me` | ✅ | www redirect | BlackRoad-Operating-System |
**Host-Based Routing** (in BlackRoad-Operating-System):
```python
# backend/app/middleware/domain_routing.py
from fastapi import Request
async def domain_middleware(request: Request, call_next):
host = request.headers.get("host")
if host == "blackroad.me":
# Serve identity portal theme
request.state.theme = "identity"
response = await call_next(request)
return response
```
---
### lucidia.earth (Narrative Site)
**Registrar**: GoDaddy
**Owner Repo**: `blackboxprogramming/lucidia`
#### DNS Records
| Type | Name | Target | Proxy | Purpose | Responsible Repo |
|------|------|--------|-------|---------|------------------|
| CNAME | @ | `lucidia-narrative.vercel.app` | ✅ | Narrative site | lucidia |
| CNAME | www | `lucidia.earth` | ✅ | www redirect | lucidia |
| CNAME | api | `lucidia-api.up.railway.app` | ✅ | Lucidia API | lucidia |
**Phase 2 Launch** (Month 12+)
---
### aliceqi.com (ALICE QI Research)
**Registrar**: GoDaddy
**Owner Repo**: `blackboxprogramming/quantum-math-lab` or `lucidia-lab`
#### DNS Records
| Type | Name | Target | Proxy | Purpose | Responsible Repo |
|------|------|--------|-------|---------|------------------|
| CNAME | @ | `aliceqi-research.vercel.app` | ✅ | Research site | quantum-math-lab |
| CNAME | www | `aliceqi.com` | ✅ | www redirect | quantum-math-lab |
**Phase 2 Launch** (Month 12+)
---
### roadwallet.com (Wallet Alias)
**Registrar**: GoDaddy
**Owner Repo**: `blackboxprogramming/BlackRoad-Operating-System`
#### DNS Records
| Type | Name | Target | Proxy | Purpose | Responsible Repo |
|------|------|--------|-------|---------|------------------|
| CNAME | @ | `os.blackroad.systems` | ✅ | Alias to OS wallet | BlackRoad-Operating-System |
| CNAME | www | `roadwallet.com` | ✅ | www redirect | BlackRoad-Operating-System |
**Page Rule**: Redirect to `os.blackroad.systems#wallet` (deep link to Wallet app)
---
### aliceos.io (Legacy Alias)
**Registrar**: GoDaddy
**Owner Repo**: `blackboxprogramming/BlackRoad-Operating-System`
#### DNS Records
| Type | Name | Target | Proxy | Purpose | Responsible Repo |
|------|------|--------|-------|---------|------------------|
| CNAME | @ | `os.blackroad.systems` | ✅ | Legacy alias | BlackRoad-Operating-System |
| CNAME | www | `aliceos.io` | ✅ | www redirect | BlackRoad-Operating-System |
**Note**: Consider deprecating or redirecting to blackroad.systems in Phase 2
---
## PART 3: REPO RESPONSIBILITY MAP
### Canonical Ownership Table
| Subdomain / Domain | Repo | Service Type | Deployment Target | Phase |
|--------------------|------|--------------|-------------------|-------|
| **blackroad.systems** | blackroad.io | Static site (Astro) | Vercel | 1 |
| **os.blackroad.systems** | BlackRoad-Operating-System | FastAPI + static UI | Railway | 1 |
| **api.blackroad.systems** | blackroad-api | FastAPI API | Railway | 2 |
| **prism.blackroad.systems** | blackroad-prism-console | React SPA | Vercel | 2 |
| **operator.blackroad.systems** | blackroad-operator | Worker service | Railway | 2 |
| **lucidia.blackroad.systems** | lucidia | FastAPI AI service | Railway | 1/2 |
| **docs.blackroad.systems** | BlackRoad-Operating-System | GitHub Pages (docs/) | GitHub Pages | 1 |
| **blackroad.network** | BlackRoad-Operating-System | GitHub Pages (docs/) | GitHub Pages | 1 |
| **blackroad.me** | BlackRoad-Operating-System | Identity portal | Railway | 1 |
| **lucidia.earth** | lucidia | Narrative site | Vercel | 2 |
| **aliceqi.com** | quantum-math-lab | Research site | Vercel | 2 |
---
## PART 4: CLOUDFLARE MIGRATION CHECKLIST
### Per-Domain Migration (Repeat for all domains)
#### Step 1: Add Domain to Cloudflare
- [ ] Log in to Cloudflare dashboard
- [ ] Click "Add a site"
- [ ] Enter domain (e.g., `blackroad.systems`)
- [ ] Choose Free plan
- [ ] Cloudflare scans existing DNS records from GoDaddy
- [ ] Review imported records, add missing ones
#### Step 2: Update Nameservers
- [ ] Cloudflare provides 2 nameservers (e.g., `aaaa.ns.cloudflare.com`, `bbbb.ns.cloudflare.com`)
- [ ] Log in to GoDaddy
- [ ] Go to domain → Manage DNS → Nameservers
- [ ] Switch from GoDaddy to Custom
- [ ] Enter Cloudflare nameservers
- [ ] Save (propagation: 5-60 minutes)
#### Step 3: Verify Active
- [ ] Wait for Cloudflare to detect nameserver change
- [ ] Cloudflare dashboard should say "Active" (not "Pending")
- [ ] Test DNS resolution: `dig blackroad.systems` (should show Cloudflare IPs)
#### Step 4: Configure SSL
- [ ] Cloudflare → SSL/TLS → Set to "Full (strict)"
- [ ] SSL/TLS → Edge Certificates → Enable "Always Use HTTPS"
- [ ] SSL/TLS → Edge Certificates → Enable "Automatic HTTPS Rewrites"
#### Step 5: Configure Performance
- [ ] Speed → Optimization → Enable Auto Minify (JS, CSS, HTML)
- [ ] Speed → Optimization → Enable Brotli
- [ ] Caching → Configuration → Cache Level: Standard
#### Step 6: Test
- [ ] Visit `https://yourdomain.com` → Should load with 🔒
- [ ] Visit `http://yourdomain.com` → Should redirect to HTTPS
- [ ] Test API: `curl https://os.blackroad.systems/health`
### Domains to Migrate (Priority Order)
**Week 1**:
1. [ ] blackroad.systems (corporate site - highest priority)
2. [ ] blackroad.ai (OS alias)
3. [ ] blackroad.me (identity)
**Week 2**:
4. [ ] blackroad.network (developer docs)
5. [ ] roadwallet.com (wallet alias)
**Phase 2** (Month 12+):
6. [ ] lucidia.earth
7. [ ] aliceqi.com
8. [ ] blackroadqi.com
9. [ ] blackroadquantum.com
---
## PART 5: AUTOMATION SCRIPTS
### DNS Sync Script (Planned)
**File**: `scripts/cloudflare/sync_dns.py`
```python
#!/usr/bin/env python3
"""
Sync DNS records from config to Cloudflare
Usage: python scripts/cloudflare/sync_dns.py --domain blackroad.systems
"""
import os
import yaml
import requests
from typing import Dict, List
CF_API_TOKEN = os.getenv("CF_API_TOKEN")
CF_ZONE_ID = os.getenv("CF_ZONE_ID")
def load_config(domain: str) -> Dict:
"""Load DNS config from ops/domains/{domain}.yaml"""
with open(f"ops/domains/{domain}.yaml") as f:
return yaml.safe_load(f)
def get_existing_records(zone_id: str) -> List[Dict]:
"""Fetch existing DNS records from Cloudflare"""
url = f"https://api.cloudflare.com/client/v4/zones/{zone_id}/dns_records"
headers = {"Authorization": f"Bearer {CF_API_TOKEN}"}
response = requests.get(url, headers=headers)
return response.json()["result"]
def create_dns_record(zone_id: str, record: Dict):
"""Create DNS record in Cloudflare"""
url = f"https://api.cloudflare.com/client/v4/zones/{zone_id}/dns_records"
headers = {
"Authorization": f"Bearer {CF_API_TOKEN}",
"Content-Type": "application/json"
}
response = requests.post(url, headers=headers, json=record)
return response.json()
def update_dns_record(zone_id: str, record_id: str, record: Dict):
"""Update existing DNS record"""
url = f"https://api.cloudflare.com/client/v4/zones/{zone_id}/dns_records/{record_id}"
headers = {
"Authorization": f"Bearer {CF_API_TOKEN}",
"Content-Type": "application/json"
}
response = requests.put(url, headers=headers, json=record)
return response.json()
def sync_domain(domain: str):
"""Sync DNS records for a domain"""
print(f"Syncing DNS for {domain}...")
config = load_config(domain)
existing = get_existing_records(CF_ZONE_ID)
for record in config["dns_records"]:
# Check if record exists
existing_record = next((r for r in existing if r["name"] == record["name"] and r["type"] == record["type"]), None)
if existing_record:
print(f" Updating {record['type']} {record['name']}")
update_dns_record(CF_ZONE_ID, existing_record["id"], record)
else:
print(f" Creating {record['type']} {record['name']}")
create_dns_record(CF_ZONE_ID, record)
print(f"✅ Sync complete for {domain}")
if __name__ == "__main__":
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("--domain", required=True)
args = parser.parse_args()
sync_domain(args.domain)
```
**Config File Example** (`ops/domains/blackroad.systems.yaml`):
```yaml
domain: blackroad.systems
zone_id: your-zone-id
dns_records:
- type: CNAME
name: "@"
content: cname.vercel-dns.com
proxied: true
ttl: 1 # Auto
- type: CNAME
name: www
content: blackroad.systems
proxied: true
ttl: 1
- type: CNAME
name: os
content: blackroad-os-production.up.railway.app
proxied: true
ttl: 1
# ... more records
```
**Usage**:
```bash
export CF_API_TOKEN="your-token"
export CF_ZONE_ID="your-zone-id"
python scripts/cloudflare/sync_dns.py --domain blackroad.systems
```
---
## PART 6: MONITORING & HEALTH CHECKS
### Domain Health Check Workflow
**File**: `.github/workflows/domain-health.yml`
```yaml
name: Domain Health
on:
schedule:
- cron: '0 */6 * * *' # Every 6 hours
workflow_dispatch:
jobs:
health:
runs-on: ubuntu-latest
strategy:
matrix:
domain:
- https://blackroad.systems
- https://os.blackroad.systems
- https://api.blackroad.systems
- https://prism.blackroad.systems
- https://blackroad.network
steps:
- name: Check ${{ matrix.domain }}
run: |
STATUS=$(curl -s -o /dev/null -w "%{http_code}" ${{ matrix.domain }}/health || echo "000")
if [ "$STATUS" != "200" ]; then
echo "❌ ${{ matrix.domain }} is down (status: $STATUS)"
exit 1
else
echo "✅ ${{ matrix.domain }} is up"
fi
- name: Check SSL
run: |
echo | openssl s_client -servername $(echo ${{ matrix.domain }} | sed 's/https:\/\///') -connect $(echo ${{ matrix.domain }} | sed 's/https:\/\///'):443 2>/dev/null | openssl x509 -noout -dates
```
---
## PART 7: COST SUMMARY
### Cloudflare Costs
**Free Tier** (all Phase 1 domains):
- Unlimited DNS queries
- SSL certificates (automatic)
- DDoS protection (unmetered)
- CDN caching (100 GB/month)
- 3 Page Rules per domain
**Pro Tier** ($20/mo per domain, if needed):
- More Page Rules
- Image optimization
- Mobile redirect
- Polish (WebP/AVIF)
**Recommendation**: Stay on Free tier for Phase 1
### GoDaddy Costs
**Domain Registration** (annual):
- .systems: ~$15/year
- .com: ~$12/year
- .ai: ~$90/year (premium TLD)
- .earth: ~$20/year
- .me: ~$20/year
- .io: ~$40/year
**Total Annual**: ~$200-300/year for all domains
**DNS Hosting**: $0 (migrated to Cloudflare)
---
## CONCLUSION
**Current State**: Domains registered with GoDaddy, DNS managed by GoDaddy
**Target State**: Domains registered with GoDaddy, DNS managed by Cloudflare
**Migration Effort**: 1-2 days for Phase 1 domains
**Next Action**: Start with `blackroad.systems` migration (see NEXT_ACTIONS_ALEXA.md, Item #1)
---
**Last Updated**: 2025-11-18
**Next Review**: After Phase 1 DNS migration complete (Week 2)

953
IMPLEMENTATION.md Normal file
View File

@@ -0,0 +1,953 @@
# 🚀 IMPLEMENTATION PLAN: BlackRoad-Operating-System
## Monolith → Core OS Platform
**Repo**: `blackboxprogramming/BlackRoad-Operating-System`
**Branch**: `claude/implementation-plan/blackroad-operating-system`
**Version**: 1.0
**Date**: 2025-11-18
**Phase**: **Phase 1 (Months 0-12) - Prove the OS**
---
## EXECUTIVE SUMMARY
**BlackRoad-Operating-System** is the **canonical monolith** containing the complete OS stack:
- Windows 95-inspired web UI (frontend)
- FastAPI backend (33 routers, 100+ endpoints)
- 200+ autonomous agents across 10 categories
- Python & TypeScript SDKs
- Complete documentation & infrastructure
**Current State**: 65% complete toward vision, production-ready for Phase 1 pilots
**Target State**: Stable v1.0 release, 5 enterprise design partners, foundation for Phase 2 splits
**Role in 7-Layer Architecture**:
- **Layer 1** (DNS/CDN): Cloudflare DNS scripts
- **Layer 2** (Compute): Railway deployment config
- **Layer 3** (Data): PostgreSQL + Redis + RoadChain
- **Layer 4** (Orchestration): Agent base framework (Lucidia/Prism planned)
- **Layer 5** (API Gateway): FastAPI with 33 routers
- **Layer 6** (Application): Windows 95 UI shell
- **Layer 7** (User Experience): OS interface at os.blackroad.systems
---
## PART 1: PURPOSE & FINAL ROLE
### Current Purpose (Phase 1)
**The monolith is the complete product.** It owns:
- User authentication & identity
- All backend APIs (email, social, video, blockchain, etc.)
- Frontend OS shell and embedded apps
- Agent library and execution framework
- Database schemas for all features
- Deployment infrastructure
### Final Role (Phase 2+)
**The monolith becomes the OS core.** It retains:
- OS shell (Windows 95 UI)
- Core identity system (PS-SHA∞)
- RoadChain audit ledger
- Wallet & blockchain primitives
**What splits out to other repos**:
- API gateway → `blackroad-api`
- Agent orchestration → `blackroad-operator`
- Admin UI → `blackroad-prism-console`
- Video platform → `BlackStream`
- Corporate site → `blackroad.io`
### Strategic Position
```
┌─────────────────────────────────────────────────────────────┐
│ PHASE 1 (Current) │
│ ┌─────────────────────────────────────────────────────────┐ │
│ │ BlackRoad-Operating-System (Monolith) │ │
│ │ ├── Frontend (UI shell) │ │
│ │ ├── Backend (API gateway) │ │
│ │ ├── Agents (200+ autonomous agents) │ │
│ │ ├── SDKs (Python, TypeScript) │ │
│ │ └── Docs (architecture, guides) │ │
│ └─────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────┐
│ PHASE 2 (Target) │
│ ┌───────────────────┐ ┌──────────────┐ ┌───────────────┐ │
│ │ BlackRoad-OS │ │ blackroad-api│ │ blackroad- │ │
│ │ (Core) │ │ (Gateway) │ │ operator │ │
│ │ - UI shell │←─│ - 33 routers │←─│ - Agents │ │
│ │ - Identity │ │ - REST API │ │ - Scheduler │ │
│ │ - RoadChain │ │ - WebSocket │ │ - Workflows │ │
│ └───────────────────┘ └──────────────┘ └───────────────┘ │
│ ↓ ↓ ↓ │
│ PostgreSQL Redis Queue │
└─────────────────────────────────────────────────────────────┘
```
**Decision**: **Stay monolith for Phase 1** (0-12 months), split in Phase 2 (12-18 months)
---
## PART 2: REQUIRED WORKFLOWS
### Current GitHub Actions (7 workflows)
| Workflow | File | Status | Next Actions |
|----------|------|--------|--------------|
| **CI** | `.github/workflows/ci.yml` | ✅ Green | Add type checking for Python |
| **Backend Tests** | `.github/workflows/backend-tests.yml` | ✅ Green | Increase coverage to 80% |
| **Deploy to Pages** | `.github/workflows/deploy.yml` | ✅ Green | Verify GitHub Pages active |
| **Railway Deploy** | `.github/workflows/railway-deploy.yml` | ✅ Green | Add deployment health check |
| **Railway Automation** | `.github/workflows/railway-automation.yml` | ✅ Green | Add secret rotation check |
| **Domain Health** | `.github/workflows/domain-health.yml` | 🟡 Yellow | Configure after DNS migration |
| **Sync Domains** | `.github/workflows/sync-domains.yml` | 🟡 Yellow | Implement Cloudflare script |
### Workflows to Add (Phase 1)
#### 1. Security Scanning (`.github/workflows/codeql.yml`)
**Purpose**: Automated code security analysis
**Trigger**: Weekly, PR to main
**Technology**: GitHub CodeQL
```yaml
name: Security Scan
on:
push:
branches: [main]
pull_request:
branches: [main]
schedule:
- cron: '0 0 * * 0' # Weekly on Sunday
jobs:
codeql:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: github/codeql-action/init@v2
with:
languages: python, javascript
- uses: github/codeql-action/analyze@v2
```
**Effort**: 30 minutes to set up
---
#### 2. Dependency Updates (`.github/dependabot.yml`)
**Purpose**: Auto-create PRs for dependency updates
**Trigger**: Daily
**Technology**: Dependabot
```yaml
version: 2
updates:
- package-ecosystem: pip
directory: "/backend"
schedule:
interval: daily
open-pull-requests-limit: 5
- package-ecosystem: npm
directory: "/sdk/typescript"
schedule:
interval: weekly
```
**Effort**: 15 minutes to configure
---
#### 3. PR Labeler (`.github/workflows/pr-labeler.yml`)
**Purpose**: Auto-label PRs based on changed files
**Trigger**: PR opened/synchronized
**Technology**: actions/labeler
```yaml
name: PR Labeler
on:
pull_request:
types: [opened, synchronize]
jobs:
label:
runs-on: ubuntu-latest
steps:
- uses: actions/labeler@v4
with:
repo-token: ${{ secrets.GITHUB_TOKEN }}
configuration-path: .github/labeler.yml
```
**Config** (`.github/labeler.yml`):
```yaml
backend:
- backend/**
frontend:
- backend/static/**
- blackroad-os/**
agents:
- agents/**
docs:
- docs/**
- README.md
- CLAUDE.md
infra:
- .github/workflows/**
- scripts/**
- ops/**
```
**Effort**: 30 minutes
---
#### 4. Release Automation (`.github/workflows/release.yml`)
**Purpose**: Auto-generate changelog & create GitHub release
**Trigger**: Manual workflow_dispatch or tag push
**Technology**: release-drafter
```yaml
name: Release
on:
push:
tags:
- 'v*'
workflow_dispatch:
jobs:
release:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: release-drafter/release-drafter@v5
with:
config-name: release-drafter.yml
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
```
**Effort**: 1 hour (includes changelog template)
---
### Branch Protection Rules
**Branch**: `main`
- ✅ Require pull request before merging
- ✅ Require 1 approval (can be self-approved for solo dev)
- ✅ Require status checks:
- `CI / lint`
- `CI / test-backend`
- `Security Scan / codeql` (once added)
- ✅ Require branches to be up to date before merging
- ❌ Do NOT allow force pushes
- ❌ Do NOT allow deletions
**Branch**: `claude/*` (for AI agent work)
- ✅ Allow direct commits
- ✅ Require status checks (optional)
- ❌ No protection (temporary branches)
**Setup Command**:
```bash
gh api \
--method PUT \
-H "Accept: application/vnd.github+json" \
/repos/blackboxprogramming/BlackRoad-Operating-System/branches/main/protection \
-f required_status_checks='{"strict":true,"contexts":["CI / lint","CI / test-backend"]}' \
-f enforce_admins=false \
-f required_pull_request_reviews='{"required_approving_review_count":1}' \
-f restrictions=null
```
---
## PART 3: SECRETS & ENVIRONMENT VARIABLES
### Required GitHub Secrets
| Secret | Purpose | How to Get | Status |
|--------|---------|-----------|--------|
| **RAILWAY_TOKEN** | Deploy to Railway | `railway login --browserless` | 🟡 Add |
| **CF_API_TOKEN** | Cloudflare DNS sync | Cloudflare dashboard → API Tokens | 🟡 Add |
| **CF_ZONE_ID** | Cloudflare zone ID | Cloudflare dashboard → Zone overview | 🟡 Add |
| **SENTRY_DSN** | Error monitoring | Sentry project settings | 🟡 Optional |
| **CODECOV_TOKEN** | Coverage reporting | CodeCov project settings | 🟡 Optional |
**Setup Commands**:
```bash
# Add Railway token
railway login --browserless # Copy token from output
gh secret set RAILWAY_TOKEN --body "your-railway-token"
# Add Cloudflare tokens (get from dashboard)
gh secret set CF_API_TOKEN --body "your-cloudflare-token"
gh secret set CF_ZONE_ID --body "your-zone-id"
# Verify
gh secret list
```
### Railway Environment Variables (26 required)
From `backend/.env.example`:
**Core** (required):
```bash
ENVIRONMENT=production
DEBUG=False
SECRET_KEY=<openssl rand -hex 32>
DATABASE_URL=${{Postgres.DATABASE_URL}}
DATABASE_ASYNC_URL=${{Postgres.DATABASE_ASYNC_URL}}
REDIS_URL=${{Redis.REDIS_URL}}
ALLOWED_ORIGINS=https://os.blackroad.systems,https://blackroad.ai
PORT=8000
API_BASE_URL=https://os.blackroad.systems
FRONTEND_URL=https://os.blackroad.systems
WALLET_MASTER_KEY=<generated securely>
```
**AI/ML** (Phase 1):
```bash
OPENAI_API_KEY=sk-...
ANTHROPIC_API_KEY=sk-ant-... # For Lucidia in Phase 2
```
**Cloud** (optional, Phase 2):
```bash
AWS_ACCESS_KEY_ID=AKIA...
AWS_SECRET_ACCESS_KEY=...
STRIPE_SECRET_KEY=sk_live_...
STRIPE_PUBLISHABLE_KEY=pk_live_...
```
**Communication** (optional):
```bash
SMTP_HOST=smtp.gmail.com
SMTP_PORT=587
SMTP_USER=noreply@blackroad.systems
SMTP_PASSWORD=...
TWILIO_ACCOUNT_SID=AC...
TWILIO_AUTH_TOKEN=...
SLACK_BOT_TOKEN=xoxb-...
DISCORD_BOT_TOKEN=...
```
**Monitoring** (optional):
```bash
SENTRY_DSN=https://...
PROMETHEUS_ENABLED=True
```
**Setup Process**:
```bash
# 1. Create .env.production from template
cp backend/.env.example backend/.env.production
# 2. Generate secrets
openssl rand -hex 32 # For SECRET_KEY
openssl rand -hex 32 # For WALLET_MASTER_KEY
# 3. Fill in values in .env.production
# 4. Upload to Railway
railway variables set --file backend/.env.production
# Or set individually
railway variables set SECRET_KEY=<value>
railway variables set WALLET_MASTER_KEY=<value>
# ... etc
```
**Validation**:
```bash
# Run validation script
python scripts/railway/validate_env_template.py
```
---
## PART 4: CLOUDFLARE & DOMAIN WIRING
### Domains Owned by This Repo
| Domain | Purpose | CNAME Target | Status |
|--------|---------|--------------|--------|
| **os.blackroad.systems** | OS interface (canonical) | `blackroad-os-production.up.railway.app` | 🎯 Primary |
| **api.blackroad.systems** | Explicit API subdomain | `blackroad-os-production.up.railway.app` | Phase 1 |
| **blackroad.ai** | Alias to OS | `os.blackroad.systems` | Phase 1 |
| **blackroad.me** | Personal identity portal | `os.blackroad.systems` | Phase 1 |
### DNS Records Configuration
**Add to Cloudflare** (for `blackroad.systems` zone):
| Type | Name | Target | Proxy | TTL | Notes |
|------|------|--------|-------|-----|-------|
| CNAME | os | `blackroad-os-production.up.railway.app` | ✅ | Auto | Canonical OS URL |
| CNAME | api | `blackroad-os-production.up.railway.app` | ✅ | Auto | Explicit API access |
| CNAME | @ | `blackroad-os-production.up.railway.app` | ✅ | Auto | Root domain (CNAME flattening) |
| CNAME | www | `blackroad.systems` | ✅ | Auto | www redirect |
**Setup Process**:
1. Add `blackroad.systems` to Cloudflare (see NEXT_ACTIONS_ALEXA.md)
2. Update nameservers at GoDaddy
3. Wait for DNS propagation (5-60 minutes)
4. Configure Railway custom domain
5. Test: `curl https://os.blackroad.systems/health`
**Automation** (Phase 1, Week 2):
```bash
# Create script: scripts/cloudflare/sync_dns.py
# Reference: MASTER_ORCHESTRATION_PLAN.md Part 2
python scripts/cloudflare/sync_dns.py \
--zone-id $CF_ZONE_ID \
--config ops/domains.yaml
```
**Health Checks** (`.github/workflows/domain-health.yml`):
```yaml
name: Domain Health
on:
schedule:
- cron: '0 */6 * * *' # Every 6 hours
workflow_dispatch:
jobs:
health:
runs-on: ubuntu-latest
steps:
- name: Check os.blackroad.systems
run: |
curl -f https://os.blackroad.systems/health || exit 1
- name: Check API
run: |
curl -f https://api.blackroad.systems/health || exit 1
```
---
## PART 5: MIGRATION NOTES
### Deprecated: `blackroad-os/` Directory
**Issue**: Duplicate frontend codebase
| Location | Status | Action |
|----------|--------|--------|
| `backend/static/` | **CANONICAL** | ✅ Maintain & deploy |
| `blackroad-os/` | **LEGACY** | 🚫 Deprecate, mark read-only |
**Steps to Deprecate**:
1. Add warning README:
```markdown
# ⚠️ DEPRECATED: Legacy UI
This directory is **superseded** by `backend/static/`.
**Do not edit files here.** They will not be deployed.
The canonical OS interface is served from `backend/static/index.html` by FastAPI.
This directory is preserved for reference only and may be removed in a future release.
```
2. Update all documentation:
- Replace references to `blackroad-os/` with `backend/static/`
- Update CLAUDE.md, README.md, CODEBASE_STATUS.md
3. Update CI/CD:
- Remove validation of `blackroad-os/` from workflows
- Only validate `backend/static/`
4. Archive after verification:
```bash
git mv blackroad-os/ archive/blackroad-os-legacy/
git commit -m "Archive legacy frontend (superseded by backend/static)"
```
**Timeline**: Week 1 (immediate)
---
### What to Migrate OUT (Phase 2, Months 6-12)
#### 1. API Gateway → `blackroad-api`
**Target Repo**: `blackboxprogramming/blackroad-api`
**What Moves**:
- `backend/app/routers/` → All 33 routers
- `backend/app/models/` → Database models
- `backend/app/services/` → Business logic
- `backend/app/utils/` → Shared utilities
**What Stays**:
- `backend/static/` → OS shell frontend
- `backend/app/config.py` → Core settings (duplicated in both repos)
- `backend/app/main.py` → Simplified (just serves static files + proxy to API)
**Migration Script** (create in Phase 1 for Phase 2 use):
```bash
#!/bin/bash
# scripts/migrate_api.sh
# Create new repo
gh repo create blackboxprogramming/blackroad-api --public
# Clone both repos
git clone https://github.com/blackboxprogramming/BlackRoad-Operating-System.git os
git clone https://github.com/blackboxprogramming/blackroad-api.git api
# Copy API code
cp -r os/backend/app/routers api/app/
cp -r os/backend/app/models api/app/
cp -r os/backend/app/services api/app/
cp -r os/backend/app/utils api/app/
# Update imports (manual or script)
# Push to new repo
cd api
git add .
git commit -m "Migrate API from monolith"
git push origin main
```
**Effort**: 2-3 weeks (includes testing, deployment)
---
#### 2. Agent Orchestration → `blackroad-operator`
**Target Repo**: `blackboxprogramming/blackroad-operator`
**What Moves**:
- `agents/` → All 200+ agents
- New: Scheduler (cron-like for recurring agent jobs)
- New: Prism job queue integration
**What Stays**:
- Agent API endpoint in `backend/app/routers/agents.py` (becomes proxy)
**Migration Script**:
```bash
#!/bin/bash
# scripts/migrate_agents.sh
gh repo create blackboxprogramming/blackroad-operator --public
cp -r os/agents operator/
# Add new scheduler code
# Push to repo
```
**Effort**: 3-4 weeks (includes building scheduler, Prism integration)
---
#### 3. Admin UI → `blackroad-prism-console`
**Target Repo**: `blackboxprogramming/blackroad-prism-console`
**What's New**: React or Vanilla JS dashboard for:
- Prism job queue visualization
- Agent execution monitoring
- System metrics (Prometheus)
- Audit logs (Vault)
**Technology Options**:
- **Option A**: React + TypeScript (modern, scalable)
- **Option B**: Vanilla JS (consistent with OS aesthetic)
**Recommendation**: React (separate SPA, professional admin tool)
**Effort**: 4-6 weeks (full dashboard build)
---
### What Stays in Monolith (Always)
**Core OS Responsibilities**:
- OS shell frontend (`backend/static/`)
- User authentication & identity (PS-SHA∞)
- RoadChain audit ledger
- Wallet & blockchain primitives
- Core configuration & settings
**Why These Stay**:
- Tightly coupled to OS lifecycle
- Shared by all other services
- Single source of truth for identity
- Performance-critical (low latency)
---
## PART 6: PHASE LABEL & MILESTONES
### Phase 1: Prove the OS (Months 0-12)
**Quarter-by-Quarter Breakdown**:
#### Q1 (Months 0-3): Foundation ✅ 70% Complete
**Completed**:
- [x] FastAPI backend with 33 routers
- [x] Windows 95 UI shell
- [x] 200+ agent library (base framework)
- [x] PostgreSQL + Redis stack
- [x] Docker + Railway deployment
- [x] CI/CD pipelines (7 workflows)
- [x] Complete documentation (MASTER_ORCHESTRATION_PLAN, CLAUDE.md, etc.)
**Remaining** (Weeks 1-4):
- [ ] Cloudflare DNS migration (1 day)
- [ ] Deprecate `blackroad-os/` legacy frontend (1 day)
- [ ] Add CodeQL security scanning (30 min)
- [ ] Add Dependabot (15 min)
- [ ] Increase test coverage to 80% (1 week)
- [ ] Implement Vault compliance logging (2 weeks)
- [ ] Expose agent orchestration API (2 weeks)
- [ ] Real-time WebSocket for Prism jobs (1 week)
**Success Metrics**:
- ✅ All infrastructure solid (Cloudflare, Railway, GitHub)
- ✅ No blocking bugs in OS
- ✅ Test coverage ≥ 80%
- ✅ Security scan passing
- ✅ Ready for private alpha launch
---
#### Q2 (Months 3-6): Design Partners
**Goals**:
- [ ] Onboard first 3 design partners
- [ ] Incorporate alpha feedback
- [ ] Expand documentation (tutorials, examples)
- [ ] Build demo environment
- [ ] Create sales materials
**Technical Deliverables**:
- [ ] Multi-domain routing (blackroadai.com, blackroad.me)
- [ ] RoadChain network (3 nodes on DigitalOcean)
- [ ] Lucidia integration (basic multi-model support)
- [ ] Prism job queue UI (in monolith or new repo)
- [ ] Python SDK published to PyPI
- [ ] TypeScript SDK published to npm
**Success Metrics**:
- ✅ 3 design partners using OS in production
- ✅ Weekly usage: 10+ hours per partner
- ✅ Net Promoter Score (NPS): +40 or higher
- ✅ 5-10 feature requests collected
---
#### Q3 (Months 6-9): Public Beta
**Goals**:
- [ ] Open blackroad.network to public (with waitlist)
- [ ] Launch developer community (Discord)
- [ ] Begin content marketing (blog, tutorials)
- [ ] First community office hours
- [ ] Optimize developer onboarding
**Technical Deliverables**:
- [ ] Rate limiting middleware (Redis-based)
- [ ] API versioning (v1 stable, v2 alpha)
- [ ] Advanced analytics (Mixpanel or PostHog)
- [ ] Error monitoring (Sentry integration complete)
- [ ] Performance optimization (reduce API latency by 30%)
**Success Metrics**:
- ✅ 100 developers signed up (waitlist)
- ✅ 20 active weekly users
- ✅ 50+ GitHub stars
- ✅ 3 community contributions (PRs merged)
---
#### Q4 (Months 9-12): Prove & Convert
**Goals**:
- [ ] Design partners moving to production
- [ ] Collect ROI data and success metrics
- [ ] Convert 2-3 pilots to paying customers
- [ ] First enterprise contracts signed ($50-200K each)
- [ ] Launch Team tier pricing (self-serve)
**Technical Deliverables**:
- [ ] Billing integration (Stripe complete)
- [ ] Usage-based metering
- [ ] Enterprise SSO (SAML, OAuth)
- [ ] Advanced compliance (SOX-like audit trail)
- [ ] SLA guarantees (99.9% uptime)
**Success Metrics**:
- ✅ $500K ARR from pilots
- ✅ 5 design partners in production
- ✅ 100 active developers
- ✅ 3 case studies published
- ✅ 2 technical whitepapers published
- ✅ Ready for Series Seed fundraise
---
### Phase 2: Expand Intelligence (Months 12-18)
**What This Repo Does**:
- Becomes "OS Core" (UI shell, identity, RoadChain)
- Delegates to `blackroad-api`, `blackroad-operator`, `blackroad-prism-console`
- Maintains SDKs and developer tools
**Key Milestones**:
- [ ] Complete repo split (API, operator, console)
- [ ] Lucidia multi-model orchestration live
- [ ] Prism job queue with 1000+ jobs/day
- [ ] Quantum Lab research published
**Success Metrics**:
- ✅ $3M ARR
- ✅ 20 enterprise customers
- ✅ 500 active developers
- ✅ ALICE QI recognized contribution
---
### Phase 3: Ecosystem & Orbit (Months 18-24+)
**What This Repo Does**:
- Pure OS shell (minimal, performant)
- Gateway to ecosystem (links to all services)
- Identity & wallet primitives only
**Ecosystem Repos** (all new):
- `blackroad-vault` → Compliance ledger
- `blackroad-cloudway` → Infrastructure automation
- `blackroad-metacity` → Gaming platform
- `lucidia-studio` → Creative production tools
**Success Metrics**:
- ✅ $10M ARR
- ✅ 50+ enterprise customers
- ✅ 5,000 active developers
- ✅ Multiple revenue streams operational
---
## PART 7: IMMEDIATE NEXT ACTIONS
### Week 1 Checklist (Do First)
- [ ] **1. Migrate DNS to Cloudflare** (see NEXT_ACTIONS_ALEXA.md, Item #1)
- Add `blackroad.systems` to Cloudflare
- Update nameservers at GoDaddy
- Configure SSL "Full (strict)"
- Test: `curl https://os.blackroad.systems/health`
- [ ] **2. Add GitHub Secrets**
- `RAILWAY_TOKEN` for auto-deployment
- `CF_API_TOKEN` for DNS sync
- `CF_ZONE_ID` for Cloudflare
- [ ] **3. Deprecate `blackroad-os/` Directory**
- Add deprecation README
- Update docs to reference `backend/static/` only
- Remove from CI/CD validation
- [ ] **4. Add Security Scanning**
- Create `.github/workflows/codeql.yml`
- Verify scan passes
- [ ] **5. Add Dependabot**
- Create `.github/dependabot.yml`
- Review first batch of PRs
---
### Week 2-4 Priorities
- [ ] **6. Implement Vault Compliance Logging**
- Wire `compliance_event` model into all routers
- Auto-log user actions (create, update, delete)
- Create `/api/compliance/report` endpoint
- Test with sample audit trail
- [ ] **7. Expose Agent Orchestration API**
- `GET /api/agents/library` → List 200+ agents
- `POST /api/agents/{agent_id}/execute` → Run agent
- `GET /api/jobs` → Prism job queue (mock data first)
- `WebSocket /api/jobs/{job_id}/stream` → Real-time updates
- [ ] **8. Increase Test Coverage to 80%**
- Add tests for all routers (currently 6 test files)
- Add agent execution tests
- Add integration tests for Stripe, Slack, Discord
- [ ] **9. Fix Known Issues**
- Review GitHub issues: `gh issue list`
- Fix blocking bugs
- Address UX feedback
- [ ] **10. Polish OS UI**
- Test all embedded apps
- Fix window dragging/resizing edge cases
- Improve mobile experience (basic responsiveness)
---
### Month 2-3 Goals
- [ ] **11. Multi-Domain Routing**
- Add host-based middleware to FastAPI
- Route `blackroadai.com` → Prism theme
- Route `blackroad.me` → Identity portal theme
- [ ] **12. RoadChain Network**
- Spin up 3 DigitalOcean droplets
- Implement P2P networking
- Consensus mechanism (PoW → PoS)
- [ ] **13. Lucidia Integration (Alpha)**
- Create `backend/app/routers/lucidia.py`
- Multi-model routing (Claude, GPT-4, Llama)
- Simple prompt orchestration
- [ ] **14. Performance Optimization**
- Add Redis caching for frequently accessed endpoints
- Reduce API latency by 30%
- Optimize database queries (add indexes)
- [ ] **15. Documentation Expansion**
- API reference (auto-generated from FastAPI)
- Quick start guide (5-minute setup)
- Video walkthrough (5-minute demo)
---
## PART 8: CRITICAL RISKS & MITIGATION
| Risk | Impact | Likelihood | Mitigation | Owner |
|------|--------|------------|-----------|-------|
| **Frontend duplication causes drift** | Deploy wrong UI | High | Deprecate `blackroad-os/` immediately | Alexa |
| **Missing env vars break production** | Outage | Medium | Run `validate_env_template.py` on every PR | CI/CD |
| **SQLite default in production** | Data loss | Medium | Change config to fail-fast on missing DATABASE_URL | Alexa |
| **No rate limiting** | DDoS vulnerability | Medium | Implement Redis-based rate limiter (Week 3) | Backend team |
| **Vault compliance untouched** | No audit trail | High | Auto-log all user actions (Week 2) | Backend team |
| **Single RoadChain node** | Centralized blockchain | Medium | Deploy multi-node network (Month 3) | DevOps |
| **Agent integrations untested** | Broken features | High | Add agent execution tests (Week 3) | AI/ML team |
| **WebSocket not fully utilized** | Delayed updates | Low | Expand WebSocket for Prism jobs (Week 4) | Backend team |
---
## PART 9: SUCCESS CRITERIA
### Phase 1 Success (Months 0-12)
**Product**:
- ✅ Stable v1.0 release (no blocking bugs)
- ✅ Test coverage ≥ 80%
- ✅ Security scan passing
- ✅ 99.5% uptime (Railway health checks)
**Business**:
- ✅ 5 enterprise design partners in production
- ✅ 100 active developers building on platform
- ✅ $500K ARR from pilots
- ✅ 3 case studies published
**Technical**:
- ✅ Cloudflare DNS live
- ✅ Railway deployment automated
- ✅ Vault compliance logging operational
- ✅ Agent orchestration API exposed
- ✅ RoadChain multi-node network (3 nodes)
**Community**:
- ✅ Developer community launched (Discord)
- ✅ 50+ GitHub stars
- ✅ 5 community contributions merged
- ✅ Monthly office hours established
---
## PART 10: QUICK REFERENCE
### Key Files
| File | Purpose | Location |
|------|---------|----------|
| **Main App** | FastAPI application | `backend/app/main.py:8` |
| **Config** | Pydantic settings | `backend/app/config.py:1` |
| **Database** | Session management | `backend/app/database.py:1` |
| **Frontend** | OS interface (canonical) | `backend/static/index.html` |
| **Agents** | Base agent class | `agents/base/agent.py:1` |
| **Tests** | Test suite | `backend/tests/` |
| **Dockerfile** | Container definition | `backend/Dockerfile` |
| **Railway Config** | Deployment settings | `railway.toml` |
### Essential Commands
```bash
# Local development
cd backend
docker-compose up
# Run tests
bash scripts/run_backend_tests.sh
# Deploy to Railway
railway up
# Check status
railway status --service backend
railway logs --service backend --tail 50
# Health check
curl https://os.blackroad.systems/health
# Create migration
cd backend
alembic revision --autogenerate -m "message"
alembic upgrade head
# Validate environment
python scripts/railway/validate_env_template.py
```
### Documentation Hierarchy
```
MASTER_ORCHESTRATION_PLAN.md → Complete 7-layer blueprint
ORG_STRUCTURE.md → Repo architecture (all 23 repos)
IMPLEMENTATION.md (this file) → This repo's detailed plan
NEXT_ACTIONS_ALEXA.md → Executable checklist
CLAUDE.md → AI assistant guide (900 lines)
BLACKROAD_OS_BIG_KAHUNA_VISION.md → 18-24 month roadmap
```
---
## CONCLUSION
**BlackRoad-Operating-System** is the canonical monolith and foundation for the entire BlackRoad ecosystem. It's 65% complete toward the Big Kahuna vision and ready for Phase 1 enterprise pilots.
**Immediate Priority**: Infrastructure solidification (DNS, secrets, security) in Week 1
**Short-Term Goal**: v1.0 stable release with 5 design partners by Month 6
**Long-Term Vision**: Evolve into OS Core as other services split out in Phase 2
**Next Action**: Start with Week 1 Checklist, Item #1 (Cloudflare DNS migration)
**Ready for the next command, Operator.** 🛣️
---
**Last Updated**: 2025-11-18
**Next Review**: 2025-12-01 (monthly check-in)

579
ORG_STRUCTURE.md Normal file
View File

@@ -0,0 +1,579 @@
# 🏢 BLACKROAD GITHUB ORGANIZATION STRUCTURE
## Repository Architecture & Responsibility Map
**Version:** 1.0
**Date:** 2025-11-18
**Author:** Claude (Sonnet 4.5) - BlackRoad Implementation Planning
**Organizations:** `blackboxprogramming` & `BlackRoad-AI`
---
## EXECUTIVE SUMMARY
This document maps **23 BlackRoad repositories** across 2 GitHub organizations to the 7-layer BlackRoad OS architecture defined in `MASTER_ORCHESTRATION_PLAN.md`.
**Key Findings**:
- **1 monolith** currently holds the canonical OS (BlackRoad-Operating-System)
- **6 satellite repos** for specific layers (API, operator, console, frontend sites)
- **8 experimental repos** for advanced features (Lucidia, quantum, AI labs)
- **8 utility/legacy repos** (starters, templates, personal projects)
**Recommended Strategy**: **Consolidate Phase 1** around 4 core repos, archive or migrate the rest.
---
## PART 1: REPOSITORY INVENTORY
### Core Operational Repos (Active in Phase 1)
| Repo | Owner | Purpose | Status | Lines of Code | Last Updated |
|------|-------|---------|--------|---------------|--------------|
| **BlackRoad-Operating-System** | blackboxprogramming | Monolith: OS core, backend API, agents, SDKs, docs | **CANONICAL** | ~50K+ | Active |
| **blackroad-api** | blackboxprogramming | Standalone API gateway (future split from monolith) | Planned | TBD | Stub |
| **blackroad-operator** | blackboxprogramming | Workflow orchestration & scheduled agents | Planned | TBD | Stub |
| **blackroad-prism-console** | blackboxprogramming | Admin UI for Prism job queue & observability | Planned | TBD | Stub |
| **blackroad.io** | blackboxprogramming | Corporate marketing site (blackroad.systems) | Planned | TBD | Stub |
| **BlackRoad.io** | BlackRoad-AI | Alternate corporate site (may be duplicate) | Unknown | TBD | Unknown |
### AI & Intelligence Repos
| Repo | Owner | Purpose | Status | Integration |
|------|-------|---------|--------|-------------|
| **lucidia** | blackboxprogramming | Multi-model AI orchestration layer | Development | Phase 2 |
| **lucidia-lab** | blackboxprogramming | Experimental AI research & testing | Development | Phase 2 |
| **native-ai-quantum-energy** | blackboxprogramming | Quantum computing + AI research | Research | Phase 3 |
| **quantum-math-lab** | blackboxprogramming | Mathematical foundations for quantum OS | Research | Phase 3 |
| **codex-agent-runner** | blackboxprogramming | Agent execution runtime (may merge with operator) | Development | Phase 2 |
| **codex-infinity** | blackboxprogramming | Advanced coding agent system | Development | Phase 2 |
### Utility & Experimental Repos
| Repo | Owner | Purpose | Status | Recommendation |
|------|-------|---------|--------|----------------|
| **blackroad** | blackboxprogramming | Possible legacy or alternative frontend | Unknown | Investigate → Archive or Merge |
| **BlackStream** | blackboxprogramming | Video streaming platform (RoadStream) | Development | Merge into monolith |
| **universal-computer** | blackboxprogramming | Theoretical computing model | Research | Archive (reference only) |
| **remember** | blackboxprogramming | Memory/logging framework | Development | May integrate with Vault |
### Planning & Documentation Repos
| Repo | Owner | Purpose | Status | Recommendation |
|------|-------|---------|--------|----------------|
| **blackroad-plans** | BlackRoad-AI | Strategic planning documents | Active | Keep separate |
### Starter Templates & Examples
| Repo | Owner | Purpose | Status | Recommendation |
|------|-------|---------|--------|----------------|
| **next-video-starter** | blackboxprogramming | Next.js video app template | Template | Archive or move to examples/ |
| **nextjs-ai-chatbot** | blackboxprogramming | Next.js + AI chatbot starter | Template | Archive or integrate features |
### Personal/Meta Repos
| Repo | Owner | Purpose | Status | Recommendation |
|------|-------|---------|--------|----------------|
| **my-repository** | blackboxprogramming | Personal experiments | Unknown | Archive |
| **blackboxprogramming** | blackboxprogramming | Profile README or meta repo | Meta | Keep |
| **new_world** | blackboxprogramming | Possible game or narrative project | Unknown | Investigate → Archive or merge with MetaCity |
---
## PART 2: 7-LAYER ARCHITECTURE MAPPING
### Layer 1: DNS & CDN
**Responsibility**: Domain management, SSL, DDoS protection
| Repo | Role | Implementation |
|------|------|----------------|
| **BlackRoad-Operating-System** | Cloudflare DNS config scripts | `scripts/cloudflare/sync_dns.py` (planned) |
| **blackroad.io** | DNS CNAME targets | `CNAME @ → blackroad-os-production.up.railway.app` |
**Domains Served**:
- `blackroad.systems` → Corporate site
- `os.blackroad.systems` → OS interface
- `api.blackroad.systems` → API gateway
- `prism.blackroad.systems` → Prism Console
- `lucidia.earth` → Narrative experiences (Phase 2)
**Config Files**:
- `ops/domains.yaml` (in monolith)
- Cloudflare dashboard (manual for Phase 1)
---
### Layer 2: Compute & Infrastructure
**Responsibility**: Hosting, containers, orchestration
| Repo | Role | Deployment Target | Config |
|------|------|-------------------|--------|
| **BlackRoad-Operating-System** | Backend API + static UI | Railway | `railway.toml`, `Dockerfile` |
| **blackroad-operator** | Worker processes | Railway (future) | Docker |
| **blackroad-prism-console** | Admin UI | Railway or Vercel | Docker or static |
| **blackroad.io** | Marketing site | Vercel or GitHub Pages | Static build |
**Infrastructure Providers**:
- **Railway**: Backend API, PostgreSQL, Redis (current)
- **DigitalOcean**: RoadChain nodes (Phase 2)
- **Cloudflare Workers**: Edge functions (Phase 2)
- **GitHub Pages**: Documentation sites
---
### Layer 3: Data & State
**Responsibility**: Databases, caching, blockchain, compliance logs
| Repo | Component | Storage | Technology |
|------|-----------|---------|------------|
| **BlackRoad-Operating-System** | User data, app state | PostgreSQL (Railway) | SQLAlchemy 2.0 + asyncpg |
| **BlackRoad-Operating-System** | Session cache | Redis (Railway) | redis-py 5.0 |
| **BlackRoad-Operating-System** | RoadChain ledger | In-memory (Phase 1) → PostgreSQL | Custom blockchain |
| **BlackRoad-Operating-System** | Vault compliance | `compliance_event` table | Immutable audit log |
**Database Schema Owner**: **BlackRoad-Operating-System/backend/app/models/**
**Migration Strategy**:
- Phase 1: Monolith owns all schemas
- Phase 2: Split into service-specific DBs (Prism DB, Lucidia DB, etc.)
---
### Layer 4: Orchestration & Intelligence
**Responsibility**: Agents, jobs, AI orchestration, workflows
| Repo | Component | Purpose | Status |
|------|-----------|---------|--------|
| **BlackRoad-Operating-System** | 200+ agents | Agent library | ✅ Complete (base framework) |
| **lucidia** | Multi-model AI orchestration | Route requests to Claude, GPT, Llama | 🔨 Development |
| **lucidia-lab** | AI experiments | Test new models, prompts | 🔬 Research |
| **blackroad-operator** | Workflow scheduler | Cron-like agent execution | 📋 Planned |
| **blackroad-prism-console** | Job queue UI | Visualize Prism jobs | 📋 Planned |
| **codex-agent-runner** | Agent execution runtime | Run agents in isolated envs | 🔨 Development |
| **codex-infinity** | Advanced coding agents | Generate entire codebases | 🔬 Research |
**API Endpoints** (in BlackRoad-Operating-System):
- `GET /api/agents/library` → List 200+ agents
- `POST /api/agents/{agent_id}/execute` → Run agent
- `GET /api/jobs` → Prism job queue
- `WebSocket /api/jobs/{job_id}/stream` → Real-time updates
---
### Layer 5: API Gateway & Routing
**Responsibility**: REST API, WebSocket, authentication, routing
| Repo | Role | Technology | Endpoints |
|------|------|------------|-----------|
| **BlackRoad-Operating-System** | Canonical API gateway | FastAPI 0.104.1 | 33 routers, 100+ endpoints |
| **blackroad-api** | Future: Standalone API | FastAPI (split from monolith) | Phase 2 migration |
**Current Routers** (in monolith):
```
/api/auth/* → Authentication (JWT)
/api/email/* → RoadMail
/api/social/* → BlackRoad Social
/api/video/* → BlackStream
/api/blockchain/* → RoadChain
/api/miner/* → Mining
/api/agents/* → Agent orchestration (planned)
/api/prism/* → Job queue (planned)
... 25 more routers
```
**Host-Based Routing** (Phase 2):
- `blackroadai.com/*` → Prism theme
- `lucidia.earth/*` → Narrative theme
- `api.blackroad.systems/*` → Developer API
---
### Layer 6: Application Layer (Pocket OS)
**Responsibility**: Windows 95 UI, native apps, real-time sync
| Repo | Component | Technology | Deployment |
|------|-----------|------------|------------|
| **BlackRoad-Operating-System** | Canonical OS UI | Vanilla JS, HTML, CSS | `backend/static/index.html` |
| **blackroad-prism-console** | Admin console | React or Vanilla JS | Separate app |
| **BlackStream** | Video streaming app | Next.js (potential) | May merge into monolith |
| **blackroad** | Possible alternate UI | Unknown | **Investigate → Archive or merge** |
**Native Apps** (embedded in OS):
- RoadMail, BlackRoad Social, BlackStream, RoadChain Explorer
- RoadCoin Miner, Wallet, RoadView Browser, Terminal, File Explorer
- GitHub Manager, Raspberry Pi Manager
- RoadCity, RoadCraft, Road Life (games)
**Duplication Issue**:
- `backend/static/` → **CANONICAL**
- `blackroad-os/` → **LEGACY** (superseded, mark as deprecated)
---
### Layer 7: User Experience (Multi-Domain)
**Responsibility**: Marketing sites, developer docs, narrative experiences
| Repo | Domain | Purpose | Technology | Status |
|------|--------|---------|------------|--------|
| **blackroad.io** | blackroad.systems | Corporate site | HTML/CSS or Next.js | 📋 Planned |
| **BlackRoad.io** | TBD (may be duplicate) | Unknown | Unknown | ❓ Investigate |
| **BlackRoad-Operating-System** | os.blackroad.systems | OS interface | FastAPI + static | ✅ Live |
| **blackroad-plans** | Internal only | Planning docs | Markdown | ✅ Active |
| **BlackRoad-Operating-System** | blackroad.network | Developer docs | MkDocs or Docusaurus | 📋 Planned |
| **lucidia** | lucidia.earth | Narrative site | Next.js (Phase 2) | 📋 Planned |
**Domain Ownership** (from MASTER_ORCHESTRATION_PLAN.md):
```
blackroad.systems → blackroad.io repo (corporate)
os.blackroad.systems → BlackRoad-Operating-System (OS)
api.blackroad.systems → blackroad-api (future) or monolith (Phase 1)
prism.blackroad.systems → blackroad-prism-console
blackroad.network → BlackRoad-Operating-System/docs/ (developer)
lucidia.earth → lucidia repo (narrative)
aliceqi.com → quantum-math-lab or lucidia-lab (research)
```
---
## PART 3: RECOMMENDED REPOSITORY STRATEGY
### Phase 1 (Months 0-12): Consolidate Around Monolith
**Active Repos** (4 core):
1. **BlackRoad-Operating-System** → Canonical monolith (OS, API, agents, docs)
2. **blackroad.io** → Corporate marketing site
3. **blackroad-plans** → Strategic planning (separate org)
4. **lucidia** → AI layer (development starts mid-Phase 1)
**Archive** (move to `archive/` or mark read-only):
- `blackroad-os/` → Superseded by `backend/static/`
- `next-video-starter` → Template, not core product
- `nextjs-ai-chatbot` → Template, integrate features into monolith if needed
- `my-repository` → Personal, not product
- `universal-computer` → Research reference only
**Investigate** (determine keep/merge/archive):
- `blackroad` → If alternate UI, merge or archive
- `BlackRoad.io` → If duplicate of blackroad.io, archive
- `new_world` → Merge with MetaCity if game-related, else archive
**Keep as Experimental**:
- `lucidia-lab` → Research & testing
- `quantum-math-lab` → Long-term research
- `native-ai-quantum-energy` → Long-term research
- `remember` → May integrate into Vault
---
### Phase 2 (Months 12-18): Strategic Split
**New Active Repos**:
1. **blackroad-api** → Extract API gateway from monolith
2. **blackroad-operator** → Workflow orchestration & scheduler
3. **blackroad-prism-console** → Admin UI for observability
4. **BlackStream** → Standalone video platform
**Monolith Becomes**:
- **BlackRoad-Operating-System** → Core runtime, identity (PS-SHA∞), RoadChain
**Migration Strategy**:
```
Monolith (Phase 1)
├── backend/app/routers/* → blackroad-api/routers/
├── agents/ → blackroad-operator/agents/
├── backend/static/ → Keep in monolith (OS shell)
└── docs/ → blackroad-network (GitHub Pages)
```
---
### Phase 3 (Months 18-24+): Microservices & Ecosystem
**Additional Repos**:
- **blackroad-vault** → Compliance & audit ledger
- **blackroad-cloudway** → Infrastructure as software
- **blackroad-roadchain** → Blockchain network
- **blackroad-metacity** → Gaming platform
**Full Ecosystem** (10 repos):
```
┌─────────────────────────────────────────────────────────────┐
│ USER-FACING │
├─────────────────────────────────────────────────────────────┤
│ blackroad.io → Corporate site │
│ BlackRoad-Operating-System → OS shell (minimal) │
│ blackroad-prism-console → Admin UI │
└─────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────┐
│ BACKEND SERVICES │
├─────────────────────────────────────────────────────────────┤
│ blackroad-api → REST/GraphQL gateway │
│ blackroad-operator → Workflows & scheduler │
│ lucidia → AI orchestration │
│ blackroad-vault → Compliance ledger │
│ blackroad-cloudway → Infrastructure automation │
│ blackroad-roadchain → Blockchain nodes │
│ blackroad-metacity → Gaming backend │
└─────────────────────────────────────────────────────────────┘
```
---
## PART 4: GITHUB ORG RECOMMENDATIONS
### Current State
- **blackboxprogramming** org → 20 repos (mix of active, experimental, personal)
- **BlackRoad-AI** org → 2 repos (blackroad-plans, BlackRoad.io)
### Recommended Structure
**Option A: Consolidate into single `blackroad` org**
- Create new `blackroad` GitHub organization
- Migrate all product repos from `blackboxprogramming` and `BlackRoad-AI`
- Keep `blackboxprogramming` for personal/experimental work
- **Pros**: Clear branding, professional, easier permissions
- **Cons**: Migration effort, URL changes
**Option B: Keep `blackboxprogramming`, clean it up**
- Archive/delete non-product repos
- Rename to `blackroad-os` or `blackroad-ai` (GitHub allows org rename)
- **Pros**: No migration, URLs stay same
- **Cons**: Less professional name
**Option C: Hybrid (Recommended for Phase 1)**
- Keep `blackboxprogramming` as-is for Phase 1 (minimize disruption)
- Create `blackroad` org for Phase 2 (migrate core repos)
- Use topics/tags to distinguish product vs. experimental repos
- **Pros**: Gradual transition, low risk
- **Cons**: Temporary complexity
### GitHub Teams (Future)
Once org structure is solidified:
```
@blackroad/core → Alexa + senior engineers (all repos)
@blackroad/frontend → UI/UX contributors (OS, Prism Console)
@blackroad/backend → API/infrastructure (API, Operator, CloudWay)
@blackroad/ai → AI/ML contributors (Lucidia, agents)
@blackroad/docs → Documentation writers (docs, README)
@blackroad/community → External contributors (all repos, triage role)
```
### CODEOWNERS Consolidation
**Monolith** (`BlackRoad-Operating-System/.github/CODEOWNERS`):
```
# Global owners
* @alexa-amundson @cadillac
# Backend
/backend/app/** @blackroad/backend @alexa-amundson
# Frontend
/backend/static/** @blackroad/frontend @alexa-amundson
# Agents
/agents/** @blackroad/ai @alexa-amundson
# Infrastructure
/.github/workflows/** @blackroad/backend
/scripts/** @blackroad/backend
/ops/** @alexa-amundson
# Docs
/docs/** @blackroad/docs
/README.md @blackroad/docs
```
**Future Repos**: Each repo gets its own CODEOWNERS with `@blackroad/core` as global owner
---
## PART 5: CANONICAL WORKFLOWS
### Required GitHub Actions (All Active Repos)
| Workflow | File | Purpose | Runs On |
|----------|------|---------|---------|
| **CI** | `.github/workflows/ci.yml` | Lint, type-check, test | Push, PR → main |
| **Deploy** | `.github/workflows/deploy.yml` | Deploy to Railway/Vercel | Push → main (after CI passes) |
| **Security** | `.github/workflows/codeql.yml` | CodeQL security scan | Weekly, PR → main |
| **Dependencies** | `.github/dependabot.yml` | Auto-update deps | Daily |
### Repo-Specific Workflows
**BlackRoad-Operating-System** (monolith):
- `backend-tests.yml` → Run pytest
- `railway-deploy.yml` → Deploy to Railway
- `railway-automation.yml` → Env validation
- `domain-health.yml` → Multi-domain health checks
**blackroad.io** (marketing site):
- `build.yml` → Build Next.js/static site
- `deploy-vercel.yml` → Deploy to Vercel
**blackroad-api** (future):
- `api-tests.yml` → API contract tests
- `railway-deploy.yml` → Deploy API service
**lucidia** (AI layer):
- `model-tests.yml` → Test AI model integrations
- `deploy.yml` → Deploy to Railway
---
## PART 6: MIGRATION CHECKLIST
### Immediate Actions (This Week)
- [ ] **Deprecate `blackroad-os/` directory in monolith**
- Add README: "⚠️ LEGACY: This directory is superseded by `backend/static/`. Do not edit."
- Update all docs to reference `backend/static/` only
- Archive or delete after verification
- [ ] **Investigate duplicate repos**
- [ ] Check if `blackroad` repo has different code than `backend/static/`
- [ ] Check if `BlackRoad.io` is duplicate of `blackroad.io` plan
- [ ] Document findings, recommend archive or merge
- [ ] **Tag experimental repos**
- Add GitHub topic tags: `experimental`, `research`, `phase-2`, `phase-3`
- Update repo descriptions to clarify status
### Phase 1 Setup (Weeks 1-4)
- [ ] **Create `blackroad.io` repo**
- Use `next-video-starter` or simple HTML as template
- Implement 5-page MVP (from NEXT_ACTIONS_ALEXA.md)
- Deploy to Vercel or GitHub Pages
- Point `blackroad.systems` DNS
- [ ] **Set up `blackroad-plans` integration**
- Link from monolith README
- Reference in MASTER_ORCHESTRATION_PLAN.md
- Keep as single source of truth for strategy docs
- [ ] **Create ORG_STRUCTURE.md in all active repos**
- Link to this canonical version in monolith
- Each repo gets a "Role in Ecosystem" section
### Phase 2 Preparation (Months 6-12)
- [ ] **Scaffold future repos**
- Create `blackroad-api` stub (copy backend/app/, remove static/)
- Create `blackroad-operator` stub (copy agents/, add scheduler)
- Create `blackroad-prism-console` stub (React app)
- Mark as "UNDER DEVELOPMENT - Phase 2"
- [ ] **Plan migration scripts**
- `scripts/migrate_api.sh` → Extract backend/app/routers/ to blackroad-api
- `scripts/migrate_agents.sh` → Extract agents/ to blackroad-operator
- Document in MIGRATION_GUIDE.md
### Phase 3 (Months 12+)
- [ ] **Create new org (optional)**
- If going with Option A, create `blackroad` org
- Transfer repos from `blackboxprogramming` and `BlackRoad-AI`
- Update all docs, CI/CD with new URLs
- [ ] **Microservices repos**
- `blackroad-vault`, `blackroad-cloudway`, `blackroad-roadchain`, `blackroad-metacity`
- Each with own CI/CD, deployment, docs
---
## PART 7: REPO HEALTH METRICS
### Recommended Tracking (Per Repo)
| Metric | Tool | Frequency |
|--------|------|-----------|
| **Test Coverage** | pytest-cov, CodeCov | Every CI run |
| **Build Status** | GitHub Actions badges | Real-time |
| **Dependencies** | Dependabot alerts | Daily |
| **Security** | CodeQL, Snyk | Weekly |
| **Code Quality** | SonarCloud, CodeClimate | Every PR |
| **Deploy Status** | Railway/Vercel status checks | Every deploy |
### Monolith Health Dashboard
Add to `README.md`:
```markdown
## 📊 Repository Health
| Component | Status | Coverage | Last Deploy |
|-----------|--------|----------|-------------|
| Backend API | ![CI](https://github.com/.../workflows/ci.yml/badge.svg) | 75% | 2025-11-18 |
| Frontend | ![CI](https://github.com/.../workflows/ci.yml/badge.svg) | N/A | 2025-11-18 |
| Agents | ![Tests](https://github.com/.../workflows/backend-tests.yml/badge.svg) | 60% | 2025-11-15 |
| Deployment | ![Railway](https://img.shields.io/badge/railway-deployed-success) | - | 2025-11-18 |
```
---
## PART 8: QUICK REFERENCE
### Repo Ownership Matrix
| Repo | Primary Owner | Secondary Owner | Purpose | Status |
|------|---------------|-----------------|---------|--------|
| BlackRoad-Operating-System | @alexa-amundson | @blackroad/core | Monolith (OS, API, agents) | Active |
| blackroad.io | @alexa-amundson | @blackroad/frontend | Corporate site | Planned |
| blackroad-plans | @alexa-amundson | - | Strategic planning | Active |
| lucidia | @alexa-amundson | @blackroad/ai | AI orchestration | Development |
| blackroad-api | @alexa-amundson | @blackroad/backend | API gateway | Phase 2 |
| blackroad-operator | @alexa-amundson | @blackroad/backend | Workflow orchestration | Phase 2 |
| blackroad-prism-console | @alexa-amundson | @blackroad/frontend | Admin UI | Phase 2 |
### Documentation Hierarchy
```
┌─────────────────────────────────────────────────────────────┐
│ MASTER DOCS (in BlackRoad-Operating-System) │
├─────────────────────────────────────────────────────────────┤
│ MASTER_ORCHESTRATION_PLAN.md → Complete 7-layer blueprint │
│ ORG_STRUCTURE.md (this file) → Repo architecture │
│ NEXT_ACTIONS_ALEXA.md → Execution checklist │
│ BLACKROAD_OS_BIG_KAHUNA_VISION.md → 18-24 month roadmap │
│ CLAUDE.md → AI assistant guide │
└─────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────┐
│ REPO-SPECIFIC DOCS (in each repo) │
├─────────────────────────────────────────────────────────────┤
│ IMPLEMENTATION.md → This repo's plan │
│ README.md → Overview, setup │
│ CONTRIBUTING.md → How to contribute │
└─────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────┐
│ STRATEGIC DOCS (in blackroad-plans) │
├─────────────────────────────────────────────────────────────┤
│ [Plans, roadmaps, strategy documents] │
└─────────────────────────────────────────────────────────────┘
```
---
## CONCLUSION
**Current State**: 23 repos across 2 orgs with unclear boundaries and some duplication
**Target State** (Phase 1): 4 core active repos with clear roles, 15 archived/experimental
**Execution Strategy**:
1. **Week 1**: Deprecate duplicates, tag experimental repos
2. **Weeks 2-4**: Create blackroad.io, set up docs
3. **Months 3-6**: Start lucidia development
4. **Months 6-12**: Plan Phase 2 splits (API, operator, console)
5. **Months 12+**: Microservices architecture
**Next Steps**: Create `IMPLEMENTATION.md` in each core repo with detailed plans.
---
*This document is the single source of truth for BlackRoad repository organization. Update as architecture evolves.*
**Last Updated**: 2025-11-18
**Next Review**: 2025-12-18 (monthly)

567
implementation-plans/IMPLEMENTATION_blackroad-api.md Normal file
View File

@@ -0,0 +1,567 @@
# 🚀 IMPLEMENTATION PLAN: blackroad-api
## Standalone API Gateway & Routing Layer
**Repo**: `blackboxprogramming/blackroad-api` (to be created/populated)
**Purpose**: Extract API gateway from monolith, enable microservices architecture
**Version**: 1.0
**Phase**: **Phase 2 (Months 12-18) - Strategic Split**
---
## EXECUTIVE SUMMARY
**blackroad-api** will be the **standalone API gateway** serving all BlackRoad OS clients. It extracts the 33 routers from the monolith (`BlackRoad-Operating-System`) and adds:
- Advanced routing (host-based, versioning)
- Rate limiting & throttling
- API key management
- GraphQL support (optional)
- Centralized authentication
**Current State**: Stub/planned repo
**Target State**: Production API gateway serving 1000+ requests/second
**Migration From**: `BlackRoad-Operating-System/backend/app/routers/`
**Role in 7-Layer Architecture**:
- **Layer 5** (API Gateway): Primary responsibility
- Sits between frontend (Layer 6) and orchestration (Layer 4)
- Routes to services: Prism, Lucidia, Operator, RoadChain
---
## PART 1: PURPOSE & FINAL ROLE
### Why Split from Monolith?
**Problems with Monolith API**:
- Scaling: Frontend UI and API have different scaling needs
- Deployment: API changes require full OS redeploy
- Performance: Static files and API compete for resources
- Organization: 33 routers in one repo is hard to navigate
- Testing: Integration tests are slow and complex
**Benefits of Standalone API**:
- Independent scaling (scale API separately from UI)
- Faster deployments (API changes don't touch frontend)
- Better caching (Cloudflare can cache API responses)
- Clear boundaries (API team vs. UI team)
- Easier testing (isolated API tests)
### Final Architecture
```
┌─────────────────────────────────────────────────────────────┐
│ CLIENTS │
│ ├── Web UI (os.blackroad.systems) │
│ ├── Mobile App (future) │
│ ├── CLI (blackroad-cli) │
│ └── SDKs (Python, TypeScript, Go, Rust) │
└─────────────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────────┐
│ CLOUDFLARE (DNS, SSL, CDN, DDoS) │
│ api.blackroad.systems │
└─────────────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────────┐
│ blackroad-api (THIS REPO) │
│ ├── Authentication & Authorization (JWT, API keys) │
│ ├── Rate Limiting & Throttling │
│ ├── Routing & Versioning (v1, v2, v3) │
│ ├── Request Validation & Transformation │
│ └── Response Caching & Compression │
└─────────────────────────────────────────────────────────────┘
↓ ↓ ↓ ↓
┌───────────────┐ ┌──────────┐ ┌──────────┐ ┌──────────────┐
│ blackroad- │ │ lucidia │ │ blackroad│ │ PostgreSQL │
│ operator │ │ (AI) │ │ -os │ │ (direct) │
│ (agents) │ │ │ │ (core) │ │ │
└───────────────┘ └──────────┘ └──────────┘ └──────────────┘
```
### API Domains
| Domain | Purpose | Target | Phase |
|--------|---------|--------|-------|
| **api.blackroad.systems** | Primary API (versioned) | This repo | Phase 2 |
| **v1.api.blackroad.systems** | Explicit v1 API | This repo | Phase 2 |
| **v2.api.blackroad.systems** | v2 API (breaking changes) | This repo | Phase 3 |
| **graphql.api.blackroad.systems** | GraphQL endpoint | This repo | Phase 3 |
---
## PART 2: REQUIRED WORKFLOWS
### GitHub Actions (5 core workflows)
#### 1. CI/CD (`.github/workflows/ci.yml`)
```yaml
name: CI
on: [push, pull_request]
jobs:
lint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/setup-python@v4
with:
python-version: '3.11'
- run: pip install flake8 black mypy
- run: black --check .
- run: flake8 .
- run: mypy app/
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/setup-python@v4
- run: pip install -r requirements.txt
- run: pytest --cov=app --cov-report=xml
- uses: codecov/codecov-action@v3
contract-test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- run: pip install -r requirements.txt
- run: pytest tests/contract/ # API contract tests
```
#### 2. Deploy to Railway (`.github/workflows/deploy.yml`)
```yaml
name: Deploy
on:
push:
branches: [main]
workflow_dispatch:
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: railway/railway-deploy@v1
with:
railway-token: ${{ secrets.RAILWAY_TOKEN }}
service: blackroad-api
- name: Health Check
run: |
sleep 30 # Wait for deploy
curl -f https://api.blackroad.systems/health || exit 1
```
#### 3. Security Scan (`.github/workflows/security.yml`)
```yaml
name: Security
on:
push:
branches: [main]
schedule:
- cron: '0 0 * * 0' # Weekly
jobs:
codeql:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: github/codeql-action/init@v2
with:
languages: python
- uses: github/codeql-action/analyze@v2
dependency-scan:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: snyk/actions/python@master
with:
args: --severity-threshold=high
env:
SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }}
```
#### 4. API Documentation (`.github/workflows/docs.yml`)
```yaml
name: API Docs
on:
push:
branches: [main]
jobs:
generate:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- run: pip install -r requirements.txt
- run: python scripts/generate_openapi.py
- uses: peaceiris/actions-gh-pages@v3
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
publish_dir: ./docs/api
```
#### 5. Performance Tests (`.github/workflows/performance.yml`)
```yaml
name: Performance
on:
schedule:
- cron: '0 2 * * *' # Daily at 2am
workflow_dispatch:
jobs:
load-test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- run: pip install locust
- run: locust -f tests/load/locustfile.py --headless -u 100 -r 10 --run-time 5m --host https://api.blackroad.systems
- uses: actions/upload-artifact@v3
with:
name: performance-report
path: reports/
```
---
## PART 3: SECRETS & ENVIRONMENT VARIABLES
### GitHub Secrets
| Secret | Purpose | How to Get |
|--------|---------|-----------|
| **RAILWAY_TOKEN** | Deploy to Railway | `railway login --browserless` |
| **CODECOV_TOKEN** | Coverage reporting | CodeCov project settings |
| **SNYK_TOKEN** | Security scanning | Snyk account settings |
| **SENTRY_DSN** | Error monitoring | Sentry project |
### Railway Environment Variables
**Core** (required):
```bash
ENVIRONMENT=production
DEBUG=False
SECRET_KEY=<openssl rand -hex 32>
DATABASE_URL=${{Postgres.DATABASE_URL}}
DATABASE_ASYNC_URL=${{Postgres.DATABASE_ASYNC_URL}}
REDIS_URL=${{Redis.REDIS_URL}}
PORT=8000
```
**API-Specific**:
```bash
# CORS
ALLOWED_ORIGINS=https://os.blackroad.systems,https://blackroadai.com,https://blackroad.me
# Rate Limiting
RATE_LIMIT_ENABLED=True
RATE_LIMIT_REQUESTS_PER_MINUTE=60 # Per API key
RATE_LIMIT_BURST=100
# API Keys
API_KEY_ENCRYPTION_KEY=<openssl rand -hex 32>
# Service URLs (for routing to other services)
OPERATOR_SERVICE_URL=https://operator.blackroad.systems
LUCIDIA_SERVICE_URL=https://lucidia.blackroad.systems
OS_CORE_SERVICE_URL=https://os.blackroad.systems
# Monitoring
SENTRY_DSN=https://...
PROMETHEUS_ENABLED=True
```
---
## PART 4: CLOUDFLARE & DOMAIN WIRING
### DNS Records
**Primary API Domain** (`blackroad.systems` zone):
| Type | Name | Target | Proxy | Notes |
|------|------|--------|-------|-------|
| CNAME | api | `blackroad-api-production.up.railway.app` | ✅ | Primary API endpoint |
| CNAME | v1.api | `blackroad-api-production.up.railway.app` | ✅ | Explicit v1 |
| CNAME | v2.api | `blackroad-api-v2.up.railway.app` | ✅ | v2 (future) |
### Cloudflare Configuration
**Cache Rules**:
- Cache static OpenAPI docs for 1 day
- Cache GET requests for 5 minutes (with cache-control header)
- Don't cache POST/PUT/DELETE
- Bypass cache for authenticated requests
**Page Rules**:
```
api.blackroad.systems/docs/*
Cache Level: Everything
Edge Cache TTL: 1 day
api.blackroad.systems/v1/*
Cache Level: Cache Everything
Edge Cache TTL: 5 minutes
Respect Existing Headers: On
```
**Transform Rules** (add version header):
```javascript
// Add to all responses
X-API-Version: v1
X-Powered-By: BlackRoad OS
```
---
## PART 5: MIGRATION NOTES
### Migration from Monolith
**Step 1: Create Repo Structure**
```bash
mkdir blackroad-api
cd blackroad-api
# Initialize repo
git init
gh repo create blackboxprogramming/blackroad-api --public
# Copy structure from monolith
cp -r ../BlackRoad-Operating-System/backend/app ./
cp -r ../BlackRoad-Operating-System/backend/requirements.txt ./
cp ../BlackRoad-Operating-System/backend/Dockerfile ./
# Remove frontend (stays in monolith)
rm -rf app/static/
# Create new main.py (API-only)
cat > app/main.py << 'EOF'
from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware
from app.routers import * # Import all 33 routers
from app.config import settings
app = FastAPI(
title="BlackRoad OS API",
version="1.0.0",
docs_url="/docs",
redoc_url="/redoc"
)
# Middleware
app.add_middleware(
CORSMiddleware,
allow_origins=settings.ALLOWED_ORIGINS,
allow_credentials=True,
allow_methods=["*"],
allow_headers=["*"],
)
# Include all routers
app.include_router(auth.router)
app.include_router(email.router)
# ... all 33 routers
@app.get("/health")
async def health():
return {"status": "healthy", "version": "1.0.0"}
EOF
```
**Step 2: Update Imports**
```bash
# Find and replace import paths
find app/ -type f -name "*.py" -exec sed -i 's/from app\./from /g' {} \;
```
**Step 3: Add API-Specific Features**
```python
# app/middleware/rate_limit.py
from fastapi import Request, HTTPException
from redis import Redis
import time
class RateLimitMiddleware:
def __init__(self, redis_url: str, requests_per_minute: int = 60):
self.redis = Redis.from_url(redis_url)
self.limit = requests_per_minute
async def __call__(self, request: Request, call_next):
api_key = request.headers.get("X-API-Key")
if not api_key:
raise HTTPException(401, "API key required")
# Check rate limit
key = f"rate_limit:{api_key}"
count = self.redis.incr(key)
if count == 1:
self.redis.expire(key, 60) # 1 minute window
if count > self.limit:
raise HTTPException(429, "Rate limit exceeded")
response = await call_next(request)
response.headers["X-RateLimit-Limit"] = str(self.limit)
response.headers["X-RateLimit-Remaining"] = str(max(0, self.limit - count))
return response
```
**Step 4: Deploy to Railway**
```bash
# Link Railway project
railway link
# Set environment variables
railway variables set DATABASE_URL=$DATABASE_URL
railway variables set REDIS_URL=$REDIS_URL
# ... all other vars
# Deploy
railway up --service blackroad-api
```
**Step 5: Update Monolith**
In `BlackRoad-Operating-System/backend/app/main.py`:
```python
from fastapi import FastAPI
from fastapi.staticfiles import StaticFiles
import httpx
app = FastAPI()
# Serve static UI
app.mount("/", StaticFiles(directory="static", html=True), name="static")
# Proxy API calls to new service
@app.api_route("/api/{path:path}", methods=["GET", "POST", "PUT", "DELETE"])
async def api_proxy(path: str, request: Request):
async with httpx.AsyncClient() as client:
url = f"https://api.blackroad.systems/{path}"
response = await client.request(
request.method,
url,
headers=dict(request.headers),
content=await request.body()
)
return Response(response.content, response.status_code, dict(response.headers))
```
**Effort**: 2-3 weeks (includes testing, migration, deployment)
---
## PART 6: PHASE LABEL & MILESTONES
### Phase 2 (Months 12-18)
**Q1 (Months 12-15): API Extraction**
- [ ] Create repo, copy code from monolith
- [ ] Update imports, test locally
- [ ] Deploy to Railway staging
- [ ] Run API contract tests
- [ ] Migrate 50% of traffic to new API
- [ ] Monitor performance, errors
- [ ] Migrate 100% of traffic
**Q2 (Months 15-18): API Enhancements**
- [ ] Add rate limiting middleware
- [ ] Implement API key management
- [ ] Add GraphQL endpoint (optional)
- [ ] Performance optimization (cache, CDN)
- [ ] v2 API alpha (breaking changes)
**Success Metrics**:
- ✅ 99.9% uptime
- ✅ <100ms average latency
- ✅ 1000+ requests/second
- ✅ 0 migration-related incidents
### Phase 3 (Months 18-24+)
- [ ] GraphQL federation
- [ ] gRPC support for internal services
- [ ] Advanced caching (Redis + Cloudflare)
- [ ] API marketplace (third-party integrations)
---
## PART 7: SUCCESS CRITERIA
**Technical**:
- ✅ All 33 routers migrated successfully
- ✅ API contract tests passing (100% coverage)
- ✅ Performance equal or better than monolith
- ✅ Zero downtime during migration
**Business**:
- ✅ No customer-reported issues during migration
- ✅ Developer experience improved (faster API responses)
- ✅ Reduced deployment time (10 min → 3 min)
---
## QUICK REFERENCE
### API Endpoints (33 routers)
**Core**:
- `/api/auth/*` - Authentication
- `/api/users/*` - User management
- `/api/dashboard/*` - Dashboard data
**Social & Communication**:
- `/api/email/*` - RoadMail
- `/api/social/*` - BlackRoad Social
- `/api/discord/*`, `/api/slack/*` - Messaging
**Content & Media**:
- `/api/video/*` - BlackStream
- `/api/games/*` - Gaming
- `/api/browser/*` - Web browsing
**Infrastructure**:
- `/api/blockchain/*` - RoadChain
- `/api/miner/*` - Mining
- `/api/devices/*` - Device management
**DevOps & Cloud**:
- `/api/railway/*`, `/api/vercel/*`, `/api/digitalocean/*`, `/api/cloudflare/*`
**AI & Integrations**:
- `/api/ai_chat/*` - OpenAI
- `/api/agents/*` - Agent execution
- `/api/huggingface/*` - ML models
**Developer Tools**:
- `/api/github/*`, `/api/vscode/*`, `/api/creator/*`
### Essential Commands
```bash
# Local development
uvicorn app.main:app --reload --port 8000
# Run tests
pytest --cov=app
# Deploy
railway up --service blackroad-api
# Logs
railway logs --service blackroad-api --tail 100
# Health check
curl https://api.blackroad.systems/health
```
---
**Last Updated**: 2025-11-18
**Next Review**: Phase 2 kickoff (Month 12)

240
implementation-plans/IMPLEMENTATION_blackroad-io.md Normal file
View File

@@ -0,0 +1,240 @@
# 🚀 IMPLEMENTATION PLAN: blackroad.io
## Corporate Marketing Site
**Repo**: `blackboxprogramming/blackroad.io`
**Purpose**: Marketing site for blackroad.systems domain
**Phase**: **Phase 1 (Months 0-3)**
---
## PURPOSE
**blackroad.io** is the **corporate marketing site** hosted at `blackroad.systems`. It serves:
- Homepage (hero, value prop, CTA)
- Product overview
- Solutions (industry-specific)
- Pricing
- Blog/resources
- Contact/demo request
**Role in Architecture**: **Layer 7** (User Experience)
**Domain**: `blackroad.systems` (primary), `www.blackroad.systems`
---
## TECHNOLOGY OPTIONS
**Option A** (Recommended): Astro + Tailwind
- Static site generator (fast, SEO-friendly)
- Markdown for content
- Easy to maintain
**Option B**: Next.js
- More features (SSR, API routes)
- Vercel deployment
**Option C**: Simple HTML/CSS
- Matches OS aesthetic (Win95 theme)
- Zero build process
- Fast deployment
**Recommendation**: **Astro** (modern, fast, SEO-optimized)
---
## SITE STRUCTURE
### Pages (MVP)
1. **Homepage** (`/`)
- Hero section
- Capabilities overview
- Social proof (case studies)
- CTA (Request Demo)
2. **Architecture** (`/architecture`)
- 7-layer diagram
- Technical overview
- Security & compliance
3. **Solutions** (`/solutions/financial-services`)
- Industry-specific use cases
- ROI calculator
- Customer testimonials
4. **Pricing** (`/pricing`)
- 3 tiers: Free, Team, Enterprise
- Feature comparison table
- FAQ
5. **Contact** (`/contact`)
- Demo request form
- Sales contact
- Support links
### Blog (Phase 2)
- `/blog` - Blog homepage
- `/blog/introducing-blackroad-os` - Launch post
- `/blog/deterministic-ai` - Thought leadership
---
## REPOSITORY STRUCTURE
```
blackroad.io/
├── src/
│ ├── pages/
│ │ ├── index.astro
│ │ ├── architecture.astro
│ │ ├── solutions/
│ │ │ └── financial-services.astro
│ │ ├── pricing.astro
│ │ └── contact.astro
│ ├── components/
│ │ ├── Hero.astro
│ │ ├── Navbar.astro
│ │ └── Footer.astro
│ ├── layouts/
│ │ └── Layout.astro
│ └── styles/
│ └── global.css
├── public/
│ ├── images/
│ └── favicon.ico
├── astro.config.mjs
├── package.json
└── README.md
```
---
## REQUIRED WORKFLOWS
### 1. Deploy to Vercel (`.github/workflows/deploy.yml`)
```yaml
name: Deploy
on:
push:
branches: [main]
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/setup-node@v3
- run: npm install
- run: npm run build
- uses: amondnet/vercel-action@v20
with:
vercel-token: ${{ secrets.VERCEL_TOKEN }}
vercel-org-id: ${{ secrets.VERCEL_ORG_ID }}
vercel-project-id: ${{ secrets.VERCEL_PROJECT_ID }}
```
### 2. Lighthouse CI (`.github/workflows/lighthouse.yml`)
```yaml
name: Lighthouse
on: [pull_request]
jobs:
lighthouse:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: treosh/lighthouse-ci-action@v9
with:
urls: |
https://blackroad.systems
https://blackroad.systems/pricing
uploadArtifacts: true
```
---
## CLOUDFLARE & DOMAINS
**DNS Records** (`blackroad.systems` zone):
| Type | Name | Target | Proxy |
|------|------|--------|-------|
| CNAME | @ | `cname.vercel-dns.com` | ✅ |
| CNAME | www | `blackroad.systems` | ✅ |
**Vercel Custom Domain Setup**:
1. Add `blackroad.systems` in Vercel project settings
2. Vercel provides CNAME target
3. Add CNAME to Cloudflare
4. Wait for SSL provisioning (~5 min)
---
## CONTENT PLAN
### Homepage Copy
**Hero**:
```
Where AI Meets the Open Road
Build deterministic, auditable AI systems with BlackRoad OS.
The operating system for enterprise AI agents.
[Request Demo] [View Documentation]
```
**Capabilities**:
1. **Deterministic AI** - Repeatable, traceable, verifiable
2. **Agent Orchestration** - 200+ autonomous agents
3. **Audit Trails** - Blockchain-backed provenance
4. **Enterprise Ready** - SOC 2, GDPR, HIPAA
### Pricing Tiers
| Feature | Free | Team ($499/mo) | Enterprise (Custom) |
|---------|------|----------------|---------------------|
| Users | 1 | Up to 10 | Unlimited |
| Agents | 10 | 100 | Unlimited |
| API Calls | 1,000/mo | 100,000/mo | Unlimited |
| Support | Community | Email | Dedicated |
| SLA | - | 99.5% | 99.9% |
---
## PHASE 1 MILESTONES
**Week 1-2**: Repo setup, homepage design
**Week 3-4**: Content writing, page implementation
**Week 5**: Deploy, DNS setup
**Week 6**: Launch, promote on social media
**Success Criteria**:
- ✅ Site live at blackroad.systems
- ✅ 100/100 Lighthouse score
- ✅ 10+ demo requests in first month
- ✅ <2s page load time
---
## MARKETING INTEGRATION
**Analytics**:
- Google Analytics 4
- Mixpanel (product analytics)
**SEO**:
- Sitemap.xml
- Robots.txt
- Meta tags (Open Graph, Twitter Card)
**CRM**:
- HubSpot form integration (contact page)
- Salesforce lead creation
---
**Last Updated**: 2025-11-18

140
implementation-plans/IMPLEMENTATION_blackroad-operator.md Normal file
View File

@@ -0,0 +1,140 @@
# 🚀 IMPLEMENTATION PLAN: blackroad-operator
## Workflow Orchestration & Agent Scheduler
**Repo**: `blackboxprogramming/blackroad-operator`
**Purpose**: Agent orchestration, workflow automation, scheduled tasks
**Phase**: **Phase 2 (Months 12-18)**
---
## PURPOSE
**blackroad-operator** is the **workflow orchestration engine** that:
- Runs 200+ agents on schedules (cron-like)
- Orchestrates multi-step workflows
- Integrates with Prism (job queue) and Lucidia (AI)
- Provides human-in-the-loop approval gates
- Manages background tasks and long-running processes
**Role in Architecture**: **Layer 4** (Orchestration & Intelligence)
---
## KEY COMPONENTS
### 1. Scheduler
```python
# app/scheduler.py
from apscheduler.schedulers.asyncio import AsyncIOScheduler
from agents import registry
scheduler = AsyncIOScheduler()
@scheduler.scheduled_job('cron', hour=2) # Daily at 2am
async def backup_database():
agent = registry.get('BackupAgent')
await agent.execute()
@scheduler.scheduled_job('interval', minutes=15)
async def health_check_services():
agent = registry.get('HealthCheckAgent')
await agent.execute()
```
### 2. Workflow Engine
```python
# app/workflows/deploy_flow.py
from app.workflow import Workflow, Step
deploy_workflow = Workflow(
name="Deploy New Feature",
steps=[
Step("lint", agent="LintAgent"),
Step("test", agent="TestAgent", depends_on=["lint"]),
Step("build", agent="BuildAgent", depends_on=["test"]),
Step("deploy_staging", agent="DeployAgent", config={"env": "staging"}),
Step("smoke_test", agent="SmokeTestAgent", depends_on=["deploy_staging"]),
Step("human_approval", type="approval", timeout="24h"),
Step("deploy_prod", agent="DeployAgent", config={"env": "production"}),
]
)
```
### 3. Prism Integration
```python
# app/prism.py
import httpx
class PrismClient:
def __init__(self, base_url: str):
self.base_url = base_url
async def create_job(self, job_type: str, metadata: dict):
async with httpx.AsyncClient() as client:
response = await client.post(
f"{self.base_url}/api/jobs",
json={"type": job_type, "metadata": metadata}
)
return response.json()
async def stream_job_status(self, job_id: str):
# WebSocket connection for real-time updates
pass
```
---
## MIGRATION FROM MONOLITH
**What Moves**:
- `agents/` directory (all 200+ agents)
- New: Scheduler code
- New: Workflow definitions
- New: Prism integration
**Migration Steps**:
1. Copy `agents/` from monolith
2. Add scheduler (APScheduler or Celery Beat)
3. Create workflow engine
4. Deploy to Railway as worker service
5. Connect to Prism via API
**Effort**: 3-4 weeks
---
## REQUIRED WORKFLOWS
1. **CI/CD** - Lint, test agents, deploy to Railway
2. **Agent Tests** - Unit tests for all 200+ agents
3. **Integration Tests** - Test workflows end-to-end
4. **Performance Monitoring** - Track agent execution time
---
## CLOUDFLARE & DOMAINS
**Domain**: `operator.blackroad.systems` (internal only, not public)
**Access**: API gateway proxies requests from `api.blackroad.systems/api/agents/*`
---
## PHASE 2 MILESTONES
**Month 12-13**: Repo setup, agent migration
**Month 14-15**: Scheduler implementation
**Month 16-17**: Workflow engine
**Month 18**: Production deployment
**Success Criteria**:
- ✅ All 200+ agents migrated
- ✅ 10+ scheduled jobs running daily
- ✅ 5+ workflows in production
- ✅ 99.5% agent success rate
---
**Last Updated**: 2025-11-18

193
implementation-plans/IMPLEMENTATION_blackroad-prism-console.md Normal file
View File

@@ -0,0 +1,193 @@
# 🚀 IMPLEMENTATION PLAN: blackroad-prism-console
## Admin Dashboard for Observability & Job Management
**Repo**: `blackboxprogramming/blackroad-prism-console`
**Purpose**: Web UI for Prism job queue, system metrics, audit logs
**Phase**: **Phase 2 (Months 14-18)**
---
## PURPOSE
**blackroad-prism-console** is the **admin dashboard** for monitoring and managing the BlackRoad OS ecosystem:
- Visualize Prism job queue
- Monitor agent execution
- View system metrics (Prometheus)
- Audit logs (Vault)
- User management
- Configuration management
**Role in Architecture**: **Layer 6** (Application Layer) - Admin UI
**Domain**: `prism.blackroad.systems`
---
## TECHNOLOGY STACK
**Option A** (Recommended): React + TypeScript
- Modern, scalable
- Rich ecosystem (React Query, Recharts, Ant Design)
- TypeScript for type safety
**Option B**: Vanilla JS (consistent with OS aesthetic)
- Zero dependencies
- Windows 95 theme
- Simple, fast
**Recommendation**: **React + TypeScript** (professional admin tool, different audience than OS)
---
## KEY FEATURES
### 1. Job Queue Dashboard
**Components**:
- Job list (filterable by status, type, date)
- Job details (logs, timeline, metadata)
- Real-time updates (WebSocket)
- Job actions (cancel, retry, clone)
**Example**:
```typescript
// src/components/JobQueue.tsx
import { useQuery } from 'react-query'
import { Table, Badge } from 'antd'
export function JobQueue() {
const { data: jobs } = useQuery('jobs', fetchJobs, {
refetchInterval: 5000 // Poll every 5 seconds
})
return (
<Table
dataSource={jobs}
columns={[
{ title: 'ID', dataIndex: 'id' },
{ title: 'Type', dataIndex: 'type' },
{ title: 'Status', dataIndex: 'status', render: (status) => (
<Badge
status={status === 'running' ? 'processing' : 'default'}
text={status}
/>
)},
{ title: 'Created', dataIndex: 'created_at' },
]}
/>
)
}
```
### 2. System Metrics
**Metrics Sources**:
- Prometheus (backend metrics)
- PostgreSQL (database stats)
- Redis (cache hit rate)
- Railway (infrastructure stats)
**Dashboards**:
- API Performance (latency, throughput, errors)
- Database Performance (queries, connections, slow queries)
- Agent Execution (success rate, duration)
- User Activity (active users, sessions)
### 3. Audit Logs (Vault)
**Features**:
- Search logs by user, action, date
- Export logs (CSV, JSON)
- Compliance reports (SOX, GDPR)
- Immutable log viewer
---
## REPOSITORY STRUCTURE
```
blackroad-prism-console/
├── src/
│ ├── components/
│ │ ├── JobQueue.tsx
│ │ ├── Metrics.tsx
│ │ ├── AuditLogs.tsx
│ │ └── UserManagement.tsx
│ ├── pages/
│ │ ├── Dashboard.tsx
│ │ ├── Jobs.tsx
│ │ ├── Metrics.tsx
│ │ └── Settings.tsx
│ ├── api/
│ │ └── client.ts # API client for blackroad-api
│ ├── hooks/
│ │ └── useWebSocket.ts # Real-time updates
│ └── App.tsx
├── public/
├── package.json
├── tsconfig.json
├── vite.config.ts # Build tool (Vite)
└── README.md
```
---
## REQUIRED WORKFLOWS
1. **CI/CD** - Build, test, deploy to Vercel/Railway
2. **Type Check** - TypeScript type checking
3. **E2E Tests** - Playwright for UI tests
4. **Lighthouse** - Performance audits
---
## CLOUDFLARE & DOMAINS
**DNS Records** (`blackroad.systems` zone):
| Type | Name | Target | Proxy |
|------|------|--------|-------|
| CNAME | prism | `blackroad-prism-console.vercel.app` | ✅ |
**Deployment**: Vercel (recommended for React SPAs)
---
## MIGRATION NOTES
**New Repo** (not migrated from monolith)
**Bootstrap**:
```bash
# Create React app with TypeScript
npm create vite@latest blackroad-prism-console -- --template react-ts
# Install dependencies
cd blackroad-prism-console
npm install react-query axios antd recharts
# Connect to blackroad-api
# Create .env.production
VITE_API_URL=https://api.blackroad.systems
VITE_WS_URL=wss://api.blackroad.systems/ws
```
**Effort**: 4-6 weeks (full dashboard build)
---
## PHASE 2 MILESTONES
**Month 14-15**: Repo setup, basic job queue UI
**Month 16-17**: Metrics dashboards, audit logs
**Month 18**: User management, production deployment
**Success Criteria**:
- ✅ Real-time job queue updates
- ✅ <1s dashboard load time
- ✅ 100% mobile responsive
- ✅ Used daily by team
---
**Last Updated**: 2025-11-18

233
implementation-plans/IMPLEMENTATION_blackroad.md Normal file
View File

@@ -0,0 +1,233 @@
# 🚀 IMPLEMENTATION PLAN: blackroad
## Investigation & Recommendation
**Repo**: `blackboxprogramming/blackroad`
**Status**: **REQUIRES INVESTIGATION**
**Phase**: **Phase 1 (Week 1) - Immediate**
---
## PURPOSE
**Current State**: Unknown - need to investigate repo contents
**Possible Scenarios**:
### Scenario A: Alternate Frontend
If `blackroad` contains an alternate or legacy frontend UI:
- **Recommendation**: Archive or merge into `backend/static/` in monolith
- **Reason**: Avoid duplication, maintain single canonical UI
### Scenario B: Standalone Service
If `blackroad` is a standalone service (API, worker, etc.):
- **Recommendation**: Rename to clarify purpose (e.g., `blackroad-worker`)
- **Action**: Document in ORG_STRUCTURE.md, create IMPLEMENTATION.md
### Scenario C: Experiment/Prototype
If `blackroad` is experimental or deprecated code:
- **Recommendation**: Archive (move to `archive/blackroad` or mark as read-only)
- **Action**: Add deprecation notice, link to canonical repo
### Scenario D: Empty/Stub Repo
If `blackroad` is empty or just a placeholder:
- **Recommendation**: Delete or repurpose for future use
- **Action**: Consider using for `blackroad` GitHub org (if creating new org)
---
## INVESTIGATION CHECKLIST
### Step 1: Access Repo
```bash
# Clone repo
git clone https://github.com/blackboxprogramming/blackroad.git
cd blackroad
# Check contents
ls -la
cat README.md
```
### Step 2: Analyze Contents
- [ ] Check README.md for purpose description
- [ ] Review directory structure
- [ ] Check package.json / requirements.txt for dependencies
- [ ] Review git history (`git log --oneline`)
- [ ] Check last commit date (`git log -1`)
- [ ] Look for deployment config (Dockerfile, railway.toml, vercel.json)
### Step 3: Compare with Monolith
If frontend:
```bash
# Compare with canonical frontend
diff -r blackroad/src ../BlackRoad-Operating-System/backend/static/
```
If backend:
```bash
# Check for duplicate routers
diff -r blackroad/app ../BlackRoad-Operating-System/backend/app/
```
### Step 4: Check Deployment
- [ ] Is repo deployed anywhere? (Railway, Vercel, etc.)
- [ ] Does a domain point to it?
- [ ] Is it actively used?
### Step 5: Decide Fate
Based on findings, choose action:
| Finding | Action | Timeline |
|---------|--------|----------|
| Duplicate frontend | Merge into `backend/static/`, archive | Week 1 |
| Unique service | Document, create IMPLEMENTATION.md | Week 2 |
| Experimental/prototype | Archive, add deprecation notice | Week 1 |
| Empty/stub | Delete or repurpose | Week 1 |
---
## RECOMMENDED ACTIONS (After Investigation)
### If Archiving:
1. Add deprecation README:
```markdown
# ⚠️ ARCHIVED: blackroad
This repository has been **archived** and is no longer maintained.
**Reason**: [Duplicate of X / Superseded by Y / Experimental only]
**Canonical Repo**: [Link to BlackRoad-Operating-System or other]
**Last Active**: [Date]
For the current BlackRoad OS, see: https://github.com/blackboxprogramming/BlackRoad-Operating-System
```
2. Mark as archived in GitHub settings
3. Update ORG_STRUCTURE.md
4. Remove from active CI/CD
### If Merging:
1. Create migration branch in monolith
2. Copy unique code (if any)
3. Test merged code
4. Archive old repo
5. Update documentation
### If Keeping as Standalone:
1. Clarify purpose in README
2. Create IMPLEMENTATION.md (use templates from other repos)
3. Add to ORG_STRUCTURE.md with clear role
4. Ensure CI/CD is active
5. Deploy if not already
---
## INVESTIGATION TIMELINE
**Day 1** (1-2 hours):
- [ ] Clone repo
- [ ] Analyze contents
- [ ] Compare with monolith
- [ ] Document findings
**Day 2** (2-4 hours):
- [ ] Decide action (archive, merge, keep)
- [ ] Execute decision
- [ ] Update ORG_STRUCTURE.md
- [ ] Create IMPLEMENTATION.md (if keeping)
**Day 3** (optional, if merging):
- [ ] Merge code into monolith
- [ ] Test merged code
- [ ] Archive old repo
---
## TEMPLATE INVESTIGATION REPORT
Create: `investigation-reports/blackroad-investigation.md`
```markdown
# Investigation Report: blackroad
**Date**: 2025-11-18
**Investigator**: [Your Name]
## Findings
**Repo URL**: https://github.com/blackboxprogramming/blackroad
**Last Commit**: [Date]
**Primary Language**: [Language]
**Lines of Code**: [Count]
**Purpose** (from README/code analysis):
[Description]
**Directory Structure**:
```
[Tree output]
```
**Dependencies** (key packages):
- [Package 1]
- [Package 2]
**Deployment Status**:
- [ ] Deployed to Railway/Vercel
- [ ] Domain: [domain if any]
- [ ] Active users: [Yes/No]
## Comparison with Monolith
**Unique Code**: [Yes/No - describe]
**Duplicate Code**: [Yes/No - describe]
**Integration**: [How it relates to monolith]
## Recommendation
**Action**: [Archive / Merge / Keep / Delete]
**Reasoning**: [Why this action]
**Next Steps**:
1. [Step 1]
2. [Step 2]
## Attachments
- Screenshot: [If UI]
- Code diff: [If duplicate]
```
---
## IMPORTANCE
**Priority**: **HIGH** - This investigation should happen in Week 1
**Why Important**:
- Clarifies repo landscape
- Prevents duplicate work
- Reduces maintenance burden
- Improves org cleanliness
**Blockers**:
- None (just need access to repo)
---
## SUCCESS CRITERIA
After investigation:
- ✅ Clear understanding of repo purpose
- ✅ Decision made (archive/merge/keep/delete)
- ✅ Action executed
- ✅ Documentation updated (ORG_STRUCTURE.md)
- ✅ No lingering uncertainty
---
**Last Updated**: 2025-11-18
**Next Action**: Alexa investigates repo, fills out investigation report

306
implementation-plans/IMPLEMENTATION_lucidia.md Normal file
View File

@@ -0,0 +1,306 @@
# 🚀 IMPLEMENTATION PLAN: lucidia
## Multi-Model AI Orchestration Layer
**Repo**: `blackboxprogramming/lucidia`
**Purpose**: AI orchestration, multi-model routing, agent intelligence
**Phase**: **Phase 1 (Q3-Q4) → Phase 2 (Expansion)**
---
## PURPOSE
**Lucidia** is the **AI intelligence layer** that:
- Routes requests to multiple AI models (Claude, GPT-4, Llama, Gemini)
- Orchestrates multi-agent conversations
- Manages long-term memory and context
- Provides personas (Cece, Amundson, etc.)
- Tool calling and function execution
- Cost optimization (use cheaper models when appropriate)
**Role in Architecture**: **Layer 4** (Orchestration & Intelligence)
**Domain**: `lucidia.blackroad.systems` (API), `lucidia.earth` (narrative site, Phase 2)
---
## CORE ARCHITECTURE
### 1. Multi-Model Router
```python
# lucidia/router.py
from anthropic import Anthropic
from openai import OpenAI
import requests
class ModelRouter:
def __init__(self):
self.claude = Anthropic(api_key=os.getenv("ANTHROPIC_API_KEY"))
self.openai = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))
async def route(self, prompt: str, preferences: dict):
"""Route to best model based on task, cost, latency."""
# Task classification
task_type = self.classify_task(prompt)
# Routing logic
if task_type == "code":
return await self.call_claude(prompt, model="claude-sonnet-4")
elif task_type == "creative":
return await self.call_openai(prompt, model="gpt-4")
elif task_type == "fast":
return await self.call_openai(prompt, model="gpt-3.5-turbo")
else:
# Default to Claude
return await self.call_claude(prompt)
async def call_claude(self, prompt: str, model: str = "claude-3-5-sonnet-20241022"):
response = self.claude.messages.create(
model=model,
max_tokens=4096,
messages=[{"role": "user", "content": prompt}]
)
return response.content[0].text
async def call_openai(self, prompt: str, model: str = "gpt-4"):
response = self.openai.chat.completions.create(
model=model,
messages=[{"role": "user", "content": prompt}]
)
return response.choices[0].message.content
```
### 2. Multi-Agent Orchestration
```python
# lucidia/orchestrator.py
class AgentOrchestrator:
def __init__(self):
self.agents = {
"cece": Agent(name="Cece", role="OS Architect", model="claude-sonnet-4"),
"amundson": Agent(name="Amundson", role="Quantum Physicist", model="claude-opus-4"),
"designer": Agent(name="Designer", role="UI/UX", model="gpt-4"),
}
async def orchestrate(self, task: str):
"""Orchestrate multiple agents to complete a complex task."""
# Step 1: Analyze task
plan = await self.agents["cece"].plan(task)
# Step 2: Execute subtasks
results = []
for subtask in plan.subtasks:
agent = self.agents[subtask.agent]
result = await agent.execute(subtask.description)
results.append(result)
# Step 3: Synthesize results
final = await self.agents["cece"].synthesize(results)
return final
```
### 3. Long-Term Memory
```python
# lucidia/memory.py
from chromadb import Client
class Memory:
def __init__(self):
self.chroma = Client()
self.collection = self.chroma.create_collection("lucidia_memory")
async def store(self, text: str, metadata: dict):
"""Store conversation or context."""
self.collection.add(
documents=[text],
metadatas=[metadata],
ids=[str(uuid.uuid4())]
)
async def recall(self, query: str, n_results: int = 5):
"""Retrieve relevant context."""
results = self.collection.query(
query_texts=[query],
n_results=n_results
)
return results
```
---
## INTEGRATION WITH BLACKROAD OS
### API Endpoints
**In `blackroad-api`** (proxy to Lucidia):
```python
# app/routers/lucidia.py
from fastapi import APIRouter
import httpx
router = APIRouter(prefix="/api/lucidia", tags=["lucidia"])
@router.post("/chat")
async def chat(message: str, model: str = "auto"):
async with httpx.AsyncClient() as client:
response = await client.post(
"https://lucidia.blackroad.systems/chat",
json={"message": message, "model": model}
)
return response.json()
@router.post("/orchestrate")
async def orchestrate(task: str):
"""Multi-agent task orchestration."""
async with httpx.AsyncClient() as client:
response = await client.post(
"https://lucidia.blackroad.systems/orchestrate",
json={"task": task}
)
return response.json()
```
### OS App Integration
**Lucidia Core App** (in OS UI):
```javascript
// backend/static/js/apps/lucidia.js
window.Apps.Lucidia = {
init() {
this.messages = [];
},
async sendMessage(message) {
const response = await fetch('/api/lucidia/chat', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ message, model: 'auto' })
});
const data = await response.json();
this.messages.push({ role: 'user', content: message });
this.messages.push({ role: 'assistant', content: data.response });
this.render();
},
render() {
return `
<div class="lucidia-chat">
${this.messages.map(msg => `
<div class="message ${msg.role}">
<strong>${msg.role}:</strong> ${msg.content}
</div>
`).join('')}
<input type="text" id="lucidia-input" placeholder="Ask Lucidia...">
<button onclick="Apps.Lucidia.sendMessage(document.getElementById('lucidia-input').value)">Send</button>
</div>
`;
}
};
```
---
## REPOSITORY STRUCTURE
```
lucidia/
├── lucidia/
│ ├── __init__.py
│ ├── router.py # Multi-model routing
│ ├── orchestrator.py # Multi-agent orchestration
│ ├── memory.py # Long-term memory (ChromaDB)
│ ├── personas.py # Cece, Amundson, etc.
│ ├── tools.py # Function calling
│ └── config.py # Model configs, API keys
├── api/
│ └── main.py # FastAPI service
├── tests/
│ ├── test_router.py
│ ├── test_orchestrator.py
│ └── test_memory.py
├── requirements.txt
├── Dockerfile
└── README.md
```
---
## TECHNOLOGY STACK
**Core**:
- Python 3.11+
- FastAPI (API service)
- Anthropic SDK (Claude)
- OpenAI SDK (GPT-4)
- ChromaDB (vector memory)
**Optional**:
- LangChain (agent framework)
- LlamaIndex (context management)
- Replicate (open-source models)
---
## PHASE 1 MILESTONES
**Month 6-7**: Basic multi-model routing
**Month 8-9**: Multi-agent orchestration
**Month 10-11**: Long-term memory integration
**Month 12**: Production deployment, OS integration
**Success Criteria**:
- ✅ 3+ models supported (Claude, GPT-4, Llama)
- ✅ <2s average response time
- ✅ 95% user satisfaction
- ✅ 10+ personas available
---
## PHASE 2 EXPANSION
**lucidia.earth** (Narrative Site):
- Public-facing AI experiences
- Interactive stories
- AI art gallery
- Community showcase
**Advanced Features**:
- Fine-tuned models (custom Lucidia models)
- Multimodal (image, audio, video)
- Real-time collaboration (multiple users + AI)
---
## CLOUDFLARE & DOMAINS
**DNS Records**:
| Type | Name | Target | Proxy |
|------|------|--------|-------|
| CNAME | lucidia.blackroad | `lucidia-api.up.railway.app` | ✅ |
| CNAME | lucidia.earth | `lucidia-narrative.vercel.app` | ✅ |
---
## COST OPTIMIZATION
**Model Cost Comparison**:
- Claude Sonnet 4: $3 / 1M input tokens
- GPT-4: $10 / 1M input tokens
- GPT-3.5 Turbo: $0.50 / 1M input tokens
- Llama 3 (open-source): Free (hosting cost only)
**Strategy**:
- Use GPT-3.5 for simple queries (classification, summarization)
- Use Claude Sonnet for complex reasoning (code, analysis)
- Use GPT-4 for creative tasks (copywriting, brainstorming)
- Cache common queries in Redis
**Projected Savings**: 60% vs. using GPT-4 for everything
---
**Last Updated**: 2025-11-18