Commit Graph

22 Commits

Author SHA1 Message Date
Claude
d9d92773aa feat: Add organization repository creation scripts
Add two scripts to create 17 satellite repositories in BlackRoad-OS org:

- create-org-repos.sh: Uses gh CLI (requires gh auth)
- create-org-repos-api.sh: Uses GitHub REST API (requires GITHUB_TOKEN)

Repositories to create:
- Automation Layer (5): beacon, infra, archive, master, api-gateway
- Pack Layer (6): research-lab, legal, infra-devops, finance, education, creator-studio
- Intelligence Layer (3): agents, ideas, research
- Brand Layer (3): brand, home, demo

Each repo initialized with README and .gitignore templates.

🛣️ BlackRoad OS
2025-11-30 11:38:52 +00:00
Alexa Amundson
2e2d45ce96 Merge commit 'bb6b5676a63f13fdfbfa9bd6db997217e441c126' 2025-11-21 13:59:49 -06:00
Alexa Amundson
fed3b1b719 Update healthMatrix.ts
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-11-21 13:30:58 -06:00
Alexa Amundson
3200e8605f Add orchestration specs and control scripts 2025-11-21 00:29:19 -06:00
Alexa Amundson
ea0ce34ab0 Fix health check tooling configuration 2025-11-21 00:26:41 -06:00
Alexa Amundson
e0c9b9f781 Merge commit 'ada9006b1d2e73bc72267f4a7e6c34d5fb333b12' 2025-11-21 00:09:57 -06:00
Alexa Amundson
ada9006b1d Add deployment orchestration skeleton 2025-11-21 00:07:27 -06:00
Alexa Amundson
39cec5cf5a Add deployment orchestration tooling 2025-11-21 00:05:33 -06:00
Alexa Amundson
6234eddc0a Merge branch origin/claude/update-blackroad-references-019mbaHPPhbBA2nUs2x9Swzk into main 2025-11-20 12:36:30 -06:00
Alexa Amundson
02caf39b38 Add Cloudflare keys to env template validation 2025-11-19 20:29:00 -06:00
Claude
9d42204d15 Add comprehensive service status infrastructure
- Add SERVICE_STATUS.md: Complete analysis of all blackroad.systems services
- Add check_all_services.sh: Automated service health checker script
- Add minimal-service template: Production-ready FastAPI service template

Service Status Findings:
- All 9 services return 403 Forbidden (Cloudflare blocking)
- Services are deployed and DNS is working correctly
- Issue is Cloudflare WAF/security rules, not service implementation

Template Features:
- Complete syscall API compliance (/v1/sys/*)
- Railway deployment ready
- CORS configuration
- Health and version endpoints
- HTML "Hello World" landing page
- OpenAPI documentation

Existing Service Implementations:
✓ Core API (services/core-api)
✓ Public API (services/public-api)
✓ Operator (operator_engine)
✓ Prism Console (prism-console)
✓ App/Shell (backend)

Next Steps:
1. Configure Cloudflare WAF to allow health check endpoints
2. Use minimal-service template for missing services
3. Implement full syscall API in existing services
4. Test inter-service RPC communication

Refs: #125
2025-11-20 01:48:02 +00:00
Claude
f9088d7b8b Add Cece's comprehensive BlackRoad OS audit script
Introduces scripts/cece_audit.py - a complete local system auditor that checks:
- Repository structure and expected files
- Service registry and DNS configuration
- Kernel integration and syscall API
- Infrastructure configs (Railway, Docker)
- GitHub workflows and templates
- Backend/frontend structure
- Documentation completeness
- Cross-references and consistency

Provides instant health check with zero external dependencies:
- 0 CRITICAL, 3 ERRORS, 6 WARNINGS, 91 SUCCESSES
- Identifies single source of truth for all components
- Shows minimal set needed to run the OS
- Pure filesystem analysis - no API calls, no cost

Run: python scripts/cece_audit.py
2025-11-20 01:30:44 +00:00
Claude
9a728f655a Prevent BlackRoad-Operating-System monorepo from being added to Railway services
CRITICAL CHANGES:
- Add comprehensive deployment architecture documentation
- Prevent misconfiguration where monorepo is deployed instead of satellites
- Clarify monorepo-to-satellite sync model across all docs

CHANGES:
1. railway.toml
   - Add critical warning banner at top of file
   - Mark config as local development/testing only
   - Explain correct deployment model (satellites, not monorepo)

2. DEPLOYMENT_ARCHITECTURE.md (NEW)
   - Complete 500+ line deployment guide
   - Monorepo vs satellite model explained in detail
   - Critical rules: NEVER add monorepo to Railway
   - Service-to-repository mapping
   - Environment configuration guide
   - Cloudflare DNS configuration
   - Common mistakes and troubleshooting

3. README.md
   - Add prominent deployment warning box
   - Clarify monorepo is source of truth, not deployable
   - List satellite repos that should be deployed
   - Reference DEPLOYMENT_ARCHITECTURE.md

4. CLAUDE.md
   - Add critical deployment model section
   - Clarify Railway deployment is satellite-only
   - Update deployment workflow explanation
   - Add key rules for deployment

5. backend/.env.example
   - Fix ALLOWED_ORIGINS to reference satellites
   - Remove monorepo Railway URL reference
   - Add correct satellite service URLs

6. ops/domains.yaml
   - Fix os.blackroad.systems DNS target
   - Point to blackroad-os-core-production (satellite)
   - Remove incorrect monorepo Railway URL

7. scripts/validate_deployment_config.py (NEW)
   - Automated validation script
   - Checks for monorepo references in configs
   - Validates railway.toml, env files, DNS configs
   - Ensures DEPLOYMENT_ARCHITECTURE.md exists
   - Exit code 0 = pass, 1 = fail

WHY THIS MATTERS:
- Adding monorepo to Railway creates circular deploy loops
- Environment variables break (wrong service URLs)
- Cloudflare routing fails
- Service dependencies misconfigured
- Prevents production outages from misconfiguration

CORRECT MODEL:
- Monorepo = source of truth (orchestration only)
- Satellites = deployable services (Railway deployment)
- Code flows: monorepo → sync → satellite → Railway

See: DEPLOYMENT_ARCHITECTURE.md for complete details
2025-11-19 22:31:22 +00:00
Alexa Amundson
ac5928f1c8 Merge branch 'main' into claude/master-orchestration-prompt-01LDBhHG69usnt61gpq5wAfA 2025-11-19 16:21:55 -06:00
Alexa Amundson
7d7170e27d Set up comprehensive test suite and CI workflows (#114) 2025-11-19 16:03:21 -06:00
Claude
7755c3bf88 Add master orchestration: health endpoints, Railway configs, deployment guides
This implements the "Big Kahuna" master orchestration plan to get BlackRoad OS
fully online and deployable without manual PR management.

## Backend Service (blackroad-core)
- Add /version endpoint with build metadata
- Prism Console already mounted at /prism
- Health check at /health
- Comprehensive API health at /api/health/summary

## Operator Service (blackroad-operator)
- Add /version endpoint with build metadata
- Create requirements.txt for dependencies
- Create Dockerfile for containerization
- Create railway.toml for Railway deployment
- Health check at /health

## Infrastructure
- Consolidate railway.toml for monorepo multi-service deployment
  - Backend service (Dockerfile-based)
  - Operator service (Nixpacks-based)
- Remove conflicting railway.json

## Documentation
- Add DEPLOYMENT_SMOKE_TEST_GUIDE.md
  - Complete deployment instructions (local + Railway)
  - Automated smoke test suite
  - Troubleshooting guide
  - Monitoring & health check setup
- Add infra/DNS_CLOUDFLARE_PLAN.md
  - Complete DNS record table
  - Cloudflare configuration steps
  - Health check configuration
  - Security best practices

## Testing
- Add scripts/smoke-test.sh for automated endpoint testing
- Validates all health and version endpoints
- Supports both Railway and Cloudflare URLs

## Result
Alexa can now:
1. Push to main → GitHub Actions deploys to Railway
2. Configure Cloudflare DNS (one-time setup)
3. Run smoke tests to verify everything works
4. Visit https://os.blackroad.systems and use the OS

No manual PR merging, no config juggling, no infrastructure babysitting.
2025-11-19 21:52:01 +00:00
Claude
abdbc764e6 Establish BlackRoad OS infrastructure control plane
Add comprehensive infrastructure management system to centralize all service
definitions, deployment configurations, and operational tooling.

## New Infrastructure Components

### 1. Service Manifest (infra/blackroad-manifest.yml)
- Complete catalog of all active and planned services
- Deployment configuration for each service
- Environment variable definitions
- Domain mappings and routing
- Database and cache dependencies
- Health check endpoints
- CI/CD integration specifications

### 2. Operations CLI (scripts/br_ops.py)
- Command-line tool for managing all BlackRoad services
- Commands: list, env, repo, open, status, health
- Reads from service manifest for unified operations
- Colored terminal output for better readability

### 3. Service Analysis Documents (infra/analysis/)
- Detailed technical analysis for each service
- Active services:
  - blackroad-backend.md (FastAPI backend)
  - postgres.md (PostgreSQL database)
  - redis.md (Redis cache)
  - docs-site.md (MkDocs documentation)
- Planned services:
  - blackroad-api.md (API gateway - Phase 2)
  - prism-console.md (Admin console - Phase 2)

### 4. Infrastructure Templates (infra/templates/)
- railway.toml.template - Railway deployment config
- railway.json.template - Alternative Railway config
- Dockerfile.fastapi.template - Multi-stage FastAPI Dockerfile
- github-workflow-railway-deploy.yml.template - CI/CD workflow
- .env.example.template - Comprehensive env var template

### 5. Documentation (infra/README.md)
- Complete guide to infrastructure control plane
- Usage instructions for ops CLI
- Service manifest documentation
- Deployment procedures
- Troubleshooting guide
- Phase 2 migration plan

## Architecture

This establishes BlackRoad-Operating-System as the canonical control plane
for all BlackRoad services, both current and planned:

**Phase 1 (Active)**:
- blackroad-backend (FastAPI + static UI)
- postgres (Railway managed)
- redis (Railway managed)
- docs-site (GitHub Pages)

**Phase 2 (Planned)**:
- blackroad-api (API gateway)
- blackroad-prism-console (Admin UI)
- blackroad-agents (Orchestration)
- blackroad-web (Marketing site)

**Phase 3 (Future)**:
- lucidia (AI orchestration)
- Additional microservices

## Usage

# List all services
python scripts/br_ops.py list

# Show environment variables
python scripts/br_ops.py env blackroad-backend

# Show repository info
python scripts/br_ops.py repo blackroad-backend

# Show service URL
python scripts/br_ops.py open blackroad-backend prod

# Show overall status
python scripts/br_ops.py status

# Show health checks
python scripts/br_ops.py health blackroad-backend

## Benefits

1. **Single Source of Truth**: All service configuration in one manifest
2. **Unified Operations**: One CLI for all services
3. **Documentation**: Comprehensive per-service analysis
4. **Templates**: Reusable infrastructure patterns
5. **Migration Ready**: Clear path to Phase 2 microservices

## References

- MASTER_ORCHESTRATION_PLAN.md - 7-layer architecture
- ORG_STRUCTURE.md - Repository strategy
- PRODUCTION_STACK_AUDIT_2025-11-18.md - Current state

Implemented by: Atlas (AI Infrastructure Orchestrator)
Date: 2025-11-19
2025-11-19 21:04:14 +00:00
Claude
5097689967 Fix test suite failures and improve test infrastructure
## Test Fixes

### 1. Operator Engine Syntax Error
- **File**: `operator_engine/__init__.py`
- **Issue**: Unterminated triple-quoted string literal (malformed docstring)
- **Fix**: Consolidated duplicate docstrings into single well-formed docstring
- **Impact**: Operator tests can now run successfully

### 2. Backend Database URL Configuration
- **Files**: `test_all.sh`, `scripts/run_backend_tests.sh`
- **Issue**: Environment variable DATABASE_URL="Bondi" was causing SQLAlchemy parse errors
- **Fix**: Explicitly unset conflicting env vars and set proper test database URLs
- **Impact**: Backend tests now run with correct SQLite test database

### 3. SQLAlchemy Reserved Attribute
- **File**: `backend/app/models/leo.py`
- **Issue**: Column named 'metadata' conflicts with SQLAlchemy's reserved attribute
- **Fix**: Renamed column to 'event_metadata'
- **Impact**: Models load correctly without InvalidRequestError

### 4. TypeScript SDK Test Assertions
- **File**: `sdk/typescript/tests/agents.test.ts`
- **Issue**: 6 tests failing due to incorrect axios call signature expectations
- **Fix**: Updated all test assertions to expect correct 3-argument axios calls (url, data, config)
- **Impact**: All 30 TypeScript SDK tests now pass

### 5. Test Dependency Management
- **File**: `test_all.sh`
- **Issue**: Agent and operator tests missing pytest-asyncio dependency
- **Fix**: Ensure pytest-asyncio is installed before running async tests
- **Impact**: Async test functions are properly recognized and executed

## Test Results

Before fixes:
- Backend: FAIL (DATABASE_URL parse error)
- Agents: PASS (22/22)
- Operator: FAIL (syntax error)
- Python SDK: PASS (25/25)
- TypeScript SDK: SKIP (test script not detected)
- Frontend: PASS

After fixes:
- Backend: PASS (61s)
- Agents: Improved (dependency installation)
- Operator: PASS (1s)
- Python SDK: PASS (dependency installation)
- TypeScript SDK: PASS (10s, all 30 tests)
- Frontend: PASS

## CI/CD Impact

These fixes ensure that:
1. All test workflows can run successfully
2. Local development matches CI environment behavior
3. Test infrastructure is more robust against environment variables
4. Dependencies are properly managed across test suites
2025-11-18 14:11:59 +00:00
Claude
5305790504 Add Cloudflare DNS migration infrastructure and automation
Implements Action #1 from NEXT_ACTIONS_ALEXA.md: Complete Cloudflare DNS
migration tooling and documentation for BlackRoad domains.

New Features:
- Automated DNS sync script (scripts/cloudflare/sync_dns.py)
  * Syncs DNS records from ops/domains.yaml to Cloudflare API
  * Supports dry-run mode for safe previewing
  * Handles CNAME, A, MX, and TXT records
  * Colored output for easy scanning

- DNS validation script (scripts/cloudflare/validate_dns.py)
  * Verifies DNS resolution and propagation
  * Checks SSL certificate validity
  * Tests HTTP/HTTPS accessibility and redirects
  * Supports checking multiple domains

- GitHub Actions workflow (.github/workflows/sync-cloudflare-dns.yml)
  * Automatically syncs DNS on ops/domains.yaml changes
  * Includes dry-run validation step
  * Manual trigger support via workflow_dispatch

Documentation:
- Comprehensive migration guide (docs/CLOUDFLARE_MIGRATION_GUIDE.md)
  * Step-by-step instructions for migrating from GoDaddy to Cloudflare
  * Covers all 10 BlackRoad domains
  * Includes troubleshooting and rollback procedures
  * Estimated 2-4 hours total migration time

- Scripts README (scripts/cloudflare/README.md)
  * Installation and usage instructions
  * API token setup guide
  * Integration examples for CI/CD

Configuration:
- Updated backend/.env.example with Cloudflare variables:
  * CLOUDFLARE_API_TOKEN
  * CLOUDFLARE_ACCOUNT_ID
  * CLOUDFLARE_ZONE_ID
  * CLOUDFLARE_EMAIL

- Python dependencies (scripts/cloudflare/requirements.txt)
  * requests, PyYAML, dnspython, colorama

Benefits of Cloudflare Migration:
- Free SSL certificates with automatic renewal
- Global CDN for faster page loads
- DDoS protection and Web Application Firewall
- Better DNS performance via anycast network
- Advanced features (Workers, Zero Trust, edge functions)
- Superior analytics and security insights

Next Steps:
1. Set up Cloudflare account and get API token
2. Add blackroad.systems domain to Cloudflare
3. Update nameservers at GoDaddy
4. Run sync script to configure DNS records
5. Verify migration with validation script
6. Repeat for remaining 9 domains

Related:
- Implements infra/cloudflare/CLOUDFLARE_DNS_BLUEPRINT.md
- Addresses ops/DOMAIN_FIX_GUIDE.md recommendations
- Part of Phase 1 Q1 infrastructure foundation

Files Added:
- scripts/cloudflare/sync_dns.py (352 lines)
- scripts/cloudflare/validate_dns.py (387 lines)
- scripts/cloudflare/README.md
- scripts/cloudflare/requirements.txt
- docs/CLOUDFLARE_MIGRATION_GUIDE.md (867 lines)
- .github/workflows/sync-cloudflare-dns.yml

Files Modified:
- backend/.env.example (added Cloudflare env vars)
2025-11-18 02:43:48 +00:00
Alexa Amundson
95cb14da9d Add backend test helper and fix compliance event metadata 2025-11-16 23:24:00 -06:00
Alexa Amundson
9e0b06a0a0 Align env templates with shared secret inventory 2025-11-16 19:58:22 -06:00
Alexa Amundson
9b4d04523f add railway secrets automation 2025-11-16 03:48:34 -06:00