mirror of
https://github.com/blackboxprogramming/BlackRoad-Operating-System.git
synced 2026-03-17 03:57:13 -05:00
This commit implements a unified test orchestration system that coordinates
all test suites across the BlackRoad Operating System monorepo, providing
consistent testing behavior between local development and CI/CD environments.
## Core Components
### 1. Test Orchestrator Script (test_all.sh)
- Unified interface to run all test suites
- Smart suite detection with existence checks
- Two operational modes:
* Best-effort: Run all suites, report summary (default)
* Strict mode: Fail-fast on first error (--strict)
- Color-coded, structured output with summary table
- Modular suite functions for easy extension
- Result tracking with pass/fail/skip status and duration
- Verbose mode for detailed test output
Supported test suites:
- Backend (FastAPI + pytest)
- Agents (200+ AI agent ecosystem)
- Operator Engine (GitHub automation)
- Python SDK (pytest)
- TypeScript SDK (Jest)
- Frontend (structure validation)
### 2. GitHub Actions Workflow (.github/workflows/test-orchestrator.yml)
- Runs orchestrator in CI using same script as local dev
- Service containers (Postgres, Redis) for integration tests
- Multi-language runtime setup (Python 3.11, Node 20)
- Dependency caching for faster builds
- Test artifact uploads (coverage, reports)
- Manual workflow dispatch with suite selection
- Coverage reporting for PRs (Codecov integration)
- Automatic PR status comments
### 3. Comprehensive Documentation (TESTING.md)
- Complete testing guide for developers and AI assistants
- Quick start examples
- Suite-by-suite documentation
- Local development setup instructions
- CI/CD integration guide
- Test writing best practices
- Troubleshooting FAQ with common issues and solutions
- Framework-specific examples
## Reusable Templates (.templates/test-orchestrator/)
Created generic templates for use in other repositories:
### Template Files
- test_all.sh.template - Generic orchestrator script
- test-orchestrator.yml.template - Generic CI workflow
- TESTING.md.template - Generic testing documentation
- PROMPTS.md - AI assistant prompts for implementation
- README.md - Template usage guide and customization instructions
### Key Features
- Clear placeholders ({{REPO_NAME}}, {{PROJECT_DESCRIPTION}}, etc.)
- Comprehensive inline comments
- Framework-agnostic design (Python/Node/Go/Rust examples)
- Adaptation guides for different project structures
- AI assistant prompts for Claude, Copilot, ChatGPT
### Use Cases
- Multi-language monorepos
- Microservices architectures
- Data science projects
- Infrastructure projects
- Any project needing unified test orchestration
## Benefits
1. **Consistency**: Same test experience locally and in CI
2. **Discoverability**: New contributors know exactly how to run tests
3. **Maintainability**: Single pattern to learn and maintain
4. **Extensibility**: Easy to add new test suites
5. **CI-Friendly**: Optimized for GitHub Actions
6. **Reusability**: Templates can be copied to any repo
## Usage
Local development:
./test_all.sh # Run all suites
./test_all.sh --strict # Fail-fast mode
./test_all.sh --suite backend # Run specific suite
./test_all.sh --verbose # Detailed output
CI triggers automatically on:
- Push to main, claude/**, copilot/**, codex/** branches
- Pull requests to main
- Manual workflow dispatch
## Migration Notes
This implementation:
- Preserves existing test scripts (scripts/run_backend_tests.sh)
- Works alongside existing CI workflows
- Can be adopted gradually or all at once
- Requires no changes to existing test code
## Future Enhancements
Potential additions:
- Matrix testing across Python/Node versions
- Performance benchmarking suite
- Flaky test detection
- Test result caching
- Slack/Discord notifications
---
Pattern adapted for: BlackRoad Operating System monorepo
Designed for: Maximum reusability across projects
Target audience: Developers, DevOps engineers, AI assistants
257 lines
7.7 KiB
Plaintext
257 lines
7.7 KiB
Plaintext
# GitHub Actions Workflow - Test Orchestrator
|
|
#
|
|
# Adapted from: BlackRoad Operating System Test Orchestrator Pattern
|
|
#
|
|
# TODO: Customize this workflow for your project
|
|
# 1. Update trigger branches
|
|
# 2. Add/remove service containers as needed
|
|
# 3. Adjust cache paths
|
|
# 4. Update environment variables
|
|
# 5. Modify suite options in workflow_dispatch
|
|
|
|
name: Test Orchestrator - All Suites
|
|
|
|
on:
|
|
push:
|
|
branches:
|
|
- "main"
|
|
# TODO: Add your branch patterns here
|
|
# - "develop"
|
|
# - "claude/**"
|
|
# - "feature/**"
|
|
pull_request:
|
|
branches: ["main"]
|
|
workflow_dispatch:
|
|
inputs:
|
|
suite:
|
|
description: 'Specific test suite to run (leave empty for all)'
|
|
required: false
|
|
type: choice
|
|
options:
|
|
- ''
|
|
# TODO: Add your suite names here
|
|
- 'example-suite-1'
|
|
- 'example-suite-2'
|
|
strict_mode:
|
|
description: 'Enable strict mode (fail-fast)'
|
|
required: false
|
|
type: boolean
|
|
default: false
|
|
|
|
jobs:
|
|
orchestrator:
|
|
name: Run Test Orchestrator
|
|
runs-on: ubuntu-latest
|
|
timeout-minutes: 30
|
|
|
|
# TODO: Add/remove service containers as needed
|
|
# services:
|
|
# postgres:
|
|
# image: postgres:15-alpine
|
|
# env:
|
|
# POSTGRES_USER: testuser
|
|
# POSTGRES_PASSWORD: testpass
|
|
# POSTGRES_DB: testdb
|
|
# ports:
|
|
# - 5432:5432
|
|
# options: >-
|
|
# --health-cmd pg_isready
|
|
# --health-interval 10s
|
|
# --health-timeout 5s
|
|
# --health-retries 5
|
|
#
|
|
# redis:
|
|
# image: redis:7-alpine
|
|
# ports:
|
|
# - 6379:6379
|
|
# options: >-
|
|
# --health-cmd "redis-cli ping"
|
|
# --health-interval 10s
|
|
# --health-timeout 5s
|
|
# --health-retries 5
|
|
#
|
|
# mysql:
|
|
# image: mysql:8
|
|
# env:
|
|
# MYSQL_ROOT_PASSWORD: testpass
|
|
# MYSQL_DATABASE: testdb
|
|
# ports:
|
|
# - 3306:3306
|
|
#
|
|
# mongodb:
|
|
# image: mongo:7
|
|
# ports:
|
|
# - 27017:27017
|
|
|
|
steps:
|
|
- name: Checkout repository
|
|
uses: actions/checkout@v4
|
|
|
|
# TODO: Setup language runtimes as needed
|
|
# Uncomment and customize based on your stack
|
|
|
|
# Python setup
|
|
# - name: Setup Python
|
|
# uses: actions/setup-python@v5
|
|
# with:
|
|
# python-version: '3.11' # TODO: Set your Python version
|
|
# cache: 'pip'
|
|
# cache-dependency-path: |
|
|
# requirements.txt
|
|
# # Add more dependency files here
|
|
|
|
# Node.js setup
|
|
# - name: Setup Node.js
|
|
# uses: actions/setup-node@v4
|
|
# with:
|
|
# node-version: '20' # TODO: Set your Node version
|
|
# cache: 'npm' # or 'yarn' or 'pnpm'
|
|
# cache-dependency-path: 'package-lock.json'
|
|
|
|
# Go setup
|
|
# - name: Setup Go
|
|
# uses: actions/setup-go@v5
|
|
# with:
|
|
# go-version: '1.21' # TODO: Set your Go version
|
|
# cache: true
|
|
|
|
# Rust setup
|
|
# - name: Setup Rust
|
|
# uses: actions-rs/toolchain@v1
|
|
# with:
|
|
# toolchain: stable
|
|
# profile: minimal
|
|
|
|
# Java setup
|
|
# - name: Setup Java
|
|
# uses: actions/setup-java@v4
|
|
# with:
|
|
# java-version: '17' # TODO: Set your Java version
|
|
# distribution: 'temurin'
|
|
|
|
# TODO: Install system dependencies if needed
|
|
# - name: Install system dependencies
|
|
# run: |
|
|
# sudo apt-get update
|
|
# sudo apt-get install -y --no-install-recommends \
|
|
# build-essential \
|
|
# libpq-dev \
|
|
# # Add more packages as needed
|
|
|
|
# TODO: Create environment files if needed
|
|
# - name: Create test environment file
|
|
# run: |
|
|
# cat > .env.test << EOF
|
|
# DATABASE_URL=postgresql://testuser:testpass@localhost:5432/testdb
|
|
# REDIS_URL=redis://localhost:6379/0
|
|
# SECRET_KEY=test-secret-key-$(openssl rand -hex 16)
|
|
# ENVIRONMENT=testing
|
|
# # Add more environment variables
|
|
# EOF
|
|
|
|
- name: Make test orchestrator executable
|
|
run: chmod +x test_all.sh
|
|
|
|
- name: Run Test Orchestrator (All Suites)
|
|
if: ${{ github.event.inputs.suite == '' }}
|
|
run: |
|
|
if [[ "${{ github.event.inputs.strict_mode }}" == "true" ]]; then
|
|
./test_all.sh --strict --verbose
|
|
else
|
|
./test_all.sh --verbose
|
|
fi
|
|
|
|
- name: Run Test Orchestrator (Specific Suite)
|
|
if: ${{ github.event.inputs.suite != '' }}
|
|
run: |
|
|
if [[ "${{ github.event.inputs.strict_mode }}" == "true" ]]; then
|
|
./test_all.sh --suite "${{ github.event.inputs.suite }}" --strict --verbose
|
|
else
|
|
./test_all.sh --suite "${{ github.event.inputs.suite }}" --verbose
|
|
fi
|
|
|
|
# TODO: Upload test artifacts
|
|
# Customize paths based on your test output locations
|
|
# - name: Upload test artifacts
|
|
# if: always()
|
|
# uses: actions/upload-artifact@v4
|
|
# with:
|
|
# name: test-results
|
|
# path: |
|
|
# test-results/
|
|
# coverage/
|
|
# *.log
|
|
# retention-days: 7
|
|
# if-no-files-found: ignore
|
|
|
|
- name: Generate test summary
|
|
if: always()
|
|
run: |
|
|
echo "## 🧪 Test Orchestrator Summary" >> $GITHUB_STEP_SUMMARY
|
|
echo "" >> $GITHUB_STEP_SUMMARY
|
|
echo "**Repository:** ${{ github.repository }}" >> $GITHUB_STEP_SUMMARY
|
|
echo "**Branch:** ${{ github.ref_name }}" >> $GITHUB_STEP_SUMMARY
|
|
echo "**Commit:** ${{ github.sha }}" >> $GITHUB_STEP_SUMMARY
|
|
echo "" >> $GITHUB_STEP_SUMMARY
|
|
echo "### Test Results" >> $GITHUB_STEP_SUMMARY
|
|
echo "" >> $GITHUB_STEP_SUMMARY
|
|
echo "Check the job output above for detailed test results." >> $GITHUB_STEP_SUMMARY
|
|
|
|
# TODO: Add optional coverage job
|
|
# coverage:
|
|
# name: Generate Coverage Report
|
|
# runs-on: ubuntu-latest
|
|
# needs: orchestrator
|
|
# if: github.event_name == 'pull_request'
|
|
#
|
|
# steps:
|
|
# - name: Checkout repository
|
|
# uses: actions/checkout@v4
|
|
#
|
|
# # Setup language runtime
|
|
# # Run tests with coverage
|
|
# # Upload to Codecov or similar
|
|
#
|
|
# - name: Upload coverage to Codecov
|
|
# uses: codecov/codecov-action@v4
|
|
# with:
|
|
# file: ./coverage.xml
|
|
# flags: unittests
|
|
# name: coverage
|
|
# fail_ci_if_error: false
|
|
# token: ${{ secrets.CODECOV_TOKEN }}
|
|
|
|
status-check:
|
|
name: Final Status Check
|
|
runs-on: ubuntu-latest
|
|
needs: [orchestrator]
|
|
if: always()
|
|
|
|
steps:
|
|
- name: Check orchestrator status
|
|
run: |
|
|
if [[ "${{ needs.orchestrator.result }}" != "success" ]]; then
|
|
echo "❌ Test orchestrator failed or was cancelled"
|
|
exit 1
|
|
fi
|
|
echo "✅ All test suites passed!"
|
|
|
|
# TODO: Optional PR comment
|
|
# - name: Post status to PR
|
|
# if: github.event_name == 'pull_request' && always()
|
|
# uses: actions/github-script@v7
|
|
# with:
|
|
# script: |
|
|
# const status = '${{ needs.orchestrator.result }}';
|
|
# const icon = status === 'success' ? '✅' : '❌';
|
|
# const message = status === 'success'
|
|
# ? 'All test suites passed!'
|
|
# : 'One or more test suites failed. Check the orchestrator job for details.';
|
|
#
|
|
# github.rest.issues.createComment({
|
|
# issue_number: context.issue.number,
|
|
# owner: context.repo.owner,
|
|
# repo: context.repo.repo,
|
|
# body: `## ${icon} Test Orchestrator\n\n${message}\n\n[View Details](${context.payload.pull_request.html_url}/checks)`
|
|
# });
|