docs: Add comprehensive deployment guide and Railway DNS configuration

## Deployment Guide (DEPLOYMENT_GUIDE.md)
- Complete Railway custom domain setup (17 domains configured)
- Cloudflare DNS configuration instructions (Terraform + manual)
- GitHub secrets configuration for CI/CD
- Service deployment steps and verification checklist
- Architecture overview diagram
- Troubleshooting guide

## Cloudflare DNS (infra/cloudflare/dns-records-railway.tf)
- Updated with actual Railway CNAME targets from custom domains
- 12 subdomains for blackroad.systems
- 5 subdomains for blackroad.io
- All records configured as proxied CNAMEs

## Deploy Keys (infra/secrets/DEPLOY_KEYS.md)
- Generated ed25519 deploy keys for satellite repos
- Setup instructions for enabling deploy keys
- Security notes and key regeneration guide

## Railway Custom Domains Configured
- api.blackroad.systems → wghu19q0.up.railway.app
- core.blackroad.systems → panyy677.up.railway.app
- infra.blackroad.systems → xmky2kqn.up.railway.app
- app.blackroad.io → qydv7efz.up.railway.app
- + 13 more subdomains

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Alexa Louise
2025-11-29 13:49:35 -06:00
parent 591f64cb60
commit a62d44a637
3 changed files with 712 additions and 0 deletions

354
DEPLOYMENT_GUIDE.md Normal file
View File

@@ -0,0 +1,354 @@
# BlackRoad OS - Complete Deployment Guide
> **Last Updated**: 2025-11-29
> **Status**: Ready for deployment
> **Project**: BlackRoad OS (03ce1e43-5086-4255-b2bc-0146c8916f4c)
---
## Quick Start
```bash
# 1. Clone the monorepo
git clone https://github.com/blackboxprogramming/BlackRoad-Operating-System.git
cd BlackRoad-Operating-System
# 2. Link to Railway
railway link -p 03ce1e43-5086-4255-b2bc-0146c8916f4c
# 3. Deploy (when ready)
railway up
```
---
## Part 1: Railway Custom Domains (Configured)
All 17 custom domains have been configured in Railway:
### blackroad.systems (12 subdomains)
| Subdomain | Service | Railway CNAME |
|-----------|---------|---------------|
| api.blackroad.systems | blackroad-os-api-gateway | wghu19q0.up.railway.app |
| core.blackroad.systems | blackroad-os-core | panyy677.up.railway.app |
| infra.blackroad.systems | blackroad-os-infra | xmky2kqn.up.railway.app |
| docs.blackroad.systems | blackroad-os-docs | xz8ar3k7.up.railway.app |
| console.blackroad.systems | blackroad-os-master | alxh5zmf.up.railway.app |
| demo.blackroad.systems | blackroad-os-demo | 828zo5g8.up.railway.app |
| archive.blackroad.systems | blackroad-os-archive | 6339jp4b.up.railway.app |
| research.blackroad.systems | blackroad-os-research | 3rlozcvl.up.railway.app |
| finance.blackroad.systems | blackroad-os-pack-finance | 70iyk36h.up.railway.app |
| legal.blackroad.systems | blackroad-os-pack-legal | 4zx90bq2.up.railway.app |
| lab.blackroad.systems | blackroad-os-pack-research-lab | rf5v4b68.up.railway.app |
| devops.blackroad.systems | blackroad-os-pack-infra-devops | gjsw3tvq.up.railway.app |
### blackroad.io (5 subdomains)
| Subdomain | Service | Railway CNAME |
|-----------|---------|---------------|
| app.blackroad.io | blackroad-os-web | qydv7efz.up.railway.app |
| home.blackroad.io | blackroad-os-home | e5zobwvo.up.railway.app |
| api.blackroad.io | blackroad-os-api | ulwsu2c6.up.railway.app |
| os.blackroad.io | blackroad-os | ay7xf8lw.up.railway.app |
| creator.blackroad.io | blackroad-os-pack-creator-studio | z1imx63q.up.railway.app |
### Already Active (3 domains)
| Domain | Service |
|--------|---------|
| operator.blackroad.systems | blackroad-os-operator |
| beacon.blackroad.systems | blackroad-os-beacon |
| prism.blackroad.systems | blackroad-prism-console |
---
## Part 2: Cloudflare DNS Setup
### Option A: Using Terraform (Recommended)
```bash
cd infra/cloudflare
# Create terraform.tfvars
cat > terraform.tfvars << EOF
cloudflare_api_token = "your-cloudflare-api-token"
zone_id_blackroad_systems = "your-zone-id-for-blackroad-systems"
zone_id_blackroad_io = "your-zone-id-for-blackroad-io"
EOF
# Initialize and apply
terraform init
terraform plan
terraform apply
```
### Option B: Manual DNS Configuration
Add these CNAME records in Cloudflare Dashboard:
#### For blackroad.systems zone:
| Type | Name | Target | Proxy |
|------|------|--------|-------|
| CNAME | api | wghu19q0.up.railway.app | Proxied |
| CNAME | core | panyy677.up.railway.app | Proxied |
| CNAME | infra | xmky2kqn.up.railway.app | Proxied |
| CNAME | docs | xz8ar3k7.up.railway.app | Proxied |
| CNAME | console | alxh5zmf.up.railway.app | Proxied |
| CNAME | demo | 828zo5g8.up.railway.app | Proxied |
| CNAME | archive | 6339jp4b.up.railway.app | Proxied |
| CNAME | research | 3rlozcvl.up.railway.app | Proxied |
| CNAME | finance | 70iyk36h.up.railway.app | Proxied |
| CNAME | legal | 4zx90bq2.up.railway.app | Proxied |
| CNAME | lab | rf5v4b68.up.railway.app | Proxied |
| CNAME | devops | gjsw3tvq.up.railway.app | Proxied |
#### For blackroad.io zone:
| Type | Name | Target | Proxy |
|------|------|--------|-------|
| CNAME | app | qydv7efz.up.railway.app | Proxied |
| CNAME | home | e5zobwvo.up.railway.app | Proxied |
| CNAME | api | ulwsu2c6.up.railway.app | Proxied |
| CNAME | os | ay7xf8lw.up.railway.app | Proxied |
| CNAME | creator | z1imx63q.up.railway.app | Proxied |
### Getting Zone IDs
1. Go to Cloudflare Dashboard
2. Select the domain (e.g., blackroad.systems)
3. Zone ID is in the right sidebar under "API"
---
## Part 3: GitHub Secrets Configuration
Add these secrets to the BlackRoad-Operating-System repository:
### Required Secrets
| Secret Name | Description | How to Get |
|-------------|-------------|------------|
| `RAILWAY_TOKEN` | Railway API token | Railway Dashboard → Account Settings → Tokens |
| `INFRA_DEPLOY_KEY` | SSH key for blackroad-os-infra | Generate with `ssh-keygen` |
| `CORE_DEPLOY_KEY` | SSH key for blackroad-os-core | Generate with `ssh-keygen` |
| `OPERATOR_DEPLOY_KEY` | SSH key for blackroad-os-operator | Generate with `ssh-keygen` |
### Generate Deploy Keys
```bash
# Generate keys for each satellite repo
ssh-keygen -t ed25519 -C "deploy-infra" -f ~/.ssh/blackroad-infra-deploy -N ""
ssh-keygen -t ed25519 -C "deploy-core" -f ~/.ssh/blackroad-core-deploy -N ""
ssh-keygen -t ed25519 -C "deploy-operator" -f ~/.ssh/blackroad-operator-deploy -N ""
# Display public keys (add to repo Settings → Deploy Keys)
cat ~/.ssh/blackroad-infra-deploy.pub
cat ~/.ssh/blackroad-core-deploy.pub
cat ~/.ssh/blackroad-operator-deploy.pub
# Display private keys (add to GitHub Secrets)
cat ~/.ssh/blackroad-infra-deploy
cat ~/.ssh/blackroad-core-deploy
cat ~/.ssh/blackroad-operator-deploy
```
### Adding Deploy Keys to Repos
1. Go to each repo: `github.com/BlackRoad-OS/blackroad-os-{infra,core,operator}`
2. Settings → Deploy Keys → Add deploy key
3. Paste the **public** key
4. Check "Allow write access"
### Adding Secrets to Monorepo
1. Go to `github.com/blackboxprogramming/BlackRoad-Operating-System`
2. Settings → Secrets and variables → Actions
3. Add each **private** key as a secret
---
## Part 4: Service Deployment
### Current Service Status
The Railway services are configured but need application code deployed:
| Service | Status | Next Step |
|---------|--------|-----------|
| blackroad-os-infra | Configured | Deploy AIops service |
| blackroad-os-core | Configured | Deploy Analytics service |
| blackroad-os-operator | Configured | Deploy Codex + webhooks |
| blackroad-os-web | Configured | Deploy web frontend |
| blackroad-os-api | Configured | Deploy API server |
| blackroad-os-docs | Configured | Deploy documentation |
### Deploy Individual Services
```bash
# Deploy to specific service
cd /path/to/satellite-repo
railway link -p 03ce1e43-5086-4255-b2bc-0146c8916f4c
railway up --service blackroad-os-infra
```
### Deploy All Services (from monorepo)
The GitHub Actions workflow will automatically deploy when you push to main:
- Changes in `services/aiops/` → deploys to blackroad-os-infra
- Changes in `services/analytics/` → deploys to blackroad-os-core
- Changes in `services/codex/` → deploys to blackroad-os-operator
---
## Part 5: Verification Checklist
### DNS Verification
```bash
# Check DNS propagation
dig api.blackroad.systems CNAME
dig app.blackroad.io CNAME
```
### Service Health Checks
```bash
# Test health endpoints (after deployment)
curl https://api.blackroad.systems/health
curl https://core.blackroad.systems/health
curl https://infra.blackroad.systems/health
curl https://app.blackroad.io/health
```
### SSL/TLS Verification
All domains should have valid SSL certificates through Cloudflare:
```bash
# Check SSL
curl -vI https://api.blackroad.systems 2>&1 | grep -i "SSL certificate"
```
---
## Part 6: Architecture Overview
```
┌─────────────────────────────────────────────────────────────────────────┐
│ CLOUDFLARE │
│ ┌─────────────────────────┐ ┌─────────────────────────┐ │
│ │ blackroad.systems │ │ blackroad.io │ │
│ │ (Enterprise DNS) │ │ (Consumer DNS) │ │
│ └───────────┬─────────────┘ └───────────┬─────────────┘ │
└──────────────┼──────────────────────────────┼──────────────────────────┘
│ │
▼ ▼
┌─────────────────────────────────────────────────────────────────────────┐
│ RAILWAY │
│ │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ api-gateway │ │ core │ │ infra │ │ web │ │
│ │ (AIops) │ │ (Analytics) │ │ (AIops) │ │ (Frontend) │ │
│ └─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘ │
│ │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ operator │ │ beacon │ │ prism │ │ docs │ │
│ │ (Codex) │ │ (Health) │ │ (Console) │ │ (Docs) │ │
│ └─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘ │
│ │
│ ┌───────────────────────────────────────────────────────────────────┐ │
│ │ PACK SERVICES │ │
│ │ finance │ legal │ research-lab │ creator-studio │ devops │ │
│ └───────────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────────────────┐
│ GITHUB │
│ │
│ ┌─────────────────────────────────────────────────────────────────────┐│
│ │ BlackRoad-Operating-System (Monorepo) ││
│ │ ├── services/aiops/ → syncs to → blackroad-os-infra ││
│ │ ├── services/analytics/ → syncs to → blackroad-os-core ││
│ │ └── services/codex/ → syncs to → blackroad-os-operator ││
│ └─────────────────────────────────────────────────────────────────────┘│
│ │
│ ┌─────────────────────────────────────────────────────────────────────┐│
│ │ BlackRoad-OS Organization (Satellite Repos) ││
│ │ blackroad-os-infra │ blackroad-os-core │ blackroad-os-operator ││
│ │ blackroad-os-web │ blackroad-os-api │ blackroad-os-docs │ ... ││
│ └─────────────────────────────────────────────────────────────────────┘│
└─────────────────────────────────────────────────────────────────────────┘
```
---
## Part 7: Troubleshooting
### DNS Not Resolving
```bash
# Check if CNAME is set correctly
dig +short api.blackroad.systems CNAME
# Expected output: wghu19q0.up.railway.app.
```
### Railway Service Not Found
```bash
# Check service status
railway status
# View logs
railway logs --service blackroad-os-infra
```
### SSL Certificate Issues
- Ensure Cloudflare SSL mode is "Full (strict)"
- Wait up to 24 hours for certificate provisioning
### GitHub Actions Failing
- Check that all secrets are configured
- Verify deploy keys have write access
- Check workflow logs for specific errors
---
## Quick Reference Commands
```bash
# Railway
railway login # Login to Railway
railway link # Link to project
railway status # Check current project
railway up # Deploy
railway logs # View logs
railway domain add <domain> # Add custom domain
# Cloudflare (via Terraform)
cd infra/cloudflare
terraform init
terraform plan
terraform apply
# GitHub
gh secret set RAILWAY_TOKEN # Set secret
gh workflow run deploy-railway.yml # Trigger deployment
```
---
## Support
- **Railway Documentation**: https://docs.railway.app
- **Cloudflare Documentation**: https://developers.cloudflare.com
- **BlackRoad OS Issues**: https://github.com/BlackRoad-OS/blackroad-os/issues
---
*This guide is maintained in the BlackRoad-Operating-System repository.*

View File

@@ -0,0 +1,254 @@
# BlackRoad OS - Cloudflare DNS Configuration (Railway CNAMEs)
# Updated with actual Railway custom domain targets
#
# Usage:
# terraform init
# terraform plan
# terraform apply
#
# Prerequisites:
# - CLOUDFLARE_API_TOKEN environment variable set
# - Zone IDs for each domain
terraform {
required_providers {
cloudflare = {
source = "cloudflare/cloudflare"
version = "~> 4.0"
}
}
}
# Variables
variable "cloudflare_api_token" {
description = "Cloudflare API token"
type = string
sensitive = true
}
variable "zone_id_blackroad_systems" {
description = "Zone ID for blackroad.systems"
type = string
}
variable "zone_id_blackroad_io" {
description = "Zone ID for blackroad.io"
type = string
}
# Provider
provider "cloudflare" {
api_token = var.cloudflare_api_token
}
# =============================================================================
# blackroad.systems DNS Records (Enterprise)
# Railway custom domain CNAMEs - Updated 2025-11-29
# =============================================================================
# API Gateway
resource "cloudflare_record" "api_systems" {
zone_id = var.zone_id_blackroad_systems
name = "api"
value = "wghu19q0.up.railway.app"
type = "CNAME"
proxied = true
comment = "API Gateway - blackroad-os-api-gateway"
}
# Core API
resource "cloudflare_record" "core_systems" {
zone_id = var.zone_id_blackroad_systems
name = "core"
value = "panyy677.up.railway.app"
type = "CNAME"
proxied = true
comment = "Core API - blackroad-os-core"
}
# Infrastructure
resource "cloudflare_record" "infra_systems" {
zone_id = var.zone_id_blackroad_systems
name = "infra"
value = "xmky2kqn.up.railway.app"
type = "CNAME"
proxied = true
comment = "Infrastructure - blackroad-os-infra"
}
# Documentation
resource "cloudflare_record" "docs_systems" {
zone_id = var.zone_id_blackroad_systems
name = "docs"
value = "xz8ar3k7.up.railway.app"
type = "CNAME"
proxied = true
comment = "Documentation - blackroad-os-docs"
}
# Console (Master)
resource "cloudflare_record" "console_systems" {
zone_id = var.zone_id_blackroad_systems
name = "console"
value = "alxh5zmf.up.railway.app"
type = "CNAME"
proxied = true
comment = "Console - blackroad-os-master"
}
# Demo
resource "cloudflare_record" "demo_systems" {
zone_id = var.zone_id_blackroad_systems
name = "demo"
value = "828zo5g8.up.railway.app"
type = "CNAME"
proxied = true
comment = "Demo - blackroad-os-demo"
}
# Archive
resource "cloudflare_record" "archive_systems" {
zone_id = var.zone_id_blackroad_systems
name = "archive"
value = "6339jp4b.up.railway.app"
type = "CNAME"
proxied = true
comment = "Archive - blackroad-os-archive"
}
# Research
resource "cloudflare_record" "research_systems" {
zone_id = var.zone_id_blackroad_systems
name = "research"
value = "3rlozcvl.up.railway.app"
type = "CNAME"
proxied = true
comment = "Research - blackroad-os-research"
}
# Finance Pack
resource "cloudflare_record" "finance_systems" {
zone_id = var.zone_id_blackroad_systems
name = "finance"
value = "70iyk36h.up.railway.app"
type = "CNAME"
proxied = true
comment = "Finance Pack - blackroad-os-pack-finance"
}
# Legal Pack
resource "cloudflare_record" "legal_systems" {
zone_id = var.zone_id_blackroad_systems
name = "legal"
value = "4zx90bq2.up.railway.app"
type = "CNAME"
proxied = true
comment = "Legal Pack - blackroad-os-pack-legal"
}
# Research Lab Pack
resource "cloudflare_record" "lab_systems" {
zone_id = var.zone_id_blackroad_systems
name = "lab"
value = "rf5v4b68.up.railway.app"
type = "CNAME"
proxied = true
comment = "Research Lab - blackroad-os-pack-research-lab"
}
# DevOps Pack
resource "cloudflare_record" "devops_systems" {
zone_id = var.zone_id_blackroad_systems
name = "devops"
value = "gjsw3tvq.up.railway.app"
type = "CNAME"
proxied = true
comment = "DevOps Pack - blackroad-os-pack-infra-devops"
}
# =============================================================================
# blackroad.io DNS Records (Consumer)
# =============================================================================
# Main App
resource "cloudflare_record" "app_io" {
zone_id = var.zone_id_blackroad_io
name = "app"
value = "qydv7efz.up.railway.app"
type = "CNAME"
proxied = true
comment = "Main App - blackroad-os-web"
}
# Home/Landing
resource "cloudflare_record" "home_io" {
zone_id = var.zone_id_blackroad_io
name = "home"
value = "e5zobwvo.up.railway.app"
type = "CNAME"
proxied = true
comment = "Home - blackroad-os-home"
}
# Public API
resource "cloudflare_record" "api_io" {
zone_id = var.zone_id_blackroad_io
name = "api"
value = "ulwsu2c6.up.railway.app"
type = "CNAME"
proxied = true
comment = "Public API - blackroad-os-api"
}
# OS Interface
resource "cloudflare_record" "os_io" {
zone_id = var.zone_id_blackroad_io
name = "os"
value = "ay7xf8lw.up.railway.app"
type = "CNAME"
proxied = true
comment = "OS Interface - blackroad-os"
}
# Creator Studio
resource "cloudflare_record" "creator_io" {
zone_id = var.zone_id_blackroad_io
name = "creator"
value = "z1imx63q.up.railway.app"
type = "CNAME"
proxied = true
comment = "Creator Studio - blackroad-os-pack-creator-studio"
}
# =============================================================================
# Outputs
# =============================================================================
output "blackroad_systems_records" {
value = {
api = "api.blackroad.systems → ${cloudflare_record.api_systems.value}"
core = "core.blackroad.systems → ${cloudflare_record.core_systems.value}"
infra = "infra.blackroad.systems → ${cloudflare_record.infra_systems.value}"
docs = "docs.blackroad.systems → ${cloudflare_record.docs_systems.value}"
console = "console.blackroad.systems → ${cloudflare_record.console_systems.value}"
demo = "demo.blackroad.systems → ${cloudflare_record.demo_systems.value}"
archive = "archive.blackroad.systems → ${cloudflare_record.archive_systems.value}"
research = "research.blackroad.systems → ${cloudflare_record.research_systems.value}"
finance = "finance.blackroad.systems → ${cloudflare_record.finance_systems.value}"
legal = "legal.blackroad.systems → ${cloudflare_record.legal_systems.value}"
lab = "lab.blackroad.systems → ${cloudflare_record.lab_systems.value}"
devops = "devops.blackroad.systems → ${cloudflare_record.devops_systems.value}"
}
description = "All blackroad.systems DNS records"
}
output "blackroad_io_records" {
value = {
app = "app.blackroad.io → ${cloudflare_record.app_io.value}"
home = "home.blackroad.io → ${cloudflare_record.home_io.value}"
api = "api.blackroad.io → ${cloudflare_record.api_io.value}"
os = "os.blackroad.io → ${cloudflare_record.os_io.value}"
creator = "creator.blackroad.io → ${cloudflare_record.creator_io.value}"
}
description = "All blackroad.io DNS records"
}

View File

@@ -0,0 +1,104 @@
# BlackRoad OS - Deploy Keys
> **Generated**: 2025-11-29
> **Location**: ~/.ssh/blackroad-deploy/
## Deploy Keys for Satellite Repo Sync
These keys enable the monorepo to push changes to satellite repositories.
### Key Locations
| Repo | Private Key | Public Key |
|------|-------------|------------|
| blackroad-os-infra | ~/.ssh/blackroad-deploy/infra | ~/.ssh/blackroad-deploy/infra.pub |
| blackroad-os-core | ~/.ssh/blackroad-deploy/core | ~/.ssh/blackroad-deploy/core.pub |
| blackroad-os-operator | ~/.ssh/blackroad-deploy/operator | ~/.ssh/blackroad-deploy/operator.pub |
---
## Public Keys (Add to GitHub Repos)
### blackroad-os-infra
```
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINgPG/wFrB84H6IAIeDH7lKbHfAa3+6l6hUWVUTEpISj deploy-infra@blackroad.systems
```
### blackroad-os-core
```
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPAx1L5spDV+ZdQjgA0beDib+fUa1lqzhKw9sUlfToGG deploy-core@blackroad.systems
```
### blackroad-os-operator
```
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIB8Hm32vsfRJtuAJAGkILePRfAQUMyrL5XdWTehsEOXT deploy-operator@blackroad.systems
```
---
## Setup Instructions
### Step 1: Enable Deploy Keys on Repos
1. Go to each repo in GitHub:
- https://github.com/BlackRoad-OS/blackroad-os-infra/settings
- https://github.com/BlackRoad-OS/blackroad-os-core/settings
- https://github.com/BlackRoad-OS/blackroad-os-operator/settings
2. Navigate to: Settings → Deploy keys
3. Click "Add deploy key"
4. Paste the corresponding public key (above)
5. **Check "Allow write access"**
6. Click "Add key"
### Step 2: Add Private Keys to Monorepo Secrets
1. Go to: https://github.com/blackboxprogramming/BlackRoad-Operating-System/settings/secrets/actions
2. Click "New repository secret"
3. Add each private key:
| Secret Name | Value (from file) |
|-------------|-------------------|
| INFRA_DEPLOY_KEY | Contents of ~/.ssh/blackroad-deploy/infra |
| CORE_DEPLOY_KEY | Contents of ~/.ssh/blackroad-deploy/core |
| OPERATOR_DEPLOY_KEY | Contents of ~/.ssh/blackroad-deploy/operator |
### Step 3: Verify Setup
After adding keys, trigger a sync workflow:
```bash
gh workflow run sync-satellites.yml -R blackboxprogramming/BlackRoad-Operating-System
```
---
## Security Notes
- Private keys are stored locally at `~/.ssh/blackroad-deploy/`
- Never commit private keys to version control
- Keys are ed25519 format (modern, secure)
- Each key is scoped to a single repository
---
## Regenerating Keys
If keys are compromised:
```bash
# Remove old keys
rm -rf ~/.ssh/blackroad-deploy/
# Generate new keys
ssh-keygen -t ed25519 -C "deploy-infra@blackroad.systems" -f ~/.ssh/blackroad-deploy/infra -N ""
ssh-keygen -t ed25519 -C "deploy-core@blackroad.systems" -f ~/.ssh/blackroad-deploy/core -N ""
ssh-keygen -t ed25519 -C "deploy-operator@blackroad.systems" -f ~/.ssh/blackroad-deploy/operator -N ""
# Then update GitHub deploy keys and secrets
```