blackroad-operating-system/ops/DOMAIN_FIX_GUIDE.md

BlackRoad.systems Domain Fix Guide

Problem Summary

The blackroad.systems domain is currently returning HTTP 403 Forbidden with a fallback page ("Status: Nginx API") instead of serving the BlackRoad OS application.

Root Cause

1. Domain Configuration: The domain was configured in "forward" mode to redirect to os.blackroad.systems, but this forwarding is not properly set up at the DNS/registrar level
2. Nginx Misconfiguration: The web server doesn't have a proper server_name directive matching blackroad.systems, causing requests to fall through to a default server block
3. Missing Server Block: No dedicated Nginx configuration exists for the blackroad.systems domain

Solution Overview

We've updated the domain configuration to serve the application directly at blackroad.systems instead of using forwarding. This involves:

1. ✅ Updated DNS configuration in ops/domains.yaml to point blackroad.systems directly to the application server
2. ✅ Created Nginx server blocks in ops/nginx/blackroad.systems.conf for proper request handling
3. 📋 Deployment steps below to apply these changes

---

Changes Made

1. Domain Configuration (ops/domains.yaml)

Before:

- name: "blackroad.systems"
  mode: "forward"
  forward_to: "https://os.blackroad.systems"

After:

- name: "blackroad.systems"
  mode: "dns"
  record:
    type: "CNAME"
    value: "YOUR-PROD-RAILWAY-APP.up.railway.app"

Key Changes:

• Changed from forward mode to dns mode
• Domain now points directly to the application server (Railway)
• www.blackroad.systems redirects to the apex domain
• Establishes blackroad.systems as the canonical domain

2. Nginx Configuration (ops/nginx/blackroad.systems.conf)

Created a complete Nginx server block configuration that:

• Redirects all HTTP traffic to HTTPS
• Redirects www.blackroad.systems to blackroad.systems
• Serves the BlackRoad OS application at the apex domain
• Includes proper SSL/TLS configuration
• Implements security headers
• Provides SPA fallback routing with try_files
• Configures static asset caching
• Includes health check endpoint at /healthz

---

Deployment Steps

Prerequisites

1. Railway hostname: Replace YOUR-PROD-RAILWAY-APP.up.railway.app in ops/domains.yaml with your actual Railway deployment URL
2. SSL certificate: Obtain SSL certificates for blackroad.systems (use Let's Encrypt or your provider)
3. API credentials: Ensure you have GoDaddy API credentials set as environment variables:

   export GODADDY_API_KEY="your_key_here"
   export GODADDY_API_SECRET="your_secret_here"
   

Step 1: Update Domain DNS Records

Run the domain apply script to update DNS records at GoDaddy:

cd /path/to/BlackRoad-Operating-System
python3 ops/scripts/apply_domains.py

Expected output:

Updating DNS record for blackroad.systems: CNAME -> YOUR-PROD-RAILWAY-APP.up.railway.app
Updating forwarding for www.blackroad.systems -> https://blackroad.systems

Verification:

# Wait 5-10 minutes for DNS propagation, then check:
dig +short blackroad.systems
dig +short www.blackroad.systems

Both should resolve to your Railway server IP or CNAME.

Step 2: Deploy Nginx Configuration

Copy the Nginx configuration to your server:

# SSH to your web server
ssh user@your-server

# Copy the config file
sudo cp /path/to/ops/nginx/blackroad.systems.conf /etc/nginx/sites-available/

# Create symlink to enable the site
sudo ln -s /etc/nginx/sites-available/blackroad.systems.conf /etc/nginx/sites-enabled/

# Remove any default server blocks that might conflict
sudo rm /etc/nginx/sites-enabled/default

Update paths in the config:

Edit /etc/nginx/sites-available/blackroad.systems.conf and ensure:

1. Document root: Set root to your actual deployment directory

   root /var/www/blackroad/current;  # Adjust this path
   

2. SSL certificates: Update SSL certificate paths

   ssl_certificate /etc/ssl/certs/blackroad_systems.fullchain.pem;
   ssl_certificate_key /etc/ssl/private/blackroad_systems.key;
   

3. Backend API proxy (if needed): Uncomment and configure the /api/ location block

Step 3: Obtain SSL Certificates

If you don't have SSL certificates yet, use Let's Encrypt:

# Install certbot
sudo apt update
sudo apt install certbot python3-certbot-nginx

# Obtain certificates for both apex and www
sudo certbot --nginx -d blackroad.systems -d www.blackroad.systems

# Certbot will automatically update your Nginx config

Step 4: Test and Reload Nginx

# Test Nginx configuration syntax
sudo nginx -t

# If test passes, reload Nginx
sudo systemctl reload nginx

# Check Nginx status
sudo systemctl status nginx

Step 5: Deploy Your Application Build

Ensure your BlackRoad OS application is built and deployed to the document root:

# Example deployment (adjust to your build process)
cd /var/www/blackroad
git pull origin main
npm install
npm run build
cp -r dist/* current/

# Verify index.html exists
ls -la /var/www/blackroad/current/index.html

Step 6: Verify the Deployment

Test all endpoints:

# Test HTTP -> HTTPS redirect
curl -I http://blackroad.systems
# Should return: HTTP/1.1 301 Moved Permanently
# Location: https://blackroad.systems

# Test www -> apex redirect
curl -I https://www.blackroad.systems
# Should return: HTTP/2 301
# Location: https://blackroad.systems

# Test main site
curl -I https://blackroad.systems
# Should return: HTTP/2 200

# Test health check
curl https://blackroad.systems/healthz
# Should return: ok

# Full response check
curl -s https://blackroad.systems | head -n 20
# Should return your HTML content (not "Status: Nginx API")

Step 7: Purge CDN/Edge Caches (if applicable)

If you're using a CDN or edge cache:

# Cloudflare example
curl -X POST "https://api.cloudflare.com/client/v4/zones/{zone_id}/purge_cache" \
  -H "Authorization: Bearer {token}" \
  -H "Content-Type: application/json" \
  --data '{"purge_everything":true}'

# Fastly example
curl -X POST "https://api.fastly.com/service/{service_id}/purge_all" \
  -H "Fastly-Key: {token}"

---

Troubleshooting

Issue: Still seeing 403 Forbidden

Cause: Nginx config not properly loaded or file permissions issue

Solution:

# Check Nginx error logs
sudo tail -f /var/log/nginx/error.log

# Verify file permissions
sudo chown -R www-data:www-data /var/www/blackroad/current
sudo chmod -R 755 /var/www/blackroad/current

# Restart Nginx
sudo systemctl restart nginx

Issue: DNS not resolving

Cause: DNS propagation delay or incorrect CNAME value

Solution:

# Check current DNS records
dig blackroad.systems

# Force refresh DNS
sudo systemd-resolve --flush-caches

# Wait 5-15 minutes for global DNS propagation

Issue: SSL certificate errors

Cause: Certificate not covering domain or expired

Solution:

# Check certificate validity
sudo openssl x509 -in /etc/ssl/certs/blackroad_systems.fullchain.pem -text -noout

# Renew Let's Encrypt certificate
sudo certbot renew

# Test SSL configuration
openssl s_client -connect blackroad.systems:443 -servername blackroad.systems

Issue: 404 on SPA routes

Cause: try_files directive not working or missing index.html

Solution:

# Verify index.html exists
ls -la /var/www/blackroad/current/index.html

# Test Nginx try_files manually
curl -I https://blackroad.systems/some/spa/route
# Should return 200 and serve index.html

---

Post-Deployment Validation Checklist

• http://blackroad.systems redirects to https://blackroad.systems
• https://www.blackroad.systems redirects to https://blackroad.systems
• https://blackroad.systems returns HTTP 200 and serves the application
• /healthz endpoint returns "ok"
• No "Status: Nginx API" fallback page
• No 403 Forbidden errors
• SSL certificate is valid and trusted
• SPA client-side routes work correctly
• Static assets load with proper caching headers

---

Architecture Decision

Why Apex Domain Instead of Subdomain?

Based on the blackroad-universe/domains/blackroad-systems/DOMAIN_SPEC.md:

• blackroad.systems is defined as the flagship corporate & OS site
• It should serve as the primary entry point for enterprise decision-makers
• The domain positioning emphasizes credibility, trust, and professional authority
• Using a subdomain (os.blackroad.systems) would dilute the brand and reduce SEO authority

Domain Hierarchy

blackroad.systems (APEX)          → Main application (canonical)
├── www.blackroad.systems         → Redirects to apex
└── os.blackroad.systems          → Alternative alias (serves same app)

This establishes blackroad.systems as the canonical domain while maintaining backwards compatibility with os.blackroad.systems.

---

Monitoring

Set up monitoring to ensure the domain stays healthy:

# Uptime monitoring
curl -fsS -m 10 --retry 5 https://blackroad.systems/healthz || echo "Site down!"

# Response time monitoring
curl -w "@curl-format.txt" -o /dev/null -s https://blackroad.systems

# SSL certificate expiration monitoring
echo | openssl s_client -servername blackroad.systems -connect blackroad.systems:443 2>/dev/null | openssl x509 -noout -dates

Consider using monitoring services like:

• Uptime Robot
• Pingdom
• StatusCake
• Datadog

---

Rollback Plan

If issues occur after deployment:

# Revert DNS (run apply_domains.py with old config)
git checkout HEAD~1 ops/domains.yaml
python3 ops/scripts/apply_domains.py

# Disable new Nginx config
sudo rm /etc/nginx/sites-enabled/blackroad.systems.conf
sudo systemctl reload nginx

# Wait for DNS propagation (5-15 minutes)

---

Next Steps

1. Replace Railway placeholder: Update YOUR-PROD-RAILWAY-APP.up.railway.app in domains.yaml with actual hostname
2. Obtain SSL certificates: Use Let's Encrypt or your certificate provider
3. Deploy application build: Ensure latest build is in /var/www/blackroad/current
4. Run apply_domains.py: Update DNS records at GoDaddy
5. Deploy Nginx config: Copy and enable the server block
6. Test thoroughly: Verify all redirects and endpoints work correctly
7. Monitor: Set up uptime and SSL monitoring

---

Contact

For issues with this deployment:

• Review logs: /var/log/nginx/error.log and /var/log/nginx/access.log
• Check domain status: ops/scripts/apply_domains.py
• Verify DNS: dig blackroad.systems
• Test SSL: openssl s_client -connect blackroad.systems:443

---

Status: Ready for deployment Last Updated: 2025-11-17