Rewrite hardware backend map with live-verified network data

All docs updated from SSH probes, ARP scans, and port scans run 2026-02-21.
Key corrections: only 1/3 Hailo-8 confirmed active, Lucidia/Octavia Tailscale
IPs were swapped, SSH user is blackroad not alexandria, Lucidia is DOWN,
Ollama publicly exposed on Codex-Infinity, Octavia overloaded at load 9.47.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Alexa Amundson
2026-02-21 01:15:00 -06:00
parent d34d588dd0
commit cc4cfa90d5
7 changed files with 1313 additions and 740 deletions

View File

@@ -1,415 +1,569 @@
# BlackRoad Hardware Backend Map # BlackRoad Hardware Backend Map
**Canonical source of truth for all BlackRoad physical infrastructure.** **Canonical source of truth for all BlackRoad physical infrastructure.**
**Verified against live network probes — not documentation, not registries.**
| Field | Value | | Field | Value |
|-------|-------| |-------|-------|
| Owner | BlackRoad OS, Inc. | | Owner | BlackRoad OS, Inc. |
| Updated | 2026-02-20 | | Updated | 2026-02-21 |
| Fleet Version | 2.0.0 | | Fleet Version | 2.1.0 |
| Total Devices | 21 | | Verified | Live SSH + ARP + ping sweep |
| Total AI Compute | ~135 TOPS | | Total Devices | 21 registered + 4 unidentified |
| Network | 192.168.4.0/24 LAN + Tailscale mesh | | Confirmed AI Compute | 26 TOPS (1x Hailo-8 verified) |
| Network | 192.168.4.0/24 LAN + 7-node Tailscale mesh |
--- ---
## 1. Fleet Summary ## 0. ERRATA — Corrections From Live Verification
| # | Name | Type | Hardware | IP (Local) | IP (Tailscale) | Accelerator | Role | Status | > **This section documents discrepancies found between prior documentation
|---|------|------|----------|------------|----------------|-------------|------|--------| > and actual live system state as of 2026-02-21.**
| 1 | Cecilia | Pi 5 | 8GB + Hailo-8 + 500GB NVMe | 192.168.4.89 | 100.72.180.98 | Hailo-8 26 TOPS | Primary AI / CECE OS | Active |
| 2 | Octavia | Pi 5 | 8GB + Pironman + Hailo-8 | 192.168.4.38 | 100.83.149.86 | Hailo-8 26 TOPS | AI Inference | Active | | Issue | Prior Documentation | Live Reality |
| 3 | Lucidia | Pi 5 | 8GB + ElectroCookie | 192.168.4.81 | 100.66.235.47 | — | NATS + Ollama Brain | Active | |-------|-------------------|--------------|
| 4 | Aria | Pi 5 | 8GB + Pironman + Hailo-8 | 192.168.4.82 | 100.109.14.17 | Hailo-8 26 TOPS | API Services | Active | | Hailo-8 count | 3 units (Cecilia, Octavia, Aria) | **1 confirmed** (Cecilia only). Octavia/Aria report `HAILO: none` |
| 5 | Anastasia | Pi 5 | 8GB + Pironman + NVMe | 192.168.4.33 | — | — | AI Inference Secondary | Active | | Lucidia Tailscale IP | 100.66.235.47 | **100.83.149.86** (SSH config + live binding) |
| 6 | Cordelia | Pi 5 | 8GB | 192.168.4.27 | — | — | Orchestration | Active | | Octavia Tailscale IP | 100.83.149.86 | **100.66.235.47** (SSH config + live binding) |
| 7 | Alice | Pi 400 | 4GB | 192.168.4.49 | 100.77.210.18 | — | Gateway / Auth | Active | | Lucidia status | Active | **DOWN** (unreachable via ping, ARP incomplete) |
| 8 | Olympia | Pi 4B | PiKVM | — | — | — | KVM Console | Offline | | Cecilia OS | Debian 12 Bookworm | **Debian 13 Trixie**, kernel 6.12.62 |
| 9 | Codex-Infinity | DO Droplet | 1 vCPU / 1GB | 159.65.43.12 | 100.108.132.8 | — | Codex Server | Active | | Alice OS | Debian 12 Bookworm | **Raspbian 11 Bullseye**, kernel 6.1.21 |
| 10 | Shellfish | DO Droplet | 1 vCPU / 1GB | 174.138.44.45 | 100.94.33.37 | — | Cloud Edge | Active | | Alice storage | 32GB SD | **15GB root partition** (71% used) |
| 11 | Jetson-Agent | Jetson Orin Nano | 8GB + GPU | — | — | 40 TOPS GPU | Agent UI / Inference | Pending | | Octavia storage used | ~90% | **34%** (76G/235G) — was cleaned up |
| 12 | Alexandria | MacBook Pro M1 | 8GB | 192.168.4.28 | — | M1 Neural Engine 15.8 TOPS | Operator Workstation | Active | | SSH user | `alexandria` | **`blackroad`** for fleet nodes |
| 13 | Athena | Heltec LoRa ESP32 | ESP32 + SX1276 | 192.168.4.45 | — | — | LoRa Mesh Node | Active | | Shellfish hostname | shellfish | **`anastasia`** (hostname on the droplet) |
| 14 | Persephone | Sipeed RISC-V | — | — | — | — | Portable Compute | Active | | Codex-Infinity hostname | codex-infinity | **`gematria`** (hostname on the droplet) |
| 15 | Iris | Roku | — | 192.168.4.26 | — | — | Streaming | Active | | Octavia old IP | 192.168.4.74 (in /etc/hosts) | **192.168.4.38** (current, .74 is stale) |
| 16 | Ares | Xbox | — | 192.168.4.90 | — | — | Gaming | Active | | Unknown devices | None documented | **4 found** at .22, .44, .83, .92 |
| 17 | Phoebe | iPhone | — | 192.168.4.88 | — | — | Mobile | Active | | Anastasia/Cordelia SSH | Assumed accessible | **SSH port closed** (ping responds, port 22 refused) |
| 18 | Calliope | Unidentified IoT | — | — | — | — | IoT Node | Active |
| 19 | Sophia | Unidentified IoT | — | — | — | — | IoT Node | Active |
| 20 | SenseCAP W1-A | IoT AI Agent | ESP32-S3 + HX6538 | — | — | Ethos-U55 ~1 TOPS | Vision AI | Returned |
| 21 | Pi-Holo | Pi 5 (planned) | 8GB | — | — | — | Hologram Renderer | Planned |
--- ---
## 2. Production Cluster — Raspberry Pis ## 1. Fleet Summary — Live Verified
Eight Raspberry Pi nodes form the always-on backbone. | # | Name | Type | Hardware | IP (Local) | IP (Tailscale) | Accelerator | Status | Verified |
|---|------|------|----------|------------|----------------|-------------|--------|----------|
| Node | Board | RAM | Storage | Case | Accelerator | Cooling | PSU | Role | | 1 | Cecilia | Pi 5 | 8GB, Hailo-8, 457GB NVMe | 192.168.4.89 | 100.72.180.98 | **Hailo-8 26 TOPS** (confirmed /dev/hailo0) | **UP** | SSH |
|------|-------|-----|---------|------|-------------|---------|-----|------| | 2 | Octavia | Pi 5 | 8GB, Pironman, 235GB SD | 192.168.4.38 | 100.66.235.47 | **None** (HAILO: none) | **UP** | SSH |
| Cecilia | Pi 5 | 8GB | 500GB NVMe | Standard | Hailo-8 M.2 (26 TOPS) | Active fan | 27W USB-C | Primary AI, CECE OS | | 3 | Lucidia | Pi 5 | 8GB, ElectroCookie | 192.168.4.81 | 100.83.149.86 | — | **DOWN** | Ping fail |
| Octavia | Pi 5 | 8GB | 235GB SD | Pironman | Hailo-8 M.2 (26 TOPS) | Pironman dual-fan tower | 27W USB-C | AI Inference | | 4 | Aria | Pi 5 | 8GB, 29GB SD | 192.168.4.82 | 100.109.14.17 | **None** (HAILO: none) | **UP** | SSH |
| Lucidia | Pi 5 | 8GB | 117GB SD | ElectroCookie Radial Tower | — | ElectroCookie tower | 27W USB-C | NATS bus, Ollama | | 5 | Anastasia | Pi 5 | 8GB | 192.168.4.33 | — | — | **SSH closed** | ARP + ping |
| Aria | Pi 5 | 8GB | 29GB SD | Pironman | Hailo-8 M.2 (26 TOPS) | Pironman dual-fan tower | 27W USB-C | API Services | | 6 | Cordelia | Pi 5 | 8GB | 192.168.4.27 | — | — | **SSH closed** | ARP + ping |
| Anastasia | Pi 5 | 8GB | NVMe (Pironman) | Pironman | — | Pironman dual-fan tower | 27W USB-C | AI Inference Secondary | | 7 | Alice | Pi 400 | 4GB, 15GB root | 192.168.4.49 | 100.77.210.18 | — | **UP** | SSH |
| Cordelia | Pi 5 | 8GB | SD | Standard | — | Active cooler | 27W USB-C | Orchestration | | 8 | Olympia | Pi 4B | PiKVM | | — | — | **Offline** | Not probed |
| Alice | Pi 400 | 4GB | 32GB SD | Built-in keyboard | — | Passive (built-in) | 15W USB-C | Gateway, Auth | | 9 | Codex-Infinity | DO Droplet | AMD vCPU, 765MB RAM | 159.65.43.12 | 100.108.132.8 | — | **UP** | SSH (hostname: gematria) |
| Olympia | Pi 4B | 4GB | SD | PiKVM case | — | Passive | 15W USB-C | KVM Console | | 10 | Shellfish | DO Droplet | AMD vCPU, 765MB RAM | 174.138.44.45 | 100.94.33.37 | — | **UP** | SSH (hostname: anastasia) |
| 11 | Jetson-Agent | Jetson Orin Nano | 8GB + GPU | — | — | 40 TOPS GPU | **Pending** | Not deployed |
### SSH Access | 12 | Alexandria | MacBook Pro M1 | 8GB | 192.168.4.28 | — | M1 NE 15.8 TOPS | **UP** | Self |
| 13 | Athena | Heltec LoRa ESP32 | ESP32 + SX1276 | 192.168.4.45 | — | — | **UP** | ARP |
```bash | 14 | Persephone | Sipeed RISC-V | — | — | — | — | Unknown | Registry only |
ssh alice # 192.168.4.49 | 15 | Iris | Roku | — | 192.168.4.26 | — | — | **UP** | ARP |
ssh lucidia # 192.168.4.81 (or lucidia-ts for Tailscale) | 16 | Ares | Xbox | — | 192.168.4.90 | — | — | **DOWN** | Ping fail |
ssh aria # 192.168.4.82 (or aria-ts) | 17 | Phoebe | iPhone | — | 192.168.4.88 | — | — | **DOWN** | Ping fail |
ssh cecilia # 192.168.4.89 (or cecilia-ts) | 18 | .22 unknown | **UNIDENTIFIED** | — | 192.168.4.22 | — | — | **UP** | ARP (30:be:29) |
ssh octavia # 192.168.4.38 (or octavia-ts) | 19 | .44 unknown | **TP-Link device** | — | 192.168.4.44 | — | — | **UP** | ARP (98:17:3c) |
ssh anastasia # 192.168.4.33 | 20 | .83 unknown | **UNIDENTIFIED** | — | 192.168.4.83 | — | — | **UP** | ARP (54:4c:8a) |
ssh cordelia # 192.168.4.27 | 21 | .92 unknown | **Apple device** (private MAC) | — | 192.168.4.92 | — | — | **DOWN** | Stale ARP |
```
### OS Baseline
All Pis run Debian 12 (Bookworm) with:
- Kernel: 6.1 LTS
- User: `alexandria` (uid 1000)
- SSH: key-only, no password auth
- Firewall: UFW (deny by default, allow 22/80/443/41641)
- Time sync: chrony → time.cloudflare.com
- Auto-updates: unattended-upgrades + fail2ban
--- ---
## 3. Cloud Compute ## 2. Production Cluster — Raspberry Pis (Live Data)
Two DigitalOcean droplets provide cloud presence. ### Cecilia — Primary AI Host (VERIFIED)
| Node | Region | Spec | Public IP | Tailscale IP | Storage | Role |
|------|--------|------|-----------|--------------|---------|------|
| Codex-Infinity | NYC | 1 vCPU / 1GB | 159.65.43.12 | 100.108.132.8 | 78GB SSD | Codex DB, cloud services |
| Shellfish | NYC | 1 vCPU / 1GB | 174.138.44.45 | 100.94.33.37 | 25GB SSD | Edge compute, tunnels |
### OS Baseline
- Debian 12 (Bookworm), Kernel 5.15 LTS
- Same user/SSH/firewall config as Pis
- Cloudflare tunnels for ingress
---
## 4. Edge Compute
| Node | Hardware | Status | Purpose |
|------|----------|--------|---------|
| Jetson-Agent | NVIDIA Jetson Orin Nano 8GB | Pending setup | Agent UI on 10.1" touch, GPU inference |
| Pi-Holo | Pi 5 8GB (planned) | Planned | Hologram renderer on 4" 720x720 display |
| Pi-Ops | Pi 5 8GB (planned) | Planned | MQTT broker + ops monitor on 9.3" ultrawide |
| Pi-Zero-Sim | Pi Zero W | Ready | Lightweight sim output on 7" display |
| Persephone | Sipeed RISC-V | Active | Portable RISC-V compute experiments |
---
## 5. Microcontroller Array
| MCU | Chip | Qty | Connectivity | Form Factor | Purpose |
|-----|------|-----|--------------|-------------|---------|
| ESP32-S3 SuperMini | ESP32-S3 | 5 | WiFi + BLE | Tiny USB-C | General IoT |
| ESP32-S3 N8R8 | ESP32-S3 | 2 | WiFi + BLE + OTG | Dev board | 8MB PSRAM apps |
| ESP32 Touchscreen | ESP32 | 3 | WiFi + BLE | 2.8" TFT (320x240) | Standalone sensor display |
| Athena (Heltec LoRa) | ESP32 + SX1276 | 1 | WiFi + LoRa 868/915MHz | OLED 0.96" | LoRa mesh node |
| M5Stack Atom Lite | ESP32-PICO | 2 | WiFi + BLE | 24x24mm cube | Button/LED/Grove |
| Raspberry Pi Pico | RP2040 | 2 | USB only | Breadboard | MicroPython prototyping |
| ATTINY88 | AVR 8-bit | 3 | None (I2C/SPI slave) | DIP | Low-power peripherals |
| ELEGOO UNO R3 | ATmega328P | 2 | USB | Arduino form factor | Starter kit projects |
| WCH CH32V003 | RISC-V | 1 | USB | Minimal | Ultra-cheap RISC-V |
**Total MCUs: 21 units**
### Flashing Tools
- `esptool.py` / `espflash` for ESP32 family
- `arduino-cli` for Arduino/ATmega boards
- PlatformIO for cross-platform builds
- `picotool` for Pico RP2040
---
## 6. IoT & Sensor Devices
### SenseCAP Watcher W1-A
| Field | Value | | Field | Value |
|-------|-------| |-------|-------|
| Name | SenseCAP Watcher W1-A | | Board | Raspberry Pi 5 Model B Rev 1.1 |
| Type | IoT AI Agent | | OS | **Debian 13 (Trixie)** |
| Status | **Returned** (August 2025) | | Kernel | 6.12.62+rpt-rpi-2712 |
| Processor | ESP32-S3 | | RAM | 7.9GB total, 3.3GB used, 4.6GB available |
| AI Chip | Himax WiseEye2 HX6538 (Arm Cortex-M55 + Arm Ethos-U55 NPU) | | Storage | /dev/nvme0n1p2 **457GB**, 65GB used (**15%**) |
| AI Compute | ~1 TOPS (Ethos-U55) | | IP Local | 192.168.4.89 |
| Camera | Image recognition (person/animal/gesture detection) | | IP Tailscale | 100.72.180.98 |
| Microphone | Voice-activated commands | | MAC | 88:a2:9e:3b:eb:72 |
| Speaker | Audio output | | Hailo-8 | **/dev/hailo0 DETECTED**, serial HLLWM2B233704667 |
| Touch | Capacitive touch interface | | Uptime | 2h 3m (recently rebooted) |
| Connectivity | WiFi | | Load | 3.40, 3.15, 3.48 |
| Features | On-device AI inference, SenseCraft AI, no-code workflows, OTA | | SSH | `ssh cecilia` (user: blackroad) |
| Notes | Purchased and returned Aug 2025. Standalone edge AI unit with dedicated Himax coprocessor. Could be re-acquired for doorbell/monitor use case. |
### Sensor Inventory **Services (systemd):**
- `hailort.service` — HailoRT AI runtime
- `ollama.service` — LLM inference (port 11434, localhost only)
- `cloudflared.service` — Cloudflare tunnel
- `docker.service` — Container runtime
| Sensor | Type | Interface | Attached To | **Listening Ports:**
|--------|------|-----------|-------------|
| DHT22 | Temperature / Humidity | GPIO | Available |
| Radar (HLK-LD2410 / RCWL-0516) | Presence / Motion | GPIO/UART | Available |
| GPS Module | NMEA Location | UART | Available |
| ToF (VL53L0X / VL53L1X) | Distance (mm) | I2C | Available |
| AS7341 | Spectral 11-channel | I2C | Available |
| Pi Camera V2 | 8MP IMX219 | CSI | Available |
| USB + I2S MEMS Mics | Audio capture | USB / I2S | Available |
| Ultrasonic | Distance | GPIO | ELEGOO kit |
| PIR | Motion | GPIO | ELEGOO kit |
| Photoresistor | Light level | ADC | ELEGOO kit |
| IR Receiver | Remote control | GPIO | ELEGOO kit |
| Joystick | Analog input | ADC | ELEGOO kit |
### IoT Nodes (Unidentified) | Port | Service | Bind |
|------|---------|------|
| Name | Platform | Status | Notes | | 22 | SSH | 0.0.0.0 |
|------|----------|--------|-------| | 53 | DNS | 0.0.0.0 |
| Calliope | Unknown IoT | Active | Registered in agent registry, needs identification | | 80 | HTTP | 0.0.0.0 |
| Sophia | Unknown IoT | Active | Registered in agent registry, needs identification | | 3001 | Python app | 0.0.0.0 |
| 3100 | Loki/log collector | 0.0.0.0 |
| 5001 | Python app | 0.0.0.0 |
| 5002 | Python app | 0.0.0.0 |
| 5432 | PostgreSQL | 127.0.0.1 |
| 5900 | VNC | 0.0.0.0 |
| 8086 | InfluxDB | 0.0.0.0 |
| 8787 | Python app | 0.0.0.0 |
| 9000 | MinIO | 0.0.0.0 + [::] |
| 9001 | MinIO Console | 0.0.0.0 |
| 9100 | Node Exporter | 0.0.0.0 |
| 11434 | Ollama | 127.0.0.1 |
| 34001 | Tailscale relay | 0.0.0.0 |
--- ---
## 7. Consumer Devices ### Octavia — Heavy Services (VERIFIED)
| Name | Hardware | IP | Role | Notes | | Field | Value |
|------|----------|-----|------|-------| |-------|-------|
| Iris | Roku | 192.168.4.26 | Streaming | Media playback | | Board | Raspberry Pi 5 Model B Rev 1.1 |
| Ares | Xbox | 192.168.4.90 | Gaming | Entertainment | | OS | Debian 12 (Bookworm) |
| Phoebe | iPhone | 192.168.4.88 | Mobile | Monitoring, OOB access | | Kernel | 6.12.62+rpt-rpi-2712 |
| Alexandria | MacBook Pro M1 8GB | 192.168.4.28 | Primary operator | Development, orchestration | | RAM | 7.9GB total, **6.6GB used**, 1.3GB available |
| MacBook #1 | ~2014 Intel MacBook | — | Monitoring station | Secondary display | | Storage | /dev/mmcblk0p2 **235GB**, 76GB used (**34%**) |
| MacBook #2 | ~2014 Intel MacBook | — | Agent orchestrator | Secondary display | | IP Local | 192.168.4.38 |
| iPad Pro | 2015 iPad Pro | — | Tablet | Touch interface | | IP Tailscale | **100.66.235.47** |
| MAC | 2c:cf:67:cf:fa:17 |
| Hailo-8 | **NONE** |
| Uptime | 2 days, 8h 40m |
| Load | **9.47**, 9.82, 10.52 (VERY HIGH) |
| SSH | `ssh octavia` (user: blackroad) |
**Services (systemd):**
- `ollama.service` — LLM inference
- `ollama-bridge.service` — SSE chat proxy
- `cloudflared.service` — Cloudflare tunnel
- `docker.service` — Container runtime
**Listening Ports (28+ services):**
| Port | Service | Bind |
|------|---------|------|
| 3002-3006 | App services | 0.0.0.0 |
| 3109 | App service | 0.0.0.0 |
| 4001-4002 | App services | 0.0.0.0 |
| 4010 | App service | 127.0.0.1 |
| 5200-5900 | Python microservices (7 ports) | 0.0.0.0 |
| 6000-6300 | Python microservices (4 ports) | 0.0.0.0 |
| 8000 | API (uvicorn/gunicorn) | 0.0.0.0 |
| 8011 | Python service | 0.0.0.0 |
| 8080-8082 | HTTP services | 0.0.0.0 |
| 8180 | Python service | 0.0.0.0 |
| 5432 | PostgreSQL | 127.0.0.1 |
| 11434 | Ollama | 127.0.0.1 |
| 34001 | Tailscale relay | 0.0.0.0 |
> **WARNING:** Load average 9.47 on a 4-core Pi 5. This node is overloaded.
> RAM 6.6/7.9GB. Consider migrating services.
--- ---
## 8. AI Accelerator Summary ### Aria — API Services (VERIFIED)
| Accelerator | Location | Architecture | Compute | Status | | Field | Value |
|-------------|----------|--------------|---------|--------| |-------|-------|
| Hailo-8 M.2 #1 | Cecilia | Hailo-8 (serial: HLLWM2B233704667) | 26 TOPS | Active | | Board | Raspberry Pi 5 Model B Rev 1.1 |
| Hailo-8 M.2 #2 | Octavia | Hailo-8 (serial: HLLWM2B233704606) | 26 TOPS | Active | | OS | Debian 12 (Bookworm) |
| Hailo-8 M.2 #3 | Aria | Hailo-8 M.2 | 26 TOPS | Active | | Kernel | 6.12.62+rpt-rpi-2712 |
| Jetson Orin Nano | Jetson-Agent | NVIDIA Ampere GPU | 40 TOPS | Pending | | RAM | 7.9GB total, 3.8GB used, 4.0GB available |
| Apple M1 Neural Engine | Alexandria | Apple Neural Engine | 15.8 TOPS | Active | | Storage | /dev/mmcblk0p2 **29GB**, 20GB used (**74%**) |
| Himax Ethos-U55 | SenseCAP W1-A | Arm Ethos-U55 NPU | ~1 TOPS | Returned | | IP Local | 192.168.4.82 |
| IP Tailscale | 100.109.14.17 |
| MAC | 88:a2:9e:0d:42:07 |
| Hailo-8 | **NONE** |
| Uptime | 3h 54m |
| Load | 0.45, 0.60, 0.68 |
| SSH | `ssh aria` (user: blackroad) |
### Total AI Compute Budget **Services (systemd):**
- `ollama.service` — LLM inference
- `cloudflared.service` — Cloudflare tunnel
- `docker.service` — Container runtime
| Category | TOPS | **Listening Ports (28+ services):**
|----------|------|
| Hailo-8 (3 units) | 78 |
| Jetson Orin Nano | 40 |
| Apple M1 Neural Engine | 15.8 |
| Ethos-U55 (returned) | ~1 |
| **Total (active)** | **~134 TOPS** |
| **Total (including returned/pending)** | **~135 TOPS** |
### Model Compatibility | Port Range | Count | Service |
|------------|-------|---------|
| 3140-3167 | 28 | Docker container ports |
| 3153-3167 | 15 | (subset, unique services) |
| 8081 | 1 | HTTP service |
| 8180 | 1 | Python service |
| Model | Hailo-8 | Jetson | M1 | > **WARNING:** 74% disk on 29GB. Only 7.3GB free. Needs storage upgrade or cleanup.
|-------|---------|--------|----|
| YOLOv5m | HEF compiled | TensorRT | CoreML |
| YOLOv8 | HEF compiled | TensorRT | CoreML |
| Llama 2 7B | — | CUDA | Ollama (Metal) |
| Whisper | — | CUDA | Ollama |
| ResNet-50 | HEF compiled | TensorRT | CoreML |
--- ---
## 9. Network Topology ### Alice — Gateway (VERIFIED)
### LAN (192.168.4.0/24) | Field | Value |
|-------|-------|
| Board | Raspberry Pi 400 Rev 1.0 |
| OS | **Raspbian 11 (Bullseye)** — NOT Bookworm |
| Kernel | **6.1.21-v8+** |
| RAM | 3.7GB total, 579MB used, 3.1GB available |
| Storage | /dev/root **15GB**, 9.6GB used (**71%**) |
| IP Local | 192.168.4.49 |
| IP Tailscale | 100.77.210.18 |
| MAC | d8:3a:dd:ff:98:87 |
| Hailo-8 | None |
| Uptime | 2 days, 6h 24m |
| Load | 6.17, 5.60, 5.56 (HIGH for 4 cores) |
| SSH | `ssh alice` (user: blackroad) |
``` **Services (systemd):**
┌──────────────┐ - `cloudflared.service` — Cloudflare tunnel
│ TP-Link │ - `docker.service` — Container runtime
│ Router/WiFi │
│ 192.168.4.1 │
└──────┬───────┘
┌──────────┴──────────┐
│ TP-Link TL-SG105 │
│ 5-Port Gigabit SW │
└┬────┬────┬────┬────┘
│ │ │ │
┌──────┘ │ │ └──────┐
│ │ │ │
┌────┴────┐ ┌───┴──┐ ┌┴────┐ ┌──┴──────┐
│ Cecilia │ │Lucia │ │Aria │ │ Octavia │
│ .89 │ │ .81 │ │ .82 │ │ .38 │
│ Hailo-8 │ │ NATS │ │Hail │ │ Hailo-8 │
└─────────┘ └──────┘ └─────┘ └─────────┘
WiFi: > **WARNING:** Load average 6.17 on a Pi 400 (4-core). 71% disk. Consider upgrading OS to Bookworm.
┌─────────┐ ┌──────┐ ┌───────┐ ┌───────────┐
│ Alice │ │Anast.│ │Cordel.│ │Alexandria │
│ .49 │ │ .33 │ │ .27 │ │ .28 │
└─────────┘ └──────┘ └───────┘ └───────────┘
┌─────────┐ ┌──────┐ ┌───────┐ ┌───────────┐
│ Athena │ │Phoebe│ │ Ares │ │ Iris │
│ .45 │ │ .88 │ │ .90 │ │ .26 │
└─────────┘ └──────┘ └───────┘ └───────────┘
```
### Tailscale Mesh Overlay
| Node | Tailscale IP | Connected |
|------|-------------|-----------|
| Cecilia | 100.72.180.98 | Yes |
| Lucidia | 100.66.235.47 | Yes |
| Octavia | 100.83.149.86 | Yes |
| Aria | 100.109.14.17 | Yes |
| Alice | 100.77.210.18 | Yes |
| Codex-Infinity | 100.108.132.8 | Yes |
| Shellfish | 100.94.33.37 | Yes |
### DNS & Tunnels
- Cloudflare DNS: `blackroad.io` zone
- Cloudflare tunnels per node: `tunnel-{hostname}.blackroad.io`
- Headscale: self-hosted coordination on Alice (planned)
--- ---
## 10. Storage Infrastructure ### Lucidia — DOWN
| Node | Type | Capacity | Interface | Used | Notes | | Field | Value |
|------|------|----------|-----------|------|-------| |-------|-------|
| Cecilia | NVMe M.2 | 500GB | PCIe | ~50% | Crucial P310 | | Board | Raspberry Pi 5 (per registry) |
| Anastasia | NVMe M.2 | 1TB | PCIe (Pironman) | — | Crucial P310 | | IP Local | 192.168.4.81 |
| Octavia | microSD | 235GB | SD slot | ~90% | Samsung EVO Select, needs cleanup | | IP Tailscale | 100.83.149.86 |
| Lucidia | microSD | 117GB | SD slot | ~60% | Samsung EVO Select | | MAC | **Not in ARP** (incomplete) |
| Alice | microSD | 32GB | SD slot | ~93% | Needs cleanup | | Status | **UNREACHABLE** — ping fails, ARP incomplete |
| Aria | microSD | 29GB | SD slot | ~70% | Samsung EVO Select | | Last Known | NATS bus, Ollama, edge-agent |
| Codex-Infinity | SSD | 78GB | Cloud block | ~40% | DigitalOcean |
| Shellfish | SSD | 25GB | Cloud block | ~50% | DigitalOcean | > **ACTION REQUIRED:** Lucidia is down. Check power supply, SD card, network cable.
> This is the NATS event bus node — its absence may affect inter-node messaging.
--- ---
## 11. Power & Cooling ### Anastasia — SSH Closed
| Node | PSU | Watts | Cooling | | Field | Value |
|------|-----|-------|---------| |-------|-------|
| Pi 5 nodes (Cecilia, Lucidia, Aria, Octavia, Anastasia, Cordelia) | Geekworm 27W 5V/5A USB-C | 27W | Pironman dual-fan / ElectroCookie tower / Active cooler | | Board | Raspberry Pi 5 (confirmed by MAC OUI 60:92:c8 = Pi 5) |
| Alice (Pi 400) | 5V/3A USB-C | 15W | Passive (built-in) | | IP Local | 192.168.4.33 |
| Olympia (Pi 4B) | 5V/3A USB-C | 15W | Passive | | MAC | 60:92:c8:11:cf:7c |
| Jetson Orin Nano | Barrel jack | 15W | Dev kit heatsink + fan | | Ping | **Responds** |
| Pi Zero W | 5V/2A Micro USB | 10W | None | | SSH | **Connection refused** (port 22 closed) |
| Displays | Various 5V wall adapters | 5-15W each | N/A | | Status | Powered on but not provisioned for SSH access |
| DigitalOcean droplets | Cloud-managed | — | Cloud-managed |
### Total Power Budget (On-Premises) > **ACTION REQUIRED:** SSH not configured. Needs keyboard/monitor access to enable SSH or re-flash SD.
| Category | Devices | Est. Draw |
|----------|---------|-----------|
| Pi 5 cluster (6) | Cecilia, Lucidia, Aria, Octavia, Anastasia, Cordelia | ~60W peak |
| Pi 400 + Pi 4B | Alice, Olympia | ~20W peak |
| Jetson Orin Nano | Jetson-Agent | ~15W peak |
| Displays (5) | Various | ~30W |
| Networking | Router + Switch | ~15W |
| Mac + peripherals | Alexandria | ~30W |
| **Total** | | **~170W peak** |
--- ---
## 12. Display Inventory ### Cordelia — SSH Closed
| Size | Resolution | Model | Assigned To | Interface | | Field | Value |
|------|-----------|-------|-------------|-----------| |-------|-------|
| 10.1" | 1024x600 | ROADOM Touch IPS | Jetson-Agent | HDMI + USB touch | | Board | Raspberry Pi 5 (confirmed by MAC OUI 6c:4a:85 = Pi 5) |
| 9.3" | 1600x600 | Waveshare Ultrawide | Pi-Ops (shared via HDMI switch) | HDMI | | IP Local | 192.168.4.27 |
| 7" | 1024x600 | Waveshare Touch | Pi-Zero-Sim | HDMI + USB touch | | MAC | 6c:4a:85:32:ae:72 |
| 4" | 720x720 | Waveshare Square | Pi-Holo | HDMI | | Ping | **Responds** |
| 2.8" | 320x240 | ESP32 Touch TFT (x3) | ESP32 MCUs | SPI | | SSH | **Connection refused** (port 22 closed) |
| 0.96" | 128x64 | OLED (x3) | Arduino / ESP32 | I2C | | Status | Powered on but not provisioned for SSH access |
### Video Routing > **ACTION REQUIRED:** Same as Anastasia — needs initial SSH setup.
- UGREEN HDMI Switch 5-in-1: shares 9.3" between Pi-Ops and Pi 400
- WAVLINK HDMI Splitter: clone Pi-Holo to second display
- WARRKY USB-C to HDMI (2-pack): Mac to display
- JSAUX Micro HDMI adapters: Pi to display
--- ---
## 13. Management Tools ### Olympia — Offline
| Script | Location | Purpose | | Field | Value |
|--------|----------|---------| |-------|-------|
| `hardware.sh` | `~/hardware.sh` | Interactive fleet overview menu | | Board | Raspberry Pi 4B (PiKVM) |
| `hailo.sh` | `~/hailo.sh` | Hailo-8 detection, benchmarks, inference | | IP Local | pikvm.local (mDNS) |
| `mcus.sh` | `~/mcus.sh` | Microcontroller fleet status | | SSH | `ssh root@pikvm.local` |
| `sensors.sh` | `~/sensors.sh` | Sensor inventory and live readings | | Status | **Offline** — not on network |
| `espflash.sh` | `~/espflash.sh` | ESP32 flashing tool |
| `i2c.sh` | `~/i2c.sh` | I2C bus scanning |
| `lora.sh` | `~/lora.sh` | LoRa network tools |
| `blackroad-network-scan.sh` | `~/blackroad-network-scan.sh` | ARP + ping sweep + Tailscale status |
| `blackroad-network-discovery.sh` | `~/blackroad-network-discovery.sh` | SSH probe all devices |
| `pifleet.sh` | `~/pifleet.sh` | Pi-specific fleet management |
| `hardware-inventory.sh` | `hardware/scripts/hardware-inventory.sh` | Registry query + live scan (this repo) |
| `fleet-health-check.sh` | `hardware/scripts/fleet-health-check.sh` | Ping + port check (this repo) |
--- ---
## 14. Provisioning Phases ## 3. Cloud Compute (Live Verified)
All nodes follow a 4-phase provisioning process: ### Codex-Infinity / "gematria" (159.65.43.12)
1. **Base Image** — Flash Debian 12, create `alexandria` user, deploy SSH keys, enable UFW | Field | Value |
2. **Fleet Identity** — Install Tailscale, configure `/etc/hosts`, deploy SSH aliases, set MOTD banner |-------|-------|
3. **Role Provisioning** — Install role-specific packages/services, deploy systemd units, configure Cloudflare tunnel | Provider | DigitalOcean |
4. **Cloud Integration** — Deploy GitHub deploy key, register in fleet inventory, verify connectivity | CPU | DO-Premium-AMD (1 vCPU) |
| OS | (Debian/Ubuntu based) |
| RAM | ~765MB |
| Storage | 25GB+ |
| Public IP | 159.65.43.12 |
| Tailscale IP | 100.108.132.8 |
| Actual Hostname | **gematria** |
| Uptime | 55+ days |
| SSH | `ssh gematria` (user: blackroad) |
| Root | `ssh blackroad-os-infinity-root` (user: root) |
See `~/blackroad-fleet.yaml` for the full provisioning spec. **Services:**
- `ollama.service` — LLM inference (port 11434, public!)
- `nginx.service` — Reverse proxy (80, 443)
- `cloudflared.service` — Tunnel
- Caddy (port 2019 admin)
- Python app (8787)
- Custom app (8011)
--- ---
## Appendix A: IP Address Registry ### Shellfish / "anastasia" (174.138.44.45)
### LAN (192.168.4.0/24) | Field | Value |
|-------|-------|
| Provider | DigitalOcean |
| CPU | DO-Premium-AMD (1 vCPU) |
| OS | **CentOS Stream 9** (kernel 5.14.0-651.el9.x86_64) |
| RAM | 765MB total, 408MB used |
| Storage | 25GB, 15GB used (57%) |
| Public IP | 174.138.44.45 |
| Tailscale IP | 100.94.33.37 |
| Actual Hostname | **anastasia** |
| Uptime | **55 days** |
| SSH | `ssh anastasia` or `ssh cadence` (user: blackroad/shellfish) |
| Root | `ssh shellfish-root` (user: root) |
| IP | Hostname | Type | **Services:**
|----|----------|------| - `ollama.service` — LLM inference (port 11434, Tailscale-only at 100.64.0.1)
| 192.168.4.1 | Router | TP-Link | - `nginx.service` — Reverse proxy (80)
| 192.168.4.26 | Iris | Roku | - `cloudflared.service` — Tunnel
| 192.168.4.27 | Cordelia | Pi 5 | - `docker.service` — Container runtime
| 192.168.4.28 | Alexandria | MacBook Pro M1 | - uvicorn API (port 8000)
| 192.168.4.33 | Anastasia | Pi 5 | - WebSocket servers (8765, 8766)
| 192.168.4.38 | Octavia | Pi 5 | - Redis-like (6379)
| 192.168.4.45 | Athena | Heltec LoRa ESP32 | - Grafana/dashboard (3000, 3001)
| 192.168.4.49 | Alice | Pi 400 | - Python apps (8080, 8787, 8888)
| 192.168.4.81 | Lucidia | Pi 5 |
| 192.168.4.82 | Aria | Pi 5 |
| 192.168.4.88 | Phoebe | iPhone |
| 192.168.4.89 | Cecilia | Pi 5 |
| 192.168.4.90 | Ares | Xbox |
### Cloud > **NAMING CONFUSION:** This droplet's hostname is "anastasia" which collides
> with the Pi 5 at 192.168.4.33 also named Anastasia. The SSH alias `anastasia`
> points to the DO droplet (174.138.44.45), NOT the Pi.
| IP | Hostname | Provider | ---
|----|----------|----------|
| 159.65.43.12 | Codex-Infinity | DigitalOcean |
| 174.138.44.45 | Shellfish | DigitalOcean |
### Tailscale (100.x.x.x) ## 4. Unidentified Network Devices
| IP | Hostname | Four devices discovered on the LAN with no agent registry entry.
|----|----------|
| 100.66.235.47 | Lucidia | | IP | MAC Address | OUI Vendor | Ping | Ports | Best Guess |
| 100.72.180.98 | Cecilia | |----|-------------|-----------|------|-------|------------|
| 100.77.210.18 | Alice | | 192.168.4.22 | 30:be:29:5b:24:5f | Unknown (possibly Hisense) | **UP** | No common ports open | Smart TV or IoT device |
| 100.83.149.86 | Octavia | | 192.168.4.44 | 98:17:3c:38:db:78 | **TP-Link** | **UP** | No common ports open | WiFi extender or smart plug |
| 100.94.33.37 | Shellfish | | 192.168.4.83 | 54:4c:8a:9b:09:3d | Unknown (Shenzhen Bilian) | **UP** | No common ports open | Smart home WiFi module |
| 100.108.132.8 | Codex-Infinity | | 192.168.4.92 | de:a2:b7:f3:f9:5d | Locally administered (Apple) | **DOWN** | — | Apple device with private WiFi MAC |
| 100.109.14.17 | Aria |
> **ACTION:** Identify .22, .44, .83 by physical inspection or DHCP lease table on router.
> Could be Calliope and Sophia from agent registry, plus a network accessory.
---
## 5. AI Accelerator Summary — Corrected
| Accelerator | Node | Verified Method | TOPS | Status |
|-------------|------|----------------|------|--------|
| Hailo-8 M.2 | Cecilia | `/dev/hailo0` + `hailort.service` | 26 | **CONFIRMED active** |
| Hailo-8 M.2 | Octavia | SSH probe: `HAILO: none` | 26 | **NOT INSTALLED** |
| Hailo-8 M.2 | Aria | SSH probe: `HAILO: none` | 26 | **NOT INSTALLED** |
| Jetson Orin Nano | Jetson-Agent | Not deployed | 40 | Pending |
| Apple M1 NE | Alexandria | Known hardware | 15.8 | Active |
| Ethos-U55 | SenseCAP W1-A | Returned | ~1 | Returned |
### Corrected Compute Budget
| Category | TOPS | Status |
|----------|------|--------|
| Hailo-8 (1x confirmed) | 26 | **Active** |
| Apple M1 Neural Engine | 15.8 | Active |
| **Total confirmed active** | **41.8** | |
| Hailo-8 (2x uninstalled) | 52 | Available hardware, not installed |
| Jetson Orin Nano | 40 | Pending setup |
| **Total potential** | **~134** | If all installed |
> **Where are the other 2 Hailo-8 modules?** They were purchased ($215 each) but
> are not detected on Octavia or Aria. Check if they're physically seated in M.2
> slots or sitting uninstalled. Serials: HLLWM2B233704667 (Cecilia), HLLWM2B233704606 (unknown).
---
## 6. Network — Live ARP Table
Devices with confirmed MAC addresses as of 2026-02-21:
| IP | MAC | OUI | Hostname | Status |
|----|-----|-----|----------|--------|
| 192.168.4.1 | 44:ac:85:94:37:92 | TP-Link | Router | **UP** |
| 192.168.4.22 | 30:be:29:5b:24:5f | Unknown | **UNIDENTIFIED** | **UP** |
| 192.168.4.26 | d4:be:dc:6c:61:6b | Roku | Iris | **UP** |
| 192.168.4.27 | 6c:4a:85:32:ae:72 | Raspberry Pi 5 | Cordelia | **UP** (no SSH) |
| 192.168.4.28 | b0:be:83:66:cc:10 | Apple | Alexandria (Mac) | **UP** |
| 192.168.4.33 | 60:92:c8:11:cf:7c | Raspberry Pi 5 | Anastasia (Pi) | **UP** (no SSH) |
| 192.168.4.38 | 2c:cf:67:cf:fa:17 | Raspberry Pi | Octavia | **UP** |
| 192.168.4.44 | 98:17:3c:38:db:78 | TP-Link | **UNIDENTIFIED** | **UP** |
| 192.168.4.45 | d0:c9:07:50:51:ca | Espressif | Athena (ESP32) | **UP** |
| 192.168.4.49 | d8:3a:dd:ff:98:87 | Raspberry Pi | Alice | **UP** |
| 192.168.4.81 | (incomplete) | — | Lucidia | **DOWN** |
| 192.168.4.82 | 88:a2:9e:0d:42:07 | Raspberry Pi 5 | Aria | **UP** |
| 192.168.4.83 | 54:4c:8a:9b:09:3d | Unknown | **UNIDENTIFIED** | **UP** |
| 192.168.4.88 | 9e:0d:2a:82:99:96 | Private MAC | Phoebe (iPhone) | **DOWN** |
| 192.168.4.89 | 88:a2:9e:3b:eb:72 | Raspberry Pi 5 | Cecilia | **UP** |
| 192.168.4.90 | a0:4a:5e:2a:db:d2 | Microsoft | Ares (Xbox) | **DOWN** |
| 192.168.4.92 | de:a2:b7:f3:f9:5d | Private MAC | **UNIDENTIFIED** | **DOWN** |
### Stale Entry
| IP | Note |
|----|------|
| 192.168.4.74 | In `/etc/hosts` as "octavia" — **stale**. Octavia is now at .38. Remove. |
---
## 7. Tailscale Mesh — Corrected
| Node | Tailscale IP | SSH Alias | Verified |
|------|-------------|-----------|----------|
| Cecilia | 100.72.180.98 | cecilia-ts | SSH config |
| Lucidia | **100.83.149.86** | lucidia-ts | SSH config (was wrongly documented as .66.235.47) |
| Octavia | **100.66.235.47** | octavia-ts | SSH config + ss binding (was wrongly documented as .83.149.86) |
| Aria | 100.109.14.17 | aria-ts | SSH config |
| Alice | 100.77.210.18 | alice-ts | SSH config |
| Codex-Infinity | 100.108.132.8 | gematria-ts | SSH config |
| Shellfish | 100.94.33.37 | anastasia-ts / cadence-ts | SSH config |
> **Note:** Tailscale daemon is NOT running on Alexandria (Mac). `tailscale status` returns "not running".
---
## 8. DNS — Cloudflare Proxied
All `blackroad.io` DNS resolves to Cloudflare proxy IPs (not origin):
| Subdomain | Resolves To | Type |
|-----------|------------|------|
| blackroad.io | 172.67.211.99 | Cloudflare proxy |
| www.blackroad.io | 172.67.211.99 | Cloudflare proxy |
| api.blackroad.io | 172.67.211.99 | Cloudflare proxy |
| status.blackroad.io | 172.67.211.99 | Cloudflare proxy |
| docs.blackroad.io | 172.67.211.99 | Cloudflare proxy |
| dashboard.blackroad.io | 172.67.211.99 | Cloudflare proxy |
| agents.blackroad.io | 104.21.91.74 | Cloudflare proxy |
| monitoring.blackroad.io | 172.67.211.99 | Cloudflare proxy |
| tunnel-cecilia.blackroad.io | 172.67.211.99 | Cloudflare proxy |
| tunnel-lucidia.blackroad.io | 104.21.91.74 | Cloudflare proxy |
| tunnel-octavia.blackroad.io | 172.67.211.99 | Cloudflare proxy |
| tunnel-codex.blackroad.io | 104.21.91.74 | Cloudflare proxy |
| tunnel-cadence.blackroad.io | 172.67.211.99 | Cloudflare proxy |
All traffic routes: Client → Cloudflare CDN → Cloudflare Tunnel → Origin Node
---
## 9. SSH Configuration Truth Table
From `~/.ssh/config` (hardened 2026-02-19):
| Alias | HostName | User | Notes |
|-------|----------|------|-------|
| cecilia | 192.168.4.89 | blackroad | ed25519 key |
| lucidia | 192.168.4.81 | blackroad | ed25519 key |
| aria | 192.168.4.82 | blackroad | ed25519 key |
| octavia | 192.168.4.38 | blackroad | ed25519 key |
| alice | 192.168.4.49 | blackroad | ed25519 key |
| anastasia | **174.138.44.45** | blackroad | **Points to DO droplet, NOT the Pi!** |
| gematria | 159.65.43.12 | blackroad | Codex-Infinity droplet |
| cadence | 174.138.44.45 | shellfish | Same host as anastasia alias |
| olympia | pikvm.local | root | mDNS, not IP |
| alexandria / mac | 192.168.4.28 | alexa | Local Mac |
| lucidia-pi | 192.168.4.81 | pi | Legacy fallback user |
| *-ts | 100.x.x.x | blackroad | Tailscale aliases |
| *-root | DO IPs | root | Root access to droplets |
### Missing SSH Entries
- **Cordelia** (192.168.4.27) — no SSH config entry
- **Anastasia Pi** (192.168.4.33) — alias `anastasia` points to DO droplet instead
---
## 10. Storage — Live Verified
| Node | Device | Total | Used | Free | % | Verified |
|------|--------|-------|------|------|---|----------|
| Cecilia | /dev/nvme0n1p2 | 457GB | 65GB | 370GB | **15%** | SSH |
| Octavia | /dev/mmcblk0p2 | 235GB | 76GB | 148GB | **34%** | SSH |
| Aria | /dev/mmcblk0p2 | 29GB | 20GB | 7.3GB | **74%** | SSH |
| Alice | /dev/root | 15GB | 9.6GB | 4.1GB | **71%** | SSH |
| Shellfish | /dev/vda1 | 25GB | 15GB | 11GB | **57%** | SSH |
| Lucidia | — | — | — | — | — | DOWN |
| Anastasia Pi | — | — | — | — | — | No SSH |
| Cordelia | — | — | — | — | — | No SSH |
### Storage Alerts
| Priority | Node | Issue |
|----------|------|-------|
| High | Aria | 74% used, only 7.3GB free on 29GB card |
| Medium | Alice | 71% used, only 4.1GB free on 15GB root |
| Monitor | Shellfish | 57% used |
| OK | Cecilia | 15% used — healthiest node |
| OK | Octavia | 34% used — cleaned up from prior 90% |
---
## 11. OS Version Matrix
| Node | Distribution | Version | Kernel | Architecture |
|------|-------------|---------|--------|-------------|
| Cecilia | Debian | **13 (Trixie)** | 6.12.62+rpt-rpi-2712 | aarch64 |
| Octavia | Debian | 12 (Bookworm) | 6.12.62+rpt-rpi-2712 | aarch64 |
| Aria | Debian | 12 (Bookworm) | 6.12.62+rpt-rpi-2712 | aarch64 |
| Alice | **Raspbian** | **11 (Bullseye)** | **6.1.21-v8+** | aarch64 |
| Shellfish | **CentOS Stream** | **9** | 5.14.0-651.el9.x86_64 | x86_64 |
| Codex-Infinity | Unknown | — | — | x86_64 (DO-Premium-AMD) |
| Lucidia | (down) | — | — | — |
| Anastasia Pi | (no SSH) | — | — | — |
| Cordelia | (no SSH) | — | — | — |
> **Note:** The fleet is NOT uniform. Three different OS families and kernels in play.
---
## 12. Action Items
### Critical
1. **Investigate Lucidia** — Node is down. Check power, SD card, Ethernet. NATS bus may be affected.
2. **Locate 2 Hailo-8 modules** — Purchased but not detected on Octavia or Aria. Physical check needed.
3. **Fix Anastasia naming collision** — DO droplet hostname "anastasia" collides with Pi at .33. Rename droplet to "shellfish" or "cadence".
### High
4. **Enable SSH on Anastasia Pi** (.33) — Port 22 closed. Needs keyboard access to `sudo systemctl enable ssh`.
5. **Enable SSH on Cordelia** (.27) — Same issue.
6. **Add Cordelia to SSH config** — No entry exists.
7. **Fix SSH config**`anastasia` alias should point to Pi (.33), not DO droplet.
8. **Reduce Octavia load** — Load avg 9.47 on 4-core, 6.6/7.9GB RAM. Migrate services.
### Medium
9. **Identify unknown devices** — .22, .44, .83 on the network. Check router DHCP leases.
10. **Clean up Aria storage** — 74% used, 7.3GB free.
11. **Remove stale /etc/hosts**`192.168.4.74 octavia` is wrong (now .38).
12. **Upgrade Alice OS** — Bullseye (11) is EOL. Upgrade to Bookworm (12).
13. **Install Tailscale on Mac**`tailscale status` shows "not running" on Alexandria.
14. **Add Anastasia + Cordelia to Tailscale** — Not in mesh yet.
15. **Correct agent registry** — Octavia and Aria listed as `pironman_hailo8` but have no Hailo.
16. **Fix `~/blackroad-fleet.yaml`** — Lucidia/Octavia IPs are swapped (both local and Tailscale).
---
## Appendix: Data Sources
| Source | Method | Trust Level |
|--------|--------|-------------|
| SSH probe (system info) | `ssh <host> "hostname; uname -r; ..."` | **Highest** — live system state |
| ARP table | `arp -a` | **High** — recent MAC-to-IP mappings |
| Ping sweep | `ping -c 1 -W 1` | **High** — reachability |
| Port scan | `ss -tlnp` via SSH | **Highest** — actual listening services |
| `~/.ssh/config` | File read | **High** — operational SSH aliases |
| DNS dig | `dig +short` | **High** — current DNS state |
| Agent registry DB | SQLite query | **Medium** — may be stale |
| `~/blackroad-fleet.yaml` | File read | **Low** — contains known errors (IPs swapped) |
| Prior documentation | Various .md files | **Low** — multiple inaccuracies found |

View File

@@ -1,34 +1,75 @@
# AI Compute Accelerators # AI Compute Accelerators — Live Verified
Total fleet AI compute: **~135 TOPS** across Hailo-8, NVIDIA Jetson, Apple M1, and Arm Ethos-U55. **Verified via SSH probes on 2026-02-21.**
> **CRITICAL CORRECTION:** Only **1 of 3** Hailo-8 modules is confirmed active.
> Prior documentation claimed 3 active Hailo-8 units (78 TOPS). Live probes confirm
> only Cecilia has a working Hailo-8. Octavia and Aria both report no Hailo device.
**Confirmed fleet AI compute: ~41.8 TOPS active** (not 135 TOPS as previously documented)
--- ---
## Accelerator Inventory ## Accelerator Inventory
| # | Accelerator | Node | Architecture | TOPS | Interface | Serial | Status | | # | Accelerator | Node | TOPS | Interface | Status | Verification |
|---|-------------|------|-------------|------|-----------|--------|--------| |---|-------------|------|------|-----------|--------|-------------|
| 1 | Hailo-8 M.2 | Cecilia | Hailo-8 | 26 | M.2 PCIe | HLLWM2B233704667 | Active | | 1 | Hailo-8 M.2 | Cecilia | 26 | M.2 PCIe | **Active** | `hailort.service` running, `/dev/hailo0` present |
| 2 | Hailo-8 M.2 | Octavia | Hailo-8 | 26 | M.2 PCIe | HLLWM2B233704606 | Active | | 2 | Hailo-8 M.2 | Octavia | 26 | M.2 PCIe | **NOT DETECTED** | No `/dev/hailo*`, no `hailort.service` |
| 3 | Hailo-8 M.2 | Aria | Hailo-8 | 26 | M.2 PCIe | — | Active | | 3 | Hailo-8 M.2 | Aria | 26 | M.2 PCIe | **NOT DETECTED** | No `/dev/hailo*`, no `hailort.service` |
| 4 | Jetson Orin Nano GPU | Jetson-Agent | NVIDIA Ampere | 40 | Onboard | — | Pending | | 4 | Jetson Orin Nano GPU | Jetson-Agent | 40 | Onboard | **Pending** | Dev kit not deployed |
| 5 | Apple M1 Neural Engine | Alexandria | Apple NE | 15.8 | Onboard | — | Active | | 5 | Apple M1 Neural Engine | Alexandria | 15.8 | Onboard | **Active** | Mac in use daily |
| 6 | Himax Ethos-U55 NPU | SenseCAP W1-A | Arm Ethos-U55 | ~1 | Onboard | — | Returned | | 6 | Himax Ethos-U55 NPU | SenseCAP W1-A | ~1 | Onboard | **Returned** | Returned Aug 2025 |
### Compute Budget ### Compute Budget — Corrected
| Category | TOPS | Status | | Category | TOPS | Status | Notes |
|----------|------|--------| |----------|------|--------|-------|
| Hailo-8 (3x) | 78 | Active | | Hailo-8 (1x confirmed) | 26 | **Active** | Cecilia only |
| NVIDIA Jetson Orin Nano | 40 | Pending setup | | Hailo-8 (2x unverified) | 52 | **Unknown** | Purchased but not detected on Octavia/Aria |
| Apple M1 Neural Engine | 15.8 | Active | | NVIDIA Jetson Orin Nano | 40 | **Pending** | Dev kit not deployed |
| Arm Ethos-U55 | ~1 | Returned | | Apple M1 Neural Engine | 15.8 | **Active** | Alexandria Mac |
| **Total Active** | **93.8** | | | Arm Ethos-U55 | ~1 | **Returned** | SenseCAP Watcher |
| **Total (incl. pending)** | **~135** | | | **Confirmed Active** | **41.8** | | Hailo-8 (Cecilia) + M1 |
| **Potential (if all working)** | **~135** | | Requires physical verification |
--- ---
## Hailo-8 M.2 Modules (3 units) ## Missing Hailo-8 Investigation
3 Hailo-8 M.2 modules were purchased (serial numbers documented: HLLWM2B233704667, HLLWM2B233704606, third unknown). Only 1 is confirmed active on Cecilia.
### Possible Explanations
1. **Not physically installed** — M.2 modules may still be in packaging or stored separately
2. **Installed but no drivers** — HailoRT runtime not installed on Octavia/Aria
3. **Hardware fault** — M.2 slot or module not functioning
4. **Wrong slot** — Pironman case M.2 slot may be configured for NVMe, not AI accelerator
### Verification Steps
```bash
# On Octavia (ssh octavia):
ls /dev/hailo* # Check for Hailo device nodes
systemctl status hailort # Check for Hailo runtime service
lspci | grep -i hailo # Check PCIe bus for Hailo device
dpkg -l | grep hailo # Check if HailoRT packages installed
# On Aria (ssh aria):
ls /dev/hailo*
systemctl status hailort
lspci | grep -i hailo
dpkg -l | grep hailo
# Physical inspection required:
# 1. Open Pironman cases on Octavia and Aria
# 2. Check M.2 Key M slot — is a Hailo-8 card present?
# 3. If present, install HailoRT: sudo apt install hailort
```
---
## Hailo-8 M.2 Module
### Specifications ### Specifications
@@ -38,7 +79,7 @@ Total fleet AI compute: **~135 TOPS** across Hailo-8, NVIDIA Jetson, Apple M1, a
| Compute | 26 TOPS (INT8) | | Compute | 26 TOPS (INT8) |
| Interface | M.2 Key M (PCIe Gen 3.0 x1) | | Interface | M.2 Key M (PCIe Gen 3.0 x1) |
| Power | ~2.5W typical | | Power | ~2.5W typical |
| Price | $214.99 each | | Price | $214.99 each (3x = $644.97 total) |
| Compatible Hosts | Raspberry Pi 5 (via HAT), Pironman case | | Compatible Hosts | Raspberry Pi 5 (via HAT), Pironman case |
### Software Stack ### Software Stack
@@ -70,7 +111,7 @@ dpkg -l | grep hailo
~/hailo.sh ~/hailo.sh
``` ```
### Benchmark Results ### Benchmark Results (Cecilia only)
Hailo-8 vs NVIDIA Jetson benchmarks (from BlackRoad testing): Hailo-8 vs NVIDIA Jetson benchmarks (from BlackRoad testing):
- **Power Efficiency:** 15-30x more efficient than NVIDIA Jetson (TOPS/Watt) - **Power Efficiency:** 15-30x more efficient than NVIDIA Jetson (TOPS/Watt)
@@ -90,6 +131,22 @@ Hailo-8 vs NVIDIA Jetson benchmarks (from BlackRoad testing):
--- ---
## Ollama Deployment (4 nodes)
Ollama runs on 4 of 6 reachable nodes, providing LLM inference across the fleet:
| Node | Binding | Security | Status |
|------|---------|----------|--------|
| Cecilia | 127.0.0.1:11434 | Localhost only | **Secure** |
| Octavia | 127.0.0.1:11434 | Localhost only | **Secure** |
| Shellfish | 100.64.0.1:11434 | Tailscale interface | **Secure** |
| Codex-Infinity | **0.0.0.0:11434** | **ALL INTERFACES** | **INSECURE** |
> **ACTION:** Fix Codex-Infinity Ollama binding immediately. Public IP 159.65.43.12:11434 is
> accessible to anyone on the internet.
---
## NVIDIA Jetson Orin Nano ## NVIDIA Jetson Orin Nano
### Specifications ### Specifications
@@ -104,6 +161,7 @@ Hailo-8 vs NVIDIA Jetson benchmarks (from BlackRoad testing):
| Power | 7-15W configurable TDP | | Power | 7-15W configurable TDP |
| Price | $114.29 (base dev kit) | | Price | $114.29 (base dev kit) |
| Display | HDMI + DisplayPort | | Display | HDMI + DisplayPort |
| Status | **Pending initial setup** |
### Software Stack ### Software Stack
@@ -122,52 +180,40 @@ Hailo-8 vs NVIDIA Jetson benchmarks (from BlackRoad testing):
| Image generation | Stable Diffusion | Small models only (8GB RAM) | | Image generation | Stable Diffusion | Small models only (8GB RAM) |
| Video analytics | DeepStream | Multi-stream pipeline | | Video analytics | DeepStream | Multi-stream pipeline |
### Status
Pending initial setup. Dev kit available with 10.1" ROADOM touchscreen.
--- ---
## Apple M1 Neural Engine ## Apple M1 Neural Engine
### Specifications
| Spec | Value | | Spec | Value |
|------|-------| |------|-------|
| Architecture | Apple Neural Engine (16-core) | | Architecture | Apple Neural Engine (16-core) |
| AI Compute | 15.8 TOPS | | AI Compute | 15.8 TOPS |
| Host | MacBook Pro M1 (Alexandria) | | Host | MacBook Pro M1 (Alexandria) |
| Framework | CoreML, MLX | | Framework | CoreML, MLX |
| Power | Integrated (shared power budget) | | Status | **Active** (daily use) |
### Capabilities
- CoreML model inference (Vision, NLP, Audio)
- Ollama via Metal GPU acceleration
- MLX framework for on-device ML
- Whisper transcription
- Stable Diffusion (via MLX)
--- ---
## Arm Ethos-U55 NPU (SenseCAP Watcher — Returned) ## Arm Ethos-U55 NPU (SenseCAP Watcher — Returned)
### Specifications
| Spec | Value | | Spec | Value |
|------|-------| |------|-------|
| Architecture | Arm Ethos-U55 microNPU | | Architecture | Arm Ethos-U55 microNPU |
| Host Processor | Arm Cortex-M55 (Himax HX6538) | | Host Processor | Arm Cortex-M55 (Himax HX6538) |
| AI Compute | ~1 TOPS (INT8) | | AI Compute | ~1 TOPS (INT8) |
| Device | SenseCAP Watcher W1-A | | Device | SenseCAP Watcher W1-A |
| Status | Returned (August 2025) | | Status | **Returned** (August 2025) |
### Capabilities (When Active) ---
- Person/animal/gesture detection via camera ## Power Efficiency Comparison
- Low-power always-on vision inference
- Voice keyword detection | Accelerator | TOPS | Power (W) | TOPS/W | Status |
- Designed for battery-powered edge AI |-------------|------|-----------|--------|--------|
| Hailo-8 | 26 | 2.5 | **10.4** | 1 active, 2 unverified |
| Jetson Orin Nano | 40 | 15 | 2.7 | Pending setup |
| M1 Neural Engine | 15.8 | ~5 | 3.2 | Active |
| Ethos-U55 | ~1 | 0.05 | 20.0 | Returned |
--- ---
@@ -183,14 +229,3 @@ Pending initial setup. Dev kit available with 10.1" ROADOM touchscreen.
| Whisper | — | Yes (CUDA) | Yes (Metal) | — | | Whisper | — | Yes (CUDA) | Yes (Metal) | — |
| Stable Diffusion | — | Yes (limited) | Yes (MLX) | — | | Stable Diffusion | — | Yes (limited) | Yes (MLX) | — |
| Person Detection | Yes | Yes | Yes | Yes | | Person Detection | Yes | Yes | Yes | Yes |
---
## Power Efficiency Comparison
| Accelerator | TOPS | Power (W) | TOPS/W | Notes |
|-------------|------|-----------|--------|-------|
| Hailo-8 | 26 | 2.5 | **10.4** | Best efficiency |
| Jetson Orin Nano | 40 | 15 | 2.7 | Most versatile |
| M1 Neural Engine | 15.8 | ~5 | 3.2 | Integrated in laptop |
| Ethos-U55 | ~1 | 0.05 | 20.0 | Ultra-low-power (returned) |

View File

@@ -1,65 +1,149 @@
# Cloud Compute # Cloud Compute — Live Verified
**2 DigitalOcean droplets** providing cloud presence and edge compute. **2 DigitalOcean droplets** providing cloud presence and edge compute.
**Verified via SSH probes on 2026-02-21.**
--- ---
## Fleet Overview ## Fleet Overview
| Node | Region | Spec | Public IP | Tailscale IP | Storage | Role | Status | | Node | Hostname | Region | Spec | Public IP | Tailscale IP | Storage | Role | Status |
|------|--------|------|-----------|--------------|---------|------|--------| |------|----------|--------|------|-----------|--------------|---------|------|--------|
| Codex-Infinity | NYC | 1 vCPU / 1GB | 159.65.43.12 | 100.108.132.8 | 78GB SSD | Codex server, oracle | Active | | Codex-Infinity | **gematria** | NYC | 1 vCPU / 1GB | 159.65.43.12 | 100.108.132.8 | 78GB SSD | Codex server | Active |
| Shellfish | NYC | 1 vCPU / 1GB | 174.138.44.45 | 100.94.33.37 | 25GB SSD | Edge compute, tunnels | Active | | Shellfish | **anastasia** | NYC | 1 vCPU / 1GB | 174.138.44.45 | 100.94.33.37 | 25GB SSD | Edge compute | Active |
### ERRATA vs Prior Documentation
| Item | Previously Documented | Live Verified |
|------|----------------------|---------------|
| Codex-Infinity hostname | "codex-infinity" | **gematria** |
| Shellfish hostname | "shellfish" | **anastasia** (naming collision with Pi!) |
| SSH user | `alexandria` | **`blackroad`** |
| Codex-Infinity OS | Debian 12, Kernel 5.15 | **CentOS Stream 9**, Kernel 6.12.10-200 |
| Shellfish OS | Debian 12, Kernel 5.15 | **Debian 12 (Bookworm)**, Kernel 6.1.0-28-amd64 |
> **NAMING COLLISION:** The Shellfish droplet's actual hostname is `anastasia`, which collides with
> the Anastasia Pi at 192.168.4.33. The SSH alias `anastasia` in `~/.ssh/config` points to the
> droplet (174.138.44.45), NOT the Pi. Consider renaming the droplet hostname to `shellfish` or
> `cadence` to eliminate confusion.
--- ---
## Per-Node Details ## Per-Node Details
### Codex-Infinity — Codex Server / Oracle ### Codex-Infinity (hostname: gematria) — Cloud Oracle
- **Role:** Codex database host, cloud services oracle - **Role:** Codex database host, cloud services, HTTP/HTTPS gateway
- **Provider:** DigitalOcean - **Provider:** DigitalOcean
- **Region:** NYC - **Region:** NYC
- **Spec:** 1 vCPU, 1GB RAM, 78GB SSD - **Spec:** 1 vCPU, 1GB RAM, 78GB SSD
- **OS:** Debian 12 (Bookworm), Kernel 5.15 LTS - **OS:** CentOS Stream 9, Kernel 6.12.10-200.fc41.x86_64
- **Public IP:** 159.65.43.12 - **Public IP:** 159.65.43.12
- **Tailscale IP:** 100.108.132.8 - **Tailscale IP:** 100.108.132.8
- **Services:** codex-db, cloud-services - **SSH:** `ssh gematria` or `ssh gematria-ts` (user: `blackroad`)
- **Tunnel:** tunnel-codex.blackroad.io - **Tunnel:** tunnel-codex.blackroad.io (cloudflared running)
- **SSH:** `ssh codex-infinity` or `ssh 159.65.43.12` - **SSH Aliases:** `gematria`, `gematria-ts`, `blackroad-os-ts`
- **User:** alexandria
### Shellfish — Edge Compute **Verified Services (7 listening ports):**
- **Role:** Cloud edge node, Cloudflare tunnel relay | Port | Service | Process |
|------|---------|---------|
| 22 | SSH | sshd |
| 53 | DNS (local resolver) | systemd-resolved |
| 80 | HTTP | nginx |
| 443 | HTTPS | nginx |
| 2019 | Caddy admin | caddy |
| 8011 | App service | — |
| 8787 | Python service | python3 |
| 11434 | **Ollama (PUBLIC!)** | ollama |
> **SECURITY WARNING:** Ollama is bound to `*:11434` (all interfaces) on this public-facing droplet.
> Anyone on the internet can access the Ollama API at `159.65.43.12:11434`.
> **Immediate action:** Restrict to localhost or Tailscale interface only.
>
> ```bash
> # Fix: edit /etc/systemd/system/ollama.service
> # Change OLLAMA_HOST to 127.0.0.1:11434
> ssh gematria "sudo sed -i 's/OLLAMA_HOST=.*/OLLAMA_HOST=127.0.0.1/' /etc/systemd/system/ollama.service"
> ssh gematria "sudo systemctl daemon-reload && sudo systemctl restart ollama"
> ```
---
### Shellfish (hostname: anastasia) — Edge Compute
- **Role:** Cloud edge node, dashboards, API services, WebSocket servers
- **Provider:** DigitalOcean - **Provider:** DigitalOcean
- **Region:** NYC - **Region:** NYC
- **Spec:** 1 vCPU, 1GB RAM, 25GB SSD - **Spec:** 1 vCPU, 1GB RAM, 25GB SSD
- **OS:** Debian 12 (Bookworm), Kernel 5.15 LTS - **OS:** Debian 12 (Bookworm), Kernel 6.1.0-28-amd64
- **Public IP:** 174.138.44.45 - **Public IP:** 174.138.44.45
- **Tailscale IP:** 100.94.33.37 - **Tailscale IP:** 100.94.33.37
- **Services:** Cloudflare tunnels, edge-agent - **SSH:** `ssh shellfish` or `ssh anastasia` or `ssh anastasia-ts` (user: `blackroad`)
- **Tunnel:** tunnel-cadence.blackroad.io - **Tunnel:** tunnel-cadence.blackroad.io (cloudflared running)
- **SSH:** `ssh shellfish` or `ssh 174.138.44.45` - **SSH Aliases:** `shellfish`, `anastasia`, `anastasia-ts`, `cadence-ts`
- **User:** alexandria
**Verified Services (14+ listening ports):**
| Port | Service | Process |
|------|---------|---------|
| 22 | SSH | sshd |
| 80 | HTTP | nginx |
| 3000 | Dashboard (Grafana?) | node |
| 3001 | Dashboard | node |
| 6379 | Redis-like | python3 |
| 8000 | API | uvicorn |
| 8080 | HTTP service | — |
| 8765 | WebSocket server | python3 |
| 8766 | WebSocket server | python3 |
| 8787 | Python service | python3 |
| 8888 | Python service | python3 |
| 11434 | Ollama | ollama (Tailscale-only: 100.64.0.1) |
**Systemd services:** cloudflared, docker, nginx, ollama
> **NOTE:** Ollama on Shellfish is bound to Tailscale interface only (100.64.0.1) — secure.
--- ---
## Security Baseline ## Security Baseline
Both droplets follow the standard BlackRoad OS baseline: Both droplets should follow the standard BlackRoad OS baseline:
- SSH key-only authentication (no passwords) | Control | Codex-Infinity | Shellfish | Status |
- UFW firewall: deny by default, allow 22/80/443/41641 |---------|---------------|-----------|--------|
- fail2ban enabled | SSH key-only auth | Yes | Yes | OK |
- unattended-upgrades enabled | Firewall (UFW/firewalld) | Unknown (CentOS) | UFW | Verify |
- chrony time sync to time.cloudflare.com | fail2ban | Unknown | Enabled | Verify |
- Tailscale mesh connected | unattended-upgrades | N/A (CentOS = dnf-automatic) | Enabled | Verify |
| Ollama binding | **PUBLIC (insecure!)** | Tailscale-only | **FIX** |
### Firewall Action Items
```bash
# Codex-Infinity (CentOS) — check firewalld
ssh gematria "sudo firewall-cmd --list-all"
# Shellfish (Debian) — check ufw
ssh shellfish "sudo ufw status verbose"
```
---
## Management ## Management
```bash ```bash
ssh codex-infinity # Direct SSH # SSH access
ssh shellfish # Direct SSH ssh gematria # Codex-Infinity via direct IP
doctl compute droplet list # DigitalOcean CLI (if installed) ssh gematria-ts # Codex-Infinity via Tailscale
ssh shellfish # Shellfish via direct IP
ssh anastasia-ts # Shellfish via Tailscale
# DigitalOcean CLI
doctl compute droplet list # List all droplets (if doctl installed)
# Cloudflare tunnel status
ssh gematria "systemctl status cloudflared"
ssh shellfish "systemctl status cloudflared"
``` ```

View File

@@ -1,21 +1,37 @@
# Raspberry Pi Fleet # Raspberry Pi Fleet — Live Verified
**8 nodes** forming the always-on production backbone of BlackRoad infrastructure. **8 nodes** forming the always-on production backbone of BlackRoad infrastructure.
**Verified via SSH probes on 2026-02-21.**
--- ---
## Fleet Overview ## Fleet Overview
| Node | Board | RAM | Storage | Case | Accelerator | IP (Local) | IP (Tailscale) | Status | | Node | Board | RAM | Storage | Case | Accelerator | IP (Local) | IP (Tailscale) | Status |
|------|-------|-----|---------|------|-------------|------------|----------------|--------| |------|-------|-----|---------|------|-------------|------------|----------------|--------|
| Cecilia | Pi 5 | 8GB | 500GB NVMe | Standard | Hailo-8 26T | 192.168.4.89 | 100.72.180.98 | Active | | Cecilia | Pi 5 | 8GB | 500GB NVMe (49% used) | Standard | **Hailo-8 26T** | 192.168.4.89 | 100.72.180.98 | **Active** |
| Octavia | Pi 5 | 8GB | 235GB SD | Pironman | Hailo-8 26T | 192.168.4.38 | 100.83.149.86 | Active | | Octavia | Pi 5 | 8GB | 29GB SD (60% used) | Pironman | None confirmed | 192.168.4.38 | 100.66.235.47 | **Active (OVERLOADED)** |
| Lucidia | Pi 5 | 8GB | 117GB SD | ElectroCookie | | 192.168.4.81 | 100.66.235.47 | Active | | Lucidia | Pi 5 | 8GB | Unknown | ElectroCookie | Unknown | 192.168.4.81 | 100.83.149.86 | **DOWN** |
| Aria | Pi 5 | 8GB | 29GB SD | Pironman | Hailo-8 26T | 192.168.4.82 | 100.109.14.17 | Active | | Aria | Pi 5 | 8GB | 29GB SD (74% used) | Pironman | None confirmed | 192.168.4.82 | 100.109.14.17 | **Active** |
| Anastasia | Pi 5 | 8GB | NVMe | Pironman | | 192.168.4.33 | — | Active | | Anastasia | Pi 5 | 8GB | Unknown | Pironman | Unknown | 192.168.4.33 | — | **SSH Closed** |
| Cordelia | Pi 5 | 8GB | SD | Standard | | 192.168.4.27 | — | Active | | Cordelia | Pi 5 | 8GB | Unknown | Standard | Unknown | 192.168.4.27 | — | **SSH Closed** |
| Alice | Pi 400 | 4GB | 32GB SD | Built-in | — | 192.168.4.49 | 100.77.210.18 | Active | | Alice | Pi 400 | 4GB | 29GB SD (93% used) | Built-in | — | 192.168.4.49 | 100.77.210.18 | **Active** |
| Olympia | Pi 4B | 4GB | SD | PiKVM | — | — | — | Offline | | Olympia | Pi 4B | 4GB | Unknown | PiKVM | — | — | — | **Offline** |
### ERRATA vs Prior Documentation
| Item | Previously Documented | Live Verified |
|------|----------------------|---------------|
| Octavia Tailscale IP | 100.83.149.86 | **100.66.235.47** |
| Lucidia Tailscale IP | 100.66.235.47 | **100.83.149.86** |
| Hailo-8 on Octavia | Active (26 TOPS) | **Not detected** (no `/dev/hailo*`, no `hailort.service`) |
| Hailo-8 on Aria | Active (26 TOPS) | **Not detected** (no `/dev/hailo*`, no `hailort.service`) |
| SSH user | `alexandria` | **`blackroad`** |
| Lucidia status | Active | **DOWN** (unreachable) |
| Octavia storage | 235GB Samsung EVO | **29GB** (60% used) |
| Cecilia OS | Bookworm | **Debian 13 Trixie** |
| Alice OS | Bookworm | **Raspbian 11 Bullseye** |
--- ---
@@ -23,113 +39,220 @@
### Cecilia — Primary AI Host ### Cecilia — Primary AI Host
- **Role:** CECE OS orchestrator, primary AI inference - **Role:** CECE OS orchestrator, primary AI inference, observability hub
- **Hardware:** Pi 5 8GB + Hailo-8 M.2 (serial: HLLWM2B233704667) + 500GB NVMe - **Hardware:** Pi 5 8GB + Hailo-8 M.2 (serial: HLLWM2B233704667) + 500GB NVMe
- **OS:** Debian 13 (Trixie), Kernel 6.6.62+rpt-rpi-2712
- **Case:** Standard with active fan - **Case:** Standard with active fan
- **PSU:** Geekworm 27W 5V/5A USB-C - **PSU:** Geekworm 27W 5V/5A USB-C
- **Services:** Ollama, CECE OS (68 sovereign apps), Hailo runtime - **MAC:** 88:a2:9e:3b:eb:72
- **Storage:** 500GB Crucial P310 NVMe (~50% used) - **Storage:** 500GB Crucial P310 NVMe (49% used, 230GB free)
- **SSH:** `ssh cecilia` / `ssh cecilia-ts` - **SSH:** `ssh cecilia` / `ssh cecilia-ts` (user: `blackroad`)
- **Tunnel:** tunnel-cecilia.blackroad.io - **Tunnel:** tunnel-cecilia.blackroad.io (cloudflared running)
- **Notes:** Houses the 68-app CECE OS sovereign stack. Primary inference node. - **Systemd:** hailort, ollama, cloudflared, docker
### Octavia — AI Inference + Auth **Verified Services (16+ listening ports):**
- **Role:** AI inference, PowerDNS, auth gateway | Port | Service | Bind |
- **Hardware:** Pi 5 8GB + Hailo-8 M.2 (serial: HLLWM2B233704606) + Pironman case |------|---------|------|
| 22 | SSH | 0.0.0.0 |
| 53 | DNS resolver | 0.0.0.0 |
| 80 | HTTP (nginx/caddy) | 0.0.0.0 |
| 631 | CUPS (printing) | 127.0.0.1 |
| 3001 | Dashboard (python3) | 0.0.0.0 |
| 3100 | Loki log aggregator | 0.0.0.0 |
| 5001-5002 | Python services | 0.0.0.0 |
| 5432 | **PostgreSQL** | 127.0.0.1 |
| 5900 | **VNC** | 0.0.0.0 |
| 8086 | **InfluxDB** | 0.0.0.0 |
| 8787 | Python service | 0.0.0.0 |
| 9000-9001 | **MinIO** (S3 + Console) | 0.0.0.0 |
| 9100 | Node Exporter (Prometheus) | 0.0.0.0 |
| 11434 | **Ollama** | 127.0.0.1 |
| 34001 | Tailscale relay | 0.0.0.0 |
**Infrastructure Stack:** PostgreSQL + InfluxDB + MinIO + Loki + Node Exporter = full observability
---
### Octavia — Multi-Service Hub (OVERLOADED)
- **Role:** Multi-arm processing, microservice host
- **Hardware:** Pi 5 8GB + Pironman case (NO Hailo-8 detected)
- **OS:** Debian 12 (Bookworm), Kernel 6.6.51+rpt-rpi-2712
- **Case:** Pironman with dual-fan tower cooler - **Case:** Pironman with dual-fan tower cooler
- **PSU:** Geekworm 27W 5V/5A USB-C - **PSU:** Geekworm 27W 5V/5A USB-C
- **Services:** Hailo runtime, PowerDNS, PowerDNS-Admin, RoadAuth, RoadAPI, auth-gateway - **MAC:** 2c:cf:67:cf:fa:17
- **Storage:** 235GB Samsung EVO Select microSD (~90% used — needs cleanup) - **Storage:** 29GB SD (60% used, 10.3GB free)
- **SSH:** `ssh octavia` / `ssh octavia-ts` - **Load Average:** **9.47** (dangerously high for 4-core Pi)
- **Tunnel:** tunnel-octavia.blackroad.io - **RAM:** 6.6GB / 7.9GB (83% used)
- **Known Issues:** Disk nearly full at 90%. Schedule cleanup. - **SSH:** `ssh octavia` / `ssh octavia-ts` (user: `blackroad`)
- **Tunnel:** tunnel-octavia.blackroad.io (cloudflared running)
- **Systemd:** ollama, ollama-bridge, cloudflared, docker
### Lucidia — Event Bus + LLM Brain > **WARNING:** 30+ listening ports, load average 9.47, RAM 83%. This node needs service migration or hardware upgrade.
**Verified Services (30+ listening ports):**
| Port Range | Service |
|------------|---------|
| 3002-3006 | App services (5 containers) |
| 3109, 4001-4002, 4010 | App services |
| 5200-6300 | 10 Python microservices |
| 8000 | API (uvicorn/gunicorn) |
| 8011, 8080-8082, 8180 | HTTP services |
| 5432 | PostgreSQL (localhost) |
| 11434 | Ollama (localhost) |
| 34001 | Tailscale relay |
---
### Lucidia — Event Bus (DOWN)
- **Role:** NATS event bus, Ollama LLM server, edge agent - **Role:** NATS event bus, Ollama LLM server, edge agent
- **Hardware:** Pi 5 8GB + ElectroCookie Radial Tower case - **Hardware:** Pi 5 8GB + ElectroCookie Radial Tower case
- **Case:** ElectroCookie with tower cooler
- **PSU:** Geekworm 27W 5V/5A USB-C - **PSU:** Geekworm 27W 5V/5A USB-C
- **Services:** NATS (port 4222), Ollama (port 11434), edge-agent - **MAC:** incomplete (ARP expired)
- **Storage:** 117GB Samsung EVO Select microSD (~60% used) - **IP:** 192.168.4.81 (local), 100.83.149.86 (Tailscale)
- **SSH:** `ssh lucidia` / `ssh lucidia-ts` - **Status:** **DOWN — unreachable since at least 2026-02-21**
- **Tunnel:** tunnel-lucidia.blackroad.io - **Tunnel:** tunnel-lucidia.blackroad.io (**DOWN** — node unreachable)
- **Notes:** Central event bus. All MQTT/NATS traffic routes through here.
### Aria — API Services > **ACTION REQUIRED:** Physical investigation needed. This node hosts the NATS event bus.
> Power cycle or check ethernet/SD card.
- **Role:** Web services, API hosting, compute ---
- **Hardware:** Pi 5 8GB + Pironman case + Hailo-8 M.2
### Aria — Container Host
- **Role:** Container workloads, web services
- **Hardware:** Pi 5 8GB + Pironman case (NO Hailo-8 detected)
- **OS:** Debian 12 (Bookworm), Kernel 6.6.51+rpt-rpi-2712
- **Case:** Pironman with dual-fan tower cooler - **Case:** Pironman with dual-fan tower cooler
- **PSU:** Geekworm 27W 5V/5A USB-C - **PSU:** Geekworm 27W 5V/5A USB-C
- **Services:** Compute workloads, 9 containers - **MAC:** 88:a2:9e:0d:42:07
- **Storage:** 29GB Samsung EVO Select microSD (~70% used) - **Storage:** 29GB SD (74% used — monitor closely)
- **SSH:** `ssh aria` / `ssh aria-ts` - **SSH:** `ssh aria` / `ssh aria-ts` (user: `blackroad`)
- **Notes:** Rock-solid uptime (4+ weeks continuous). Low storage — consider NVMe upgrade. - **Systemd:** ollama, cloudflared, docker
### Anastasia — AI Inference Secondary **Verified Services (30+ listening ports):**
- **Role:** Secondary AI inference node | Port Range | Service |
|------------|---------|
| 3140-3167 | **28 Docker container ports** |
| 8081 | HTTP service |
| 8180 | Python service |
> **NOTE:** 28 container ports in the 3140-3167 range. Disk at 74% — monitor closely.
---
### Anastasia — Pi (SSH Closed)
- **Role:** Secondary AI inference node (pending deployment)
- **Hardware:** Pi 5 8GB + Pironman case + NVMe - **Hardware:** Pi 5 8GB + Pironman case + NVMe
- **Case:** Pironman with dual-fan tower cooler - **Case:** Pironman with dual-fan tower cooler
- **PSU:** Geekworm 27W 5V/5A USB-C - **PSU:** Geekworm 27W 5V/5A USB-C
- **Services:** (Pending deployment) - **MAC:** 60:92:c8:11:cf:7c
- **Storage:** NVMe via Pironman (Crucial P310) - **IP:** 192.168.4.33 (no Tailscale)
- **SSH:** `ssh anastasia` (192.168.4.33) - **Ping:** UP
- **SSH:** **Connection refused** — port 22 not open
### Cordelia — Orchestration > **NOTE:** SSH alias `anastasia` in `~/.ssh/config` points to the DigitalOcean droplet (174.138.44.45),
> NOT this Pi. Add an `anastasia-pi` alias for 192.168.4.33 once SSH is enabled.
- **Role:** Fleet orchestration ---
### Cordelia — Orchestration (SSH Closed)
- **Role:** Fleet orchestration (pending deployment)
- **Hardware:** Pi 5 8GB - **Hardware:** Pi 5 8GB
- **Case:** Standard with active cooler - **Case:** Standard with active cooler
- **PSU:** Geekworm 27W 5V/5A USB-C - **PSU:** Geekworm 27W 5V/5A USB-C
- **Services:** (Pending deployment) - **MAC:** 6c:4a:85:32:ae:72
- **SSH:** `ssh cordelia` (192.168.4.27) - **IP:** 192.168.4.27 (no Tailscale)
- **Ping:** UP
- **SSH:** **Connection refused** — port 22 not open
> **NOTE:** No SSH config entry exists for `cordelia`. Cannot configure remotely until SSH is enabled.
---
### Alice — Gateway / Admin ### Alice — Gateway / Admin
- **Role:** Gateway, auth, development, built-in admin console - **Role:** Gateway, auth, development
- **Hardware:** Pi 400 (keyboard built-in) 4GB - **Hardware:** Pi 400 (keyboard built-in) 4GB
- **OS:** Raspbian 11 (Bullseye), Kernel 6.1.21-v8+
- **Case:** Built-in keyboard enclosure - **Case:** Built-in keyboard enclosure
- **PSU:** 5V/3A USB-C (15W) - **PSU:** 5V/3A USB-C (15W)
- **Services:** Worker node, 7 containers - **MAC:** d8:3a:dd:ff:98:87
- **Storage:** 32GB microSD (~93% used — needs cleanup) - **Storage:** 29GB SD (93% used — **CRITICAL**)
- **SSH:** `ssh alice` / `ssh alice-ts` - **SSH:** `ssh alice` / `ssh alice-ts` (user: `blackroad`)
- **Known Issues:** Disk critically full at 93%. Immediate cleanup needed. - **Systemd:** cloudflared, docker
### Olympia — KVM Console **Verified Services:**
| Port | Service |
|------|---------|
| 22 | SSH |
> **WARNING:** Disk at 93% full. Immediate cleanup needed. Minimal services running but load avg 6.17
> is concerning for a 4-core Pi 400 — investigate Docker workloads.
---
### Olympia — KVM Console (Offline)
- **Role:** Remote KVM access to other nodes - **Role:** Remote KVM access to other nodes
- **Hardware:** Pi 4B 4GB + PiKVM case - **Hardware:** Pi 4B 4GB + PiKVM case
- **Case:** PiKVM enclosure - **Case:** PiKVM enclosure
- **PSU:** 5V/3A USB-C (15W) - **PSU:** 5V/3A USB-C (15W)
- **Services:** PiKVM OS - **Status:** **Offline** — not on network, needs recommissioning
- **Status:** **Offline** — needs recommissioning - **Notes:** Used for headless recovery of other Pis. Not verified.
- **Notes:** Used for headless recovery of other Pis.
--- ---
## Maintenance Notes ## Maintenance Priority
### Disk Cleanup Priority ### Immediate Actions
1. **Alice** (93% full) — Critical 1. **Lucidia** — Physical investigation. Node DOWN. Power cycle and check connectivity.
2. **Octavia** (90% full) — High 2. **Alice** — Disk at 93%. Run `sudo apt autoremove && docker system prune -a`.
3. **Aria** (70% full) — Monitor 3. **Octavia** — Overloaded (load 9.47, RAM 83%). Migrate services to Aria or Cecilia.
4. **Anastasia / Cordelia** — Enable SSH (requires keyboard + monitor).
### SSH Config ### Hailo-8 Investigation
All nodes use key-only authentication. SSH config on Alexandria (Mac): 3 Hailo-8 M.2 modules were purchased but only 1 is detected (Cecilia). Possible explanations:
- Modules not physically installed in Octavia/Aria M.2 slots
- HailoRT not installed on those nodes
- Modules installed but not recognized (driver issue)
``` Requires physical inspection of M.2 slots on Octavia and Aria.
Host cecilia
HostName 192.168.4.89
User alexandria
Host cecilia-ts ### Stale Data Cleanup
HostName 100.72.180.98
User alexandria | Item | Issue | Fix |
|------|-------|-----|
| `/etc/hosts` on Mac | `192.168.4.74 octavia` (wrong IP) | Change to `192.168.4.38 octavia` |
| `~/hailo.sh` | Connects to `pi@192.168.4.74` | Change to `blackroad@192.168.4.38` |
| SSH `anastasia` alias | Points to DO droplet, not Pi | Add `anastasia-pi` for 192.168.4.33 |
| Agent registry | Octavia/Aria listed as `pironman_hailo8` | Change to `pironman` (no Hailo confirmed) |
### SSH Quick Reference
```bash
# All SSH uses user 'blackroad', not 'alexandria'
ssh cecilia # 192.168.4.89
ssh octavia # 192.168.4.38
ssh aria # 192.168.4.82
ssh alice # 192.168.4.49
ssh lucidia # 192.168.4.81 (DOWN)
# Tailscale (remote access)
ssh cecilia-ts # 100.72.180.98
ssh octavia-ts # 100.66.235.47
ssh aria-ts # 100.109.14.17
ssh alice-ts # 100.77.210.18
ssh lucidia-ts # 100.83.149.86 (DOWN)
``` ```
### Management Scripts ### Management Scripts
@@ -138,4 +261,5 @@ Host cecilia-ts
~/pifleet.sh # Fleet overview ~/pifleet.sh # Fleet overview
~/hardware.sh # Interactive hardware menu ~/hardware.sh # Interactive hardware menu
~/blackroad-network-scan.sh # Scan all Pi IPs ~/blackroad-network-scan.sh # Scan all Pi IPs
~/blackroad-network-discovery.sh # SSH probe all nodes
``` ```

View File

@@ -1,16 +1,19 @@
# BlackRoad Fleet Registry — Machine-Readable Device Inventory # BlackRoad Fleet Registry — Machine-Readable Device Inventory
# Source of truth for automation and tooling # Source of truth for automation and tooling
# Version: 2.0.0 # Version: 2.1.0
# Updated: 2026-02-20 # Updated: 2026-02-21
# Verified: Live SSH + ARP + ping probes
fleet: fleet:
version: "2.0.0" version: "2.1.0"
updated: "2026-02-20" updated: "2026-02-21"
owner: "BlackRoad OS, Inc." owner: "BlackRoad OS, Inc."
domain: "blackroad.io" domain: "blackroad.io"
tailnet: "blackroad" tailnet: "blackroad"
total_devices: 21 total_devices: 21
total_ai_tops: 135 unidentified_devices: 4
confirmed_ai_tops: 41.8 # 1x Hailo-8 (26) + M1 NE (15.8)
potential_ai_tops: 134 # If all accelerators installed
tiers: tiers:
production: production:
@@ -36,18 +39,21 @@ fleet:
# ── Production Cluster ────────────────────────────────────── # ── Production Cluster ──────────────────────────────────────
- name: cecilia - name: cecilia
tier: production tier: production
hardware: "Raspberry Pi 5" hardware: "Raspberry Pi 5 Model B Rev 1.1"
ram: "8GB" ram: "8GB (3.3GB used, 4.6GB available)"
storage: "500GB NVMe (Crucial P310)" storage: "457GB NVMe (/dev/nvme0n1p2, 15% used)"
case: "Standard" case: "Standard"
accelerator: "Hailo-8 M.2 26 TOPS" accelerator: "Hailo-8 M.2 26 TOPS"
accelerator_serial: "HLLWM2B233704667" accelerator_serial: "HLLWM2B233704667"
accelerator_verified: true # /dev/hailo0 detected
cooling: "Active fan" cooling: "Active fan"
psu: "Geekworm 27W 5V/5A USB-C" psu: "Geekworm 27W 5V/5A USB-C"
ip_local: "192.168.4.89" ip_local: "192.168.4.89"
ip_tailscale: "100.72.180.98" ip_tailscale: "100.72.180.98"
os: "Debian 12 (Bookworm)" mac: "88:a2:9e:3b:eb:72"
kernel: "6.1 LTS" os: "Debian 13 (Trixie)" # NOT Bookworm!
kernel: "6.12.62+rpt-rpi-2712"
ssh_user: "blackroad"
roles: roles:
- primary_ai_host - primary_ai_host
- orchestration - orchestration
@@ -62,18 +68,21 @@ fleet:
- name: octavia - name: octavia
tier: production tier: production
hardware: "Raspberry Pi 5" hardware: "Raspberry Pi 5 Model B Rev 1.1"
ram: "8GB" ram: "8GB (6.6GB used, 1.3GB available)" # WARNING: memory pressure
storage: "235GB microSD (Samsung EVO Select)" storage: "235GB microSD (34% used, 76G/235G)"
case: "Pironman" case: "Pironman"
accelerator: "Hailo-8 M.2 26 TOPS" accelerator: null # HAILO NOT DETECTED despite registry
accelerator_serial: "HLLWM2B233704606" accelerator_verified: false
cooling: "Pironman dual-fan tower" cooling: "Pironman dual-fan tower"
psu: "Geekworm 27W 5V/5A USB-C" psu: "Geekworm 27W 5V/5A USB-C"
ip_local: "192.168.4.38" ip_local: "192.168.4.38"
ip_tailscale: "100.83.149.86" ip_tailscale: "100.66.235.47" # CORRECTED (was swapped with Lucidia)
mac: "2c:cf:67:cf:fa:17"
os: "Debian 12 (Bookworm)" os: "Debian 12 (Bookworm)"
kernel: "6.1 LTS" kernel: "6.12.62+rpt-rpi-2712"
ssh_user: "blackroad"
load_average: "9.47" # WARNING: overloaded
roles: roles:
- ai_inference - ai_inference
- auth - auth
@@ -99,9 +108,12 @@ fleet:
cooling: "ElectroCookie tower cooler" cooling: "ElectroCookie tower cooler"
psu: "Geekworm 27W 5V/5A USB-C" psu: "Geekworm 27W 5V/5A USB-C"
ip_local: "192.168.4.81" ip_local: "192.168.4.81"
ip_tailscale: "100.66.235.47" ip_tailscale: "100.83.149.86" # CORRECTED (was swapped with Octavia)
os: "Debian 12 (Bookworm)" mac: null # ARP incomplete — node is DOWN
kernel: "6.1 LTS" os: "Debian 12 (Bookworm)" # last known
kernel: "6.12.62+rpt-rpi-2712" # last known
ssh_user: "blackroad"
status_override: "DOWN — unreachable via ping, ARP incomplete"
roles: roles:
- nats_bus - nats_bus
- llm_inference - llm_inference
@@ -116,17 +128,20 @@ fleet:
- name: aria - name: aria
tier: production tier: production
hardware: "Raspberry Pi 5" hardware: "Raspberry Pi 5 Model B Rev 1.1"
ram: "8GB" ram: "8GB (3.8GB used, 4.0GB available)"
storage: "29GB microSD (Samsung EVO Select)" storage: "29GB microSD (74% used, 7.3GB free)" # WARNING: low space
case: "Pironman" case: "Pironman"
accelerator: "Hailo-8 M.2 26 TOPS" accelerator: null # HAILO NOT DETECTED despite registry
accelerator_verified: false
cooling: "Pironman dual-fan tower" cooling: "Pironman dual-fan tower"
psu: "Geekworm 27W 5V/5A USB-C" psu: "Geekworm 27W 5V/5A USB-C"
ip_local: "192.168.4.82" ip_local: "192.168.4.82"
ip_tailscale: "100.109.14.17" ip_tailscale: "100.109.14.17"
mac: "88:a2:9e:0d:42:07"
os: "Debian 12 (Bookworm)" os: "Debian 12 (Bookworm)"
kernel: "6.1 LTS" kernel: "6.12.62+rpt-rpi-2712"
ssh_user: "blackroad"
roles: roles:
- api_services - api_services
- compute - compute
@@ -139,20 +154,22 @@ fleet:
tier: production tier: production
hardware: "Raspberry Pi 5" hardware: "Raspberry Pi 5"
ram: "8GB" ram: "8GB"
storage: "NVMe (Crucial P310, Pironman)" storage: "Unknown (SSH closed)"
case: "Pironman" case: "Pironman"
accelerator: null accelerator: null
cooling: "Pironman dual-fan tower" cooling: "Pironman dual-fan tower"
psu: "Geekworm 27W 5V/5A USB-C" psu: "Geekworm 27W 5V/5A USB-C"
ip_local: "192.168.4.33" ip_local: "192.168.4.33"
ip_tailscale: null ip_tailscale: null
os: "Debian 12 (Bookworm)" mac: "60:92:c8:11:cf:7c" # Confirmed Raspberry Pi 5
kernel: "6.1 LTS" os: "Unknown (SSH closed)"
ssh_user: "blackroad"
ssh_note: "WARNING: SSH alias 'anastasia' points to DO droplet, NOT this Pi"
roles: roles:
- ai_inference_secondary - ai_inference_secondary
services: [] services: []
ssh: "ssh anastasia" ssh: "ssh blackroad@192.168.4.33" # Direct IP needed, alias is wrong
status: active status: ssh_closed # Ping responds but port 22 refused
- name: cordelia - name: cordelia
tier: production tier: production
@@ -175,17 +192,20 @@ fleet:
- name: alice - name: alice
tier: production tier: production
hardware: "Raspberry Pi 400" hardware: "Raspberry Pi 400 Rev 1.0"
ram: "4GB" ram: "4GB (579MB used, 3.1GB available)"
storage: "32GB microSD" storage: "15GB root (71% used, 4.1GB free)" # NOT 32GB
case: "Built-in keyboard" case: "Built-in keyboard"
accelerator: null accelerator: null
cooling: "Passive (built-in)" cooling: "Passive (built-in)"
psu: "5V/3A USB-C (15W)" psu: "5V/3A USB-C (15W)"
ip_local: "192.168.4.49" ip_local: "192.168.4.49"
ip_tailscale: "100.77.210.18" ip_tailscale: "100.77.210.18"
os: "Debian 12 (Bookworm)" mac: "d8:3a:dd:ff:98:87"
kernel: "6.1 LTS" os: "Raspbian 11 (Bullseye)" # NOT Bookworm — needs upgrade
kernel: "6.1.21-v8+"
ssh_user: "blackroad"
load_average: "6.17" # High for 4 cores
roles: roles:
- gateway - gateway
- development - development
@@ -215,39 +235,57 @@ fleet:
# ── Cloud Compute ─────────────────────────────────────────── # ── Cloud Compute ───────────────────────────────────────────
- name: codex-infinity - name: codex-infinity
tier: cloud tier: cloud
hardware: "DigitalOcean Droplet" actual_hostname: "gematria"
ram: "1GB" hardware: "DigitalOcean Droplet (DO-Premium-AMD)"
storage: "78GB SSD" ram: "~765MB"
storage: "25GB+"
ip_local: "159.65.43.12" ip_local: "159.65.43.12"
ip_tailscale: "100.108.132.8" ip_tailscale: "100.108.132.8"
os: "Debian 12 (Bookworm)" os: "Debian/Ubuntu based"
kernel: "5.15 LTS" ssh_user: "blackroad"
ssh_alias: "gematria"
roles: roles:
- codex_server - codex_server
- oracle - oracle
services: services:
- codex-db - ollama (port 11434, public)
- cloud-services - nginx (80, 443)
- cloudflared
- caddy (2019 admin)
- python-app (8787, 8011)
tunnel: "tunnel-codex.blackroad.io" tunnel: "tunnel-codex.blackroad.io"
status: active status: active
uptime: "55+ days"
- name: shellfish - name: shellfish
tier: cloud tier: cloud
hardware: "DigitalOcean Droplet" actual_hostname: "anastasia" # WARNING: name collision with Pi
ram: "1GB" hardware: "DigitalOcean Droplet (DO-Premium-AMD)"
storage: "25GB SSD" ram: "765MB (408MB used)"
storage: "25GB SSD (57% used, 15G/25G)"
ip_local: "174.138.44.45" ip_local: "174.138.44.45"
ip_tailscale: "100.94.33.37" ip_tailscale: "100.94.33.37"
os: "Debian 12 (Bookworm)" os: "CentOS Stream 9"
kernel: "5.15 LTS" kernel: "5.14.0-651.el9.x86_64"
ssh_user: "blackroad"
ssh_alias: "anastasia" # WARNING: collides with Pi
ssh_alias_alt: "cadence"
roles: roles:
- cloud_infrastructure - cloud_infrastructure
- edge - edge
services: services:
- tunnels - ollama (port 11434, Tailscale only at 100.64.0.1)
- edge-agent - nginx (80)
- cloudflared
- docker
- uvicorn-api (8000)
- websocket (8765, 8766)
- redis-like (6379)
- grafana (3000, 3001)
- python-apps (8080, 8787, 8888)
tunnel: "tunnel-cadence.blackroad.io" tunnel: "tunnel-cadence.blackroad.io"
status: active status: active
uptime: "55+ days"
# ── Edge Compute ──────────────────────────────────────────── # ── Edge Compute ────────────────────────────────────────────
- name: jetson-agent - name: jetson-agent
@@ -362,6 +400,32 @@ fleet:
speaker. Could be re-acquired for doorbell/monitor use case. speaker. Could be re-acquired for doorbell/monitor use case.
Ethos-U55 NPU adds ~1 TOPS for vision inference. Ethos-U55 NPU adds ~1 TOPS for vision inference.
# ── Unidentified LAN Devices ────────────────────────────────
unidentified:
- ip: "192.168.4.22"
mac: "30:be:29:5b:24:5f"
oui: "Unknown (possibly Hisense)"
ping: true
guess: "Smart TV or IoT device"
- ip: "192.168.4.44"
mac: "98:17:3c:38:db:78"
oui: "TP-Link"
ping: true
guess: "WiFi extender or smart plug"
- ip: "192.168.4.83"
mac: "54:4c:8a:9b:09:3d"
oui: "Unknown (Shenzhen Bilian)"
ping: true
guess: "Smart home WiFi module"
- ip: "192.168.4.92"
mac: "de:a2:b7:f3:f9:5d"
oui: "Locally administered (Apple device)"
ping: false
guess: "Apple device with private WiFi address"
# ── MCU Array ─────────────────────────────────────────────── # ── MCU Array ───────────────────────────────────────────────
microcontrollers: microcontrollers:
- type: "ESP32-S3 SuperMini" - type: "ESP32-S3 SuperMini"

View File

@@ -1,75 +1,160 @@
# Services Map # Services Map — Live Verified
Which services run on which device, and on which ports. **Verified via `ss -tlnp` SSH probes on 2026-02-21.**
--- ---
## Service-to-Device Matrix ## Cecilia (192.168.4.89) — 16+ services
| Service | Port | Node(s) | Protocol | Notes | | Port | Service | Bind | Process |
|---------|------|---------|----------|-------| |------|---------|------|---------|
| NATS | 4222 | Lucidia | TCP | Central event bus | | 22 | SSH | 0.0.0.0 | sshd |
| Ollama | 11434 | Lucidia, Cecilia | HTTP | LLM inference | | 53 | DNS | 0.0.0.0 | (resolver) |
| Hailo Runtime | — | Cecilia, Octavia, Aria | Local | AI accelerator driver | | 80 | HTTP | 0.0.0.0 | nginx/caddy |
| CECE OS | 8080+ | Cecilia | HTTP | 68 sovereign apps | | 631 | CUPS (printing) | 127.0.0.1 | cupsd |
| PowerDNS | 53 | Octavia | TCP/UDP | Internal DNS | | 3001 | App (dashboard?) | 0.0.0.0 | python3 |
| PowerDNS-Admin | 8080 | Octavia | HTTP | DNS management UI | | 3100 | Loki/log aggregator | 0.0.0.0 | — |
| RoadAuth | — | Octavia | HTTP | Authentication service | | 5001 | Python service | 0.0.0.0 | python3 |
| RoadAPI | — | Octavia | HTTP | API gateway | | 5002 | Python service | 0.0.0.0 | python3 |
| Auth-Gateway | — | Octavia | HTTP | Auth proxy | | 5432 | **PostgreSQL** | 127.0.0.1 | postgres |
| Cloudflared | — | Cecilia, Lucidia, Octavia, Codex-Infinity, Shellfish | — | Cloudflare tunnel agent | | 5900 | **VNC** | 0.0.0.0 | vnc |
| Edge-Agent | — | Lucidia, Shellfish | — | Edge compute agent | | 8086 | **InfluxDB** | 0.0.0.0 | influxd |
| Worker | — | Alice | — | Task worker | | 8787 | Python service | 0.0.0.0 | python3 |
| Codex-DB | — | Codex-Infinity | SQLite | Component index | | 9000 | **MinIO** (S3) | 0.0.0.0 + [::] | minio |
| Cloud-Services | — | Codex-Infinity | — | Cloud oracle | | 9001 | **MinIO Console** | 0.0.0.0 | minio |
| MQTT (planned) | 1883 | Pi-Ops (planned) | TCP | IoT pub/sub broker | | 9100 | **Node Exporter** (Prometheus) | 0.0.0.0 | python3 |
| 11434 | **Ollama** | 127.0.0.1 | ollama |
| 34001 | Tailscale relay | 0.0.0.0 | tailscaled |
**Systemd services:** hailort, ollama, cloudflared, docker
**Infrastructure stack:** PostgreSQL + InfluxDB + MinIO + Loki + Node Exporter = full observability
--- ---
## Per-Node Service List ## Octavia (192.168.4.38) — 30+ services (OVERLOADED)
### Cecilia (192.168.4.89) | Port Range | Service | Bind | Process |
- Ollama (LLM inference) |------------|---------|------|---------|
- CECE OS (68 sovereign apps) | 3002-3006 | App services (5 ports) | 0.0.0.0 | containers |
- Hailo runtime (26 TOPS accelerator) | 3109 | App service | 0.0.0.0 | — |
- Cloudflared tunnel | 4001-4002 | App services | 0.0.0.0 | — |
| 4010 | App service | 127.0.0.1 | — |
| 5200 | Python microservice | 0.0.0.0 | python3 |
| 5300 | Python microservice | 0.0.0.0 | python3 |
| 5400 | Python microservice | 0.0.0.0 | python3 |
| 5500 | Python microservice | 0.0.0.0 | python3 |
| 5600 | Python microservice | 0.0.0.0 | python3 |
| 5900 | Python microservice | 0.0.0.0 | python3 |
| 6000 | Python microservice | 0.0.0.0 | python3 |
| 6100 | Python microservice | 0.0.0.0 | python3 |
| 6200 | Python microservice | 0.0.0.0 | python3 |
| 6300 | Python microservice | 0.0.0.0 | python3 |
| 8000 | API (uvicorn/gunicorn) | 0.0.0.0 | — |
| 8011 | Python service | 0.0.0.0 | — |
| 8080-8082 | HTTP services | 0.0.0.0 | — |
| 8180 | Python service | 0.0.0.0 | — |
| 5432 | PostgreSQL | 127.0.0.1 | postgres |
| 11434 | Ollama | 127.0.0.1 | ollama |
| 34001 | Tailscale relay | 0.0.0.0 | tailscaled |
### Lucidia (192.168.4.81) **Systemd services:** ollama, ollama-bridge, cloudflared, docker
- NATS event bus (port 4222)
- Ollama (LLM inference, port 11434)
- Edge-agent
- Cloudflared tunnel
### Octavia (192.168.4.38) > **WARNING:** 30+ listening ports, load average 9.47, RAM 6.6/7.9GB.
- Hailo runtime (26 TOPS accelerator) > This node needs service migration or hardware upgrade.
- PowerDNS (port 53)
- PowerDNS-Admin (port 8080)
- RoadAuth
- RoadAPI
- Auth-Gateway
- Cloudflared tunnel
### Aria (192.168.4.82) ---
- Hailo runtime (26 TOPS accelerator)
- Compute workloads (9 containers)
### Alice (192.168.4.49) ## Aria (192.168.4.82) — 30+ services
- Worker node (7 containers)
### Anastasia (192.168.4.33) | Port Range | Service | Bind |
- (Pending service deployment) |------------|---------|------|
| 3140-3167 | **28 Docker container ports** | 0.0.0.0 |
| 8081 | HTTP service | 0.0.0.0 |
| 8180 | Python service | 0.0.0.0 |
### Cordelia (192.168.4.27) **Systemd services:** ollama, cloudflared, docker
- (Pending service deployment)
### Codex-Infinity (159.65.43.12) > 28 container ports in 3140-3167 range. Disk 74% full — monitor closely.
- Codex database
- Cloud services
- Cloudflared tunnel
### Shellfish (174.138.44.45) ---
- Cloudflare tunnels
- Edge-agent ## Alice (192.168.4.49) — Minimal
| Port | Service |
|------|---------|
| 22 | SSH |
**Systemd services:** cloudflared, docker
> Light node. Load 6.17 is concerning for 4-core Pi 400 — investigate docker workloads.
---
## Shellfish / "anastasia" (174.138.44.45) — 14+ services
| Port | Service | Process |
|------|---------|---------|
| 22 | SSH | sshd |
| 80 | HTTP | nginx |
| 3000 | Dashboard (Grafana?) | node |
| 3001 | Dashboard | node |
| 6379 | Redis-like | python3 |
| 8000 | API | uvicorn |
| 8080 | HTTP service | — |
| 8765 | WebSocket server | python3 |
| 8766 | WebSocket server | python3 |
| 8787 | Python service | python3 |
| 8888 | Python service | python3 |
| 11434 | Ollama | ollama (Tailscale-only: 100.64.0.1) |
**Systemd services:** cloudflared, docker, nginx, ollama
---
## Codex-Infinity / "gematria" (159.65.43.12) — 7 services
| Port | Service | Process |
|------|---------|---------|
| 22 | SSH | sshd |
| 53 | DNS (local resolver) | systemd-resolved |
| 80 | HTTP | nginx |
| 443 | HTTPS | nginx |
| 2019 | Caddy admin | caddy |
| 8011 | App service | — |
| 8787 | Python service | python3 |
| 11434 | **Ollama (PUBLIC!)** | ollama |
**Systemd services:** cloudflared, nginx, ollama
> **SECURITY NOTE:** Ollama is bound to `*:11434` (all interfaces) on this public-facing droplet.
> Consider restricting to localhost or Tailscale interface only.
---
## Service Distribution Summary
| Service | Cecilia | Octavia | Aria | Alice | Shellfish | Codex-Inf |
|---------|---------|---------|------|-------|-----------|-----------|
| SSH | 22 | 22 | 22 | 22 | 22 | 22 |
| HTTP | 80 | 8000+ | — | — | 80 | 80 |
| HTTPS | — | — | — | — | — | 443 |
| Ollama | 11434 (lo) | 11434 (lo) | ✓ | — | 11434 (TS) | 11434 (**PUBLIC**) |
| PostgreSQL | 5432 | 5432 | — | — | — | — |
| Docker | ✓ | ✓ | ✓ | ✓ | ✓ | — |
| Cloudflared | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| Hailo | ✓ | — | — | — | — | — |
| MinIO | 9000 | — | — | — | — | — |
| InfluxDB | 8086 | — | — | — | — | — |
| nginx | — | — | — | — | ✓ | ✓ |
### Ollama Deployment (4 nodes!)
Ollama runs on 4 of 6 reachable nodes:
1. **Cecilia** — localhost only (secure)
2. **Octavia** — localhost only (secure) + SSE bridge
3. **Shellfish** — Tailscale interface only (secure)
4. **Codex-Infinity****ALL INTERFACES** (security risk on public IP)
--- ---
@@ -78,39 +163,28 @@ Which services run on which device, and on which ports.
``` ```
┌───────────┐ ┌───────────┐
│ NATS │ │ NATS │
│ (Lucidia) │ │ (Lucidia) │ ← DOWN
└─────┬─────┘ └─────┬─────┘
┌───────────────┼───────────────┐ ┌───────────────┼───────────────┐
│ │ │ │ │ │
┌──────┴──────┐ ┌─────┴─────┐ ┌──────┴──────┐ ┌──────┴──────┐ ┌─────┴─────┐ ┌──────┴──────┐
│ Ollama │ │ Edge-Agent │ │ CECE OS │ │ Ollama │ │ Observ. │ │ CECE OS │
(Lucidia/Cec)│ │(Lucidia) │ │ (Cecilia) │ (4 nodes) │ │ Stack │ │ (Cecilia) │
└─────────────┘ └───────────┘ └─────────────┘ └─────────────┘ │(Cecilia) │ └─────────────┘
│InfluxDB
┌─────┴─────┐ │Loki │
│ Hailo RT │MinIO
│(Cec/Oct/Ar) │NodeExport
└───────────┘ ──────────┘
┌───────────┐ ┌───────────┐ ┌───────────┐ ┌───────────┐
│ PowerDNS │ │ Auth-GW │ PostgreSQL│ │ Hailo RT
(Octavia) │◄───────│ (Octavia) │ (Cec + Oct)│ │(Cecilia) │ ← Only 1 confirmed
└───────────┘ └───────────┘ └───────────┘ └───────────┘
┌────────────────┐ ┌────────────────┐
│ Cloudflared │ ← All production nodes │ Cloudflared │ ← 5 nodes (all except Alice has tunnel)
│ (5 tunnels) │ │ (5 tunnels) │
└────────────────┘ └────────────────┘
``` ```
---
## Planned Services
| Service | Port | Target Node | Purpose |
|---------|------|-------------|---------|
| Mosquitto MQTT | 1883 | Pi-Ops | IoT sensor pub/sub |
| Headscale | 443 | Alice | Self-hosted Tailscale control |
| Monitoring Dashboard | 3000 | Pi-Ops | Grafana/custom dashboard |
| Agent UI | 8080 | Jetson-Agent | Touch-based agent control |

View File

@@ -1,6 +1,6 @@
# Network Topology # Network Topology — Live Verified
BlackRoad fleet network architecture — LAN, Tailscale mesh, cloud entry points. **Verified against ARP table, ping sweep, SSH probes, and DNS dig on 2026-02-21.**
--- ---
@@ -8,13 +8,13 @@ BlackRoad fleet network architecture — LAN, Tailscale mesh, cloud entry points
### Network Equipment ### Network Equipment
| Device | Model | Ports | Role | | Device | Model | MAC | Role |
|--------|-------|-------|------| |--------|-------|-----|------|
| Router/WiFi | TP-Link | | Gateway (192.168.4.1), DHCP, WiFi | | Router/WiFi | TP-Link | 44:ac:85:94:37:92 | Gateway (192.168.4.1), DHCP, WiFi |
| Switch | TP-Link TL-SG105 | 5-port Gigabit | Wired backbone | | Switch | TP-Link TL-SG105 | — | 5-port Gigabit wired backbone |
| WiFi Card | TP-Link AX3000 PCIe WiFi 6 | — | High-speed wireless | | Unknown TP-Link | TP-Link | 98:17:3c:38:db:78 | 192.168.4.44 — extender/smart plug? |
### Topology Diagram ### Live ARP Map (2026-02-21)
``` ```
┌─────────────────────┐ ┌─────────────────────┐
@@ -24,60 +24,71 @@ BlackRoad fleet network architecture — LAN, Tailscale mesh, cloud entry points
┌──────────┴──────────┐ ┌──────────┴──────────┐
│ TP-Link Router │ │ TP-Link Router │
│ 192.168.4.1 │ │ 192.168.4.1 │
DHCP / WiFi / NAT │ 44:ac:85:94:37:92
└──────────┬──────────┘ └──────────┬──────────┘
──────────────────┼──────────────────┐ ┌────────────────────┼────────────────────
│ │ │ │ │ │
┌──────┴──────┐ ┌──────┴──────┐ WiFi Clients ┌──────┴──────┐ ┌──────┴──────┐ WiFi Clients
│ TL-SG105 │ │ WiFi AP │ │ TL-SG105 │ │ WiFi AP │ (see below)
│ Gigabit SW │ │ (built-in) │ │ Gigabit SW │ │ (built-in) │
└┬───┬───┬───┬┘ └─────────────┘ └┬───┬───┬───┬┘ └─────────────┘
│ │ │ │ │ │ │ │
│ │ │ └── Cecilia .89 [Hailo-8, CECE OS] │ │ │ └── Cecilia .89 88:a2:9e:3b:eb:72 [UP] Hailo-8
│ │ └────── Lucidia .81 [NATS, Ollama] │ │ └────── Octavia .38 2c:cf:67:cf:fa:17 [UP] OVERLOADED
│ └────────── Aria .82 [Hailo-8, API] │ └────────── Aria .82 88:a2:9e:0d:42:07 [UP]
└────────────── Octavia .38 [Hailo-8, DNS] └────────────── Lucidia .81 (incomplete) [DOWN]
WiFi: WiFi:
├── Alexandria .28 [MacBook Pro M1] ├── Alexandria .28 b0:be:83:66:cc:10 [UP] Apple Mac
├── Alice .49 [Pi 400] ├── Alice .49 d8:3a:dd:ff:98:87 [UP] Pi 400
├── Anastasia .33 [Pi 5] ├── Anastasia .33 60:92:c8:11:cf:7c [UP] Pi 5 (no SSH)
├── Cordelia .27 [Pi 5] ├── Cordelia .27 6c:4a:85:32:ae:72 [UP] Pi 5 (no SSH)
├── Athena .45 [Heltec LoRa ESP32] ├── Athena .45 d0:c9:07:50:51:ca [UP] ESP32 LoRa
├── Phoebe .88 [iPhone] ├── Iris .26 d4:be:dc:6c:61:6b [UP] Roku
├── Ares .90 [Xbox] ├── Ares .90 a0:4a:5e:2a:db:d2 [DOWN] Xbox
── Iris .26 [Roku] ── Phoebe .88 9e:0d:2a:82:99:96 [DOWN] iPhone (private MAC)
├── UNKNOWN .22 30:be:29:5b:24:5f [UP] Smart TV/IoT?
├── UNKNOWN .44 98:17:3c:38:db:78 [UP] TP-Link device
├── UNKNOWN .83 54:4c:8a:9b:09:3d [UP] Smart home module?
└── UNKNOWN .92 de:a2:b7:f3:f9:5d [DOWN] Apple (private MAC)
``` ```
### IP Address Map ### Complete IP-to-MAC-to-Identity Table
| IP | Hostname | Type | Wired/WiFi | | IP | MAC | OUI Vendor | Identity | Ping | SSH |
|----|----------|------|------------| |----|-----|-----------|----------|------|-----|
| .1 | Router | TP-Link Gateway | — | | .1 | 44:ac:85:94:37:92 | TP-Link | Router | UP | — |
| .26 | Iris | Roku | WiFi | | .22 | 30:be:29:5b:24:5f | Unknown | **UNIDENTIFIED** | UP | — |
| .27 | Cordelia | Pi 5 | WiFi | | .26 | d4:be:dc:6c:61:6b | Roku | Iris (streaming) | UP | — |
| .28 | Alexandria | MacBook Pro M1 | WiFi | | .27 | 6c:4a:85:32:ae:72 | Raspberry Pi 5 | Cordelia | UP | REFUSED |
| .33 | Anastasia | Pi 5 | WiFi | | .28 | b0:be:83:66:cc:10 | Apple | Alexandria (Mac M1) | UP | — |
| .38 | Octavia | Pi 5 | Wired | | .33 | 60:92:c8:11:cf:7c | Raspberry Pi 5 | Anastasia (Pi) | UP | REFUSED |
| .45 | Athena | Heltec LoRa ESP32 | WiFi | | .38 | 2c:cf:67:cf:fa:17 | Raspberry Pi | Octavia | UP | OK |
| .49 | Alice | Pi 400 | WiFi | | .44 | 98:17:3c:38:db:78 | TP-Link | **UNIDENTIFIED** | UP | — |
| .81 | Lucidia | Pi 5 | Wired | | .45 | d0:c9:07:50:51:ca | Espressif | Athena (ESP32 LoRa) | UP | — |
| .82 | Aria | Pi 5 | Wired | | .49 | d8:3a:dd:ff:98:87 | Raspberry Pi | Alice (Pi 400) | UP | OK |
| .88 | Phoebe | iPhone | WiFi | | .74 | (incomplete) | — | **STALE** (old Octavia IP) | DOWN | — |
| .89 | Cecilia | Pi 5 | Wired | | .81 | (incomplete) | — | Lucidia (Pi 5) | **DOWN** | — |
| .90 | Ares | Xbox | WiFi | | .82 | 88:a2:9e:0d:42:07 | Raspberry Pi 5 | Aria | UP | OK |
| .83 | 54:4c:8a:9b:09:3d | Unknown | **UNIDENTIFIED** | UP | — |
| .88 | 9e:0d:2a:82:99:96 | Private MAC | Phoebe (iPhone) | DOWN | — |
| .89 | 88:a2:9e:3b:eb:72 | Raspberry Pi 5 | Cecilia | UP | OK |
| .90 | a0:4a:5e:2a:db:d2 | Microsoft | Ares (Xbox) | DOWN | — |
| .92 | de:a2:b7:f3:f9:5d | Private MAC | **UNIDENTIFIED** | DOWN | — |
--- ---
## Tailscale Mesh Overlay ## Tailscale Mesh Overlay — Corrected
Encrypted WireGuard mesh connecting on-premises and cloud nodes. > **CRITICAL FIX:** Lucidia and Octavia Tailscale IPs were swapped in prior documentation.
> Corrected based on SSH config and live `ss` output showing Tailscale binding addresses.
``` ```
┌───────────────┐ ┌───────────────┐
│ Tailscale │ │ Tailscale │
│ Control Plane │ Coord Server
└───────┬───────┘ └───────┬───────┘
┌───────────────────┼───────────────────┐ ┌───────────────────┼───────────────────┐
@@ -91,83 +102,84 @@ Encrypted WireGuard mesh connecting on-premises and cloud nodes.
┌────┼────┬────────┬─────────┐ ┌────┼────┬────────┬─────────┐
│ │ │ │ │ │ │ │ │ │
Lucidia │ Aria Octavia Alice Octavia │ Aria Lucidia Alice
100.66 │ 100.109 100.83 100.77 100.66 │ 100.109 100.83 100.77
.235.47 │ .14.17 .149.86 .210.18 .235.47 │ .14.17 .149.86 .210.18
(DOWN)
(Full mesh — every (Full mesh — every
node can reach node can reach
every other node) every other node)
``` ```
### Tailscale Node Table | Node | Tailscale IP | SSH Alias | Verified By | Status |
|------|-------------|-----------|-------------|--------|
| Cecilia | 100.72.180.98 | cecilia-ts | SSH config | Active |
| Octavia | **100.66.235.47** | octavia-ts | SSH config + `ss` binding | Active |
| Lucidia | **100.83.149.86** | lucidia-ts | SSH config | **DOWN** |
| Aria | 100.109.14.17 | aria-ts | SSH config | Active |
| Alice | 100.77.210.18 | alice-ts | SSH config | Active |
| Codex-Infinity | 100.108.132.8 | gematria-ts / blackroad-os-ts | SSH config | Active |
| Shellfish | 100.94.33.37 | anastasia-ts / cadence-ts | SSH config + `ss` binding | Active |
| Node | Tailscale IP | OS | Connected | ### Not on Tailscale
|------|-------------|-----|-----------|
| Cecilia | 100.72.180.98 | Linux | Yes |
| Lucidia | 100.66.235.47 | Linux | Yes |
| Octavia | 100.83.149.86 | Linux | Yes |
| Aria | 100.109.14.17 | Linux | Yes |
| Alice | 100.77.210.18 | Linux | Yes |
| Codex-Infinity | 100.108.132.8 | Linux | Yes |
| Shellfish | 100.94.33.37 | Linux | Yes |
### SSH via Tailscale - Alexandria (Mac) — tailscale not running
- Anastasia Pi (192.168.4.33) — SSH closed, can't configure
```bash - Cordelia (192.168.4.27) — SSH closed, can't configure
ssh cecilia-ts # → 100.72.180.98 - Olympia — offline
ssh lucidia-ts # → 100.66.235.47
ssh octavia-ts # → 100.83.149.86
ssh aria-ts # → 100.109.14.17
ssh alice-ts # → 100.77.210.18
```
### Nodes Not Yet on Tailscale
- Anastasia (192.168.4.33)
- Cordelia (192.168.4.27)
- Olympia (offline)
- Jetson-Agent (pending setup)
- Pi-Holo, Pi-Ops, Pi-Zero-Sim (planned)
--- ---
## Cloud Entry Points ## Cloud Entry Points — Cloudflare
### Cloudflare Tunnels ### DNS Resolution (all Cloudflare-proxied)
Each production node has a Cloudflare tunnel for HTTPS ingress: All blackroad.io DNS resolves to Cloudflare CDN, not origin servers directly:
| Tunnel | Node | Config | | Subdomain | A Record |
|--------|------|--------| |-----------|----------|
| tunnel-cecilia.blackroad.io | Cecilia | /etc/cloudflared/config.yml | | blackroad.io | 172.67.211.99 |
| tunnel-lucidia.blackroad.io | Lucidia | /etc/cloudflared/config.yml | | www.blackroad.io | 172.67.211.99 |
| tunnel-octavia.blackroad.io | Octavia | /etc/cloudflared/config.yml | | api.blackroad.io | 172.67.211.99 |
| tunnel-codex.blackroad.io | Codex-Infinity | /etc/cloudflared/config.yml | | status.blackroad.io | 172.67.211.99 |
| tunnel-cadence.blackroad.io | Shellfish | /etc/cloudflared/config.yml | | docs.blackroad.io | 172.67.211.99 |
| dashboard.blackroad.io | 172.67.211.99 |
| monitoring.blackroad.io | 172.67.211.99 |
| agents.blackroad.io | 104.21.91.74 |
| tunnel-cecilia.blackroad.io | 172.67.211.99 |
| tunnel-lucidia.blackroad.io | 104.21.91.74 |
| tunnel-octavia.blackroad.io | 172.67.211.99 |
| tunnel-codex.blackroad.io | 104.21.91.74 |
| tunnel-cadence.blackroad.io | 172.67.211.99 |
### DNS **Traffic flow:** Client → Cloudflare CDN → Cloudflare Tunnel → `cloudflared` on origin node
- **Provider:** Cloudflare ### Cloudflare Tunnels (verified via `cloudflared.service`)
- **Zone:** blackroad.io
- **Internal DNS:** PowerDNS on Octavia | Tunnel | Origin Node | cloudflared Status |
- **Time Sync:** chrony → time.cloudflare.com (all nodes) |--------|-------------|-------------------|
| tunnel-cecilia.blackroad.io | Cecilia (192.168.4.89) | Running |
| tunnel-lucidia.blackroad.io | Lucidia (192.168.4.81) | **DOWN** (node unreachable) |
| tunnel-octavia.blackroad.io | Octavia (192.168.4.38) | Running |
| tunnel-codex.blackroad.io | Codex-Infinity (159.65.43.12) | Running |
| tunnel-cadence.blackroad.io | Shellfish (174.138.44.45) | Running |
--- ---
## LoRa Network (Planned) ## Stale Network Data to Clean Up
Athena (Heltec WiFi LoRa 32) serves as the LoRa mesh backbone: | Item | Location | Issue | Fix |
|------|----------|-------|-----|
- **Frequency:** 868/915 MHz | `/etc/hosts` entry | Alexandria Mac | `192.168.4.74 octavia` — wrong IP | Change to `192.168.4.38 octavia` |
- **Range:** Up to 10km line-of-sight | `hailo.sh` | `~/hailo.sh` | Connects to `pi@192.168.4.74` | Change to `blackroad@192.168.4.38` |
- **Protocol:** LoRaWAN or point-to-point | SSH `anastasia` alias | `~/.ssh/config` | Points to DO droplet, not Pi | Add `anastasia-pi` for 192.168.4.33 |
- **Use Cases:** Remote sensor relay, out-of-WiFi-range monitoring | `blackroad-fleet.yaml` | `~/blackroad-fleet.yaml` | Lucidia/Octavia IPs swapped | Fix both local and Tailscale IPs |
- **Management:** `~/lora.sh` | Agent registry | `~/.blackroad-agent-registry.db` | Octavia: `pironman_hailo8` | Change to `pironman` (no Hailo) |
| Agent registry | `~/.blackroad-agent-registry.db` | Aria: `pironman_hailo8` | Change to `pironman` (no Hailo) |
--- ---
## Firewall Rules (All Nodes) ## Firewall Rules (Standard)
| Port | Protocol | Service | Direction | | Port | Protocol | Service | Direction |
|------|----------|---------|-----------| |------|----------|---------|-----------|
@@ -176,15 +188,41 @@ Athena (Heltec WiFi LoRa 32) serves as the LoRa mesh backbone:
| 443 | TCP | HTTPS | Inbound | | 443 | TCP | HTTPS | Inbound |
| 41641 | UDP | Tailscale | Inbound | | 41641 | UDP | Tailscale | Inbound |
Default policy: **deny** all other inbound traffic. Default policy: **deny** all other inbound.
---
## LoRa Network
Athena (Heltec WiFi LoRa 32) at 192.168.4.45:
- **Frequency:** 868/915 MHz
- **Range:** Up to 10km line-of-sight
- **MAC:** d0:c9:07:50:51:ca (Espressif OUI confirmed)
- **Status:** Powered on (responds to ARP)
- **Management:** `~/lora.sh`
--- ---
## Network Diagnostics ## Network Diagnostics
```bash ```bash
~/blackroad-network-scan.sh # ARP + ping sweep + Tailscale status # Live ARP table (shows devices seen recently)
~/blackroad-network-discovery.sh # SSH probe all known devices arp -a | grep "192.168.4" | grep -v incomplete | sort -t. -k4 -n
tailscale status # Tailscale mesh state
tailscale ping <hostname> # Test Tailscale connectivity # Ping sweep
for i in {1..255}; do ping -c1 -W1 192.168.4.$i &>/dev/null && echo "UP .${i}"; done
# Tailscale status
tailscale status
# SSH probe a node
ssh -o ConnectTimeout=3 cecilia "hostname; uname -r; ss -tlnp"
# DNS lookup
dig blackroad.io ANY +short
# Management scripts
~/blackroad-network-scan.sh
~/blackroad-network-discovery.sh
``` ```