Rewrite hardware backend map with live-verified network data
All docs updated from SSH probes, ARP scans, and port scans run 2026-02-21. Key corrections: only 1/3 Hailo-8 confirmed active, Lucidia/Octavia Tailscale IPs were swapped, SSH user is blackroad not alexandria, Lucidia is DOWN, Ollama publicly exposed on Codex-Infinity, Octavia overloaded at load 9.47. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -1,415 +1,569 @@
|
|||||||
# BlackRoad Hardware Backend Map
|
# BlackRoad Hardware Backend Map
|
||||||
|
|
||||||
**Canonical source of truth for all BlackRoad physical infrastructure.**
|
**Canonical source of truth for all BlackRoad physical infrastructure.**
|
||||||
|
**Verified against live network probes — not documentation, not registries.**
|
||||||
|
|
||||||
| Field | Value |
|
| Field | Value |
|
||||||
|-------|-------|
|
|-------|-------|
|
||||||
| Owner | BlackRoad OS, Inc. |
|
| Owner | BlackRoad OS, Inc. |
|
||||||
| Updated | 2026-02-20 |
|
| Updated | 2026-02-21 |
|
||||||
| Fleet Version | 2.0.0 |
|
| Fleet Version | 2.1.0 |
|
||||||
| Total Devices | 21 |
|
| Verified | Live SSH + ARP + ping sweep |
|
||||||
| Total AI Compute | ~135 TOPS |
|
| Total Devices | 21 registered + 4 unidentified |
|
||||||
| Network | 192.168.4.0/24 LAN + Tailscale mesh |
|
| Confirmed AI Compute | 26 TOPS (1x Hailo-8 verified) |
|
||||||
|
| Network | 192.168.4.0/24 LAN + 7-node Tailscale mesh |
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 1. Fleet Summary
|
## 0. ERRATA — Corrections From Live Verification
|
||||||
|
|
||||||
| # | Name | Type | Hardware | IP (Local) | IP (Tailscale) | Accelerator | Role | Status |
|
> **This section documents discrepancies found between prior documentation
|
||||||
|---|------|------|----------|------------|----------------|-------------|------|--------|
|
> and actual live system state as of 2026-02-21.**
|
||||||
| 1 | Cecilia | Pi 5 | 8GB + Hailo-8 + 500GB NVMe | 192.168.4.89 | 100.72.180.98 | Hailo-8 26 TOPS | Primary AI / CECE OS | Active |
|
|
||||||
| 2 | Octavia | Pi 5 | 8GB + Pironman + Hailo-8 | 192.168.4.38 | 100.83.149.86 | Hailo-8 26 TOPS | AI Inference | Active |
|
| Issue | Prior Documentation | Live Reality |
|
||||||
| 3 | Lucidia | Pi 5 | 8GB + ElectroCookie | 192.168.4.81 | 100.66.235.47 | — | NATS + Ollama Brain | Active |
|
|-------|-------------------|--------------|
|
||||||
| 4 | Aria | Pi 5 | 8GB + Pironman + Hailo-8 | 192.168.4.82 | 100.109.14.17 | Hailo-8 26 TOPS | API Services | Active |
|
| Hailo-8 count | 3 units (Cecilia, Octavia, Aria) | **1 confirmed** (Cecilia only). Octavia/Aria report `HAILO: none` |
|
||||||
| 5 | Anastasia | Pi 5 | 8GB + Pironman + NVMe | 192.168.4.33 | — | — | AI Inference Secondary | Active |
|
| Lucidia Tailscale IP | 100.66.235.47 | **100.83.149.86** (SSH config + live binding) |
|
||||||
| 6 | Cordelia | Pi 5 | 8GB | 192.168.4.27 | — | — | Orchestration | Active |
|
| Octavia Tailscale IP | 100.83.149.86 | **100.66.235.47** (SSH config + live binding) |
|
||||||
| 7 | Alice | Pi 400 | 4GB | 192.168.4.49 | 100.77.210.18 | — | Gateway / Auth | Active |
|
| Lucidia status | Active | **DOWN** (unreachable via ping, ARP incomplete) |
|
||||||
| 8 | Olympia | Pi 4B | PiKVM | — | — | — | KVM Console | Offline |
|
| Cecilia OS | Debian 12 Bookworm | **Debian 13 Trixie**, kernel 6.12.62 |
|
||||||
| 9 | Codex-Infinity | DO Droplet | 1 vCPU / 1GB | 159.65.43.12 | 100.108.132.8 | — | Codex Server | Active |
|
| Alice OS | Debian 12 Bookworm | **Raspbian 11 Bullseye**, kernel 6.1.21 |
|
||||||
| 10 | Shellfish | DO Droplet | 1 vCPU / 1GB | 174.138.44.45 | 100.94.33.37 | — | Cloud Edge | Active |
|
| Alice storage | 32GB SD | **15GB root partition** (71% used) |
|
||||||
| 11 | Jetson-Agent | Jetson Orin Nano | 8GB + GPU | — | — | 40 TOPS GPU | Agent UI / Inference | Pending |
|
| Octavia storage used | ~90% | **34%** (76G/235G) — was cleaned up |
|
||||||
| 12 | Alexandria | MacBook Pro M1 | 8GB | 192.168.4.28 | — | M1 Neural Engine 15.8 TOPS | Operator Workstation | Active |
|
| SSH user | `alexandria` | **`blackroad`** for fleet nodes |
|
||||||
| 13 | Athena | Heltec LoRa ESP32 | ESP32 + SX1276 | 192.168.4.45 | — | — | LoRa Mesh Node | Active |
|
| Shellfish hostname | shellfish | **`anastasia`** (hostname on the droplet) |
|
||||||
| 14 | Persephone | Sipeed RISC-V | — | — | — | — | Portable Compute | Active |
|
| Codex-Infinity hostname | codex-infinity | **`gematria`** (hostname on the droplet) |
|
||||||
| 15 | Iris | Roku | — | 192.168.4.26 | — | — | Streaming | Active |
|
| Octavia old IP | 192.168.4.74 (in /etc/hosts) | **192.168.4.38** (current, .74 is stale) |
|
||||||
| 16 | Ares | Xbox | — | 192.168.4.90 | — | — | Gaming | Active |
|
| Unknown devices | None documented | **4 found** at .22, .44, .83, .92 |
|
||||||
| 17 | Phoebe | iPhone | — | 192.168.4.88 | — | — | Mobile | Active |
|
| Anastasia/Cordelia SSH | Assumed accessible | **SSH port closed** (ping responds, port 22 refused) |
|
||||||
| 18 | Calliope | Unidentified IoT | — | — | — | — | IoT Node | Active |
|
|
||||||
| 19 | Sophia | Unidentified IoT | — | — | — | — | IoT Node | Active |
|
|
||||||
| 20 | SenseCAP W1-A | IoT AI Agent | ESP32-S3 + HX6538 | — | — | Ethos-U55 ~1 TOPS | Vision AI | Returned |
|
|
||||||
| 21 | Pi-Holo | Pi 5 (planned) | 8GB | — | — | — | Hologram Renderer | Planned |
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 2. Production Cluster — Raspberry Pis
|
## 1. Fleet Summary — Live Verified
|
||||||
|
|
||||||
Eight Raspberry Pi nodes form the always-on backbone.
|
| # | Name | Type | Hardware | IP (Local) | IP (Tailscale) | Accelerator | Status | Verified |
|
||||||
|
|---|------|------|----------|------------|----------------|-------------|--------|----------|
|
||||||
| Node | Board | RAM | Storage | Case | Accelerator | Cooling | PSU | Role |
|
| 1 | Cecilia | Pi 5 | 8GB, Hailo-8, 457GB NVMe | 192.168.4.89 | 100.72.180.98 | **Hailo-8 26 TOPS** (confirmed /dev/hailo0) | **UP** | SSH |
|
||||||
|------|-------|-----|---------|------|-------------|---------|-----|------|
|
| 2 | Octavia | Pi 5 | 8GB, Pironman, 235GB SD | 192.168.4.38 | 100.66.235.47 | **None** (HAILO: none) | **UP** | SSH |
|
||||||
| Cecilia | Pi 5 | 8GB | 500GB NVMe | Standard | Hailo-8 M.2 (26 TOPS) | Active fan | 27W USB-C | Primary AI, CECE OS |
|
| 3 | Lucidia | Pi 5 | 8GB, ElectroCookie | 192.168.4.81 | 100.83.149.86 | — | **DOWN** | Ping fail |
|
||||||
| Octavia | Pi 5 | 8GB | 235GB SD | Pironman | Hailo-8 M.2 (26 TOPS) | Pironman dual-fan tower | 27W USB-C | AI Inference |
|
| 4 | Aria | Pi 5 | 8GB, 29GB SD | 192.168.4.82 | 100.109.14.17 | **None** (HAILO: none) | **UP** | SSH |
|
||||||
| Lucidia | Pi 5 | 8GB | 117GB SD | ElectroCookie Radial Tower | — | ElectroCookie tower | 27W USB-C | NATS bus, Ollama |
|
| 5 | Anastasia | Pi 5 | 8GB | 192.168.4.33 | — | — | **SSH closed** | ARP + ping |
|
||||||
| Aria | Pi 5 | 8GB | 29GB SD | Pironman | Hailo-8 M.2 (26 TOPS) | Pironman dual-fan tower | 27W USB-C | API Services |
|
| 6 | Cordelia | Pi 5 | 8GB | 192.168.4.27 | — | — | **SSH closed** | ARP + ping |
|
||||||
| Anastasia | Pi 5 | 8GB | NVMe (Pironman) | Pironman | — | Pironman dual-fan tower | 27W USB-C | AI Inference Secondary |
|
| 7 | Alice | Pi 400 | 4GB, 15GB root | 192.168.4.49 | 100.77.210.18 | — | **UP** | SSH |
|
||||||
| Cordelia | Pi 5 | 8GB | SD | Standard | — | Active cooler | 27W USB-C | Orchestration |
|
| 8 | Olympia | Pi 4B | PiKVM | — | — | — | **Offline** | Not probed |
|
||||||
| Alice | Pi 400 | 4GB | 32GB SD | Built-in keyboard | — | Passive (built-in) | 15W USB-C | Gateway, Auth |
|
| 9 | Codex-Infinity | DO Droplet | AMD vCPU, 765MB RAM | 159.65.43.12 | 100.108.132.8 | — | **UP** | SSH (hostname: gematria) |
|
||||||
| Olympia | Pi 4B | 4GB | SD | PiKVM case | — | Passive | 15W USB-C | KVM Console |
|
| 10 | Shellfish | DO Droplet | AMD vCPU, 765MB RAM | 174.138.44.45 | 100.94.33.37 | — | **UP** | SSH (hostname: anastasia) |
|
||||||
|
| 11 | Jetson-Agent | Jetson Orin Nano | 8GB + GPU | — | — | 40 TOPS GPU | **Pending** | Not deployed |
|
||||||
### SSH Access
|
| 12 | Alexandria | MacBook Pro M1 | 8GB | 192.168.4.28 | — | M1 NE 15.8 TOPS | **UP** | Self |
|
||||||
|
| 13 | Athena | Heltec LoRa ESP32 | ESP32 + SX1276 | 192.168.4.45 | — | — | **UP** | ARP |
|
||||||
```bash
|
| 14 | Persephone | Sipeed RISC-V | — | — | — | — | Unknown | Registry only |
|
||||||
ssh alice # 192.168.4.49
|
| 15 | Iris | Roku | — | 192.168.4.26 | — | — | **UP** | ARP |
|
||||||
ssh lucidia # 192.168.4.81 (or lucidia-ts for Tailscale)
|
| 16 | Ares | Xbox | — | 192.168.4.90 | — | — | **DOWN** | Ping fail |
|
||||||
ssh aria # 192.168.4.82 (or aria-ts)
|
| 17 | Phoebe | iPhone | — | 192.168.4.88 | — | — | **DOWN** | Ping fail |
|
||||||
ssh cecilia # 192.168.4.89 (or cecilia-ts)
|
| 18 | .22 unknown | **UNIDENTIFIED** | — | 192.168.4.22 | — | — | **UP** | ARP (30:be:29) |
|
||||||
ssh octavia # 192.168.4.38 (or octavia-ts)
|
| 19 | .44 unknown | **TP-Link device** | — | 192.168.4.44 | — | — | **UP** | ARP (98:17:3c) |
|
||||||
ssh anastasia # 192.168.4.33
|
| 20 | .83 unknown | **UNIDENTIFIED** | — | 192.168.4.83 | — | — | **UP** | ARP (54:4c:8a) |
|
||||||
ssh cordelia # 192.168.4.27
|
| 21 | .92 unknown | **Apple device** (private MAC) | — | 192.168.4.92 | — | — | **DOWN** | Stale ARP |
|
||||||
```
|
|
||||||
|
|
||||||
### OS Baseline
|
|
||||||
|
|
||||||
All Pis run Debian 12 (Bookworm) with:
|
|
||||||
- Kernel: 6.1 LTS
|
|
||||||
- User: `alexandria` (uid 1000)
|
|
||||||
- SSH: key-only, no password auth
|
|
||||||
- Firewall: UFW (deny by default, allow 22/80/443/41641)
|
|
||||||
- Time sync: chrony → time.cloudflare.com
|
|
||||||
- Auto-updates: unattended-upgrades + fail2ban
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 3. Cloud Compute
|
## 2. Production Cluster — Raspberry Pis (Live Data)
|
||||||
|
|
||||||
Two DigitalOcean droplets provide cloud presence.
|
### Cecilia — Primary AI Host (VERIFIED)
|
||||||
|
|
||||||
| Node | Region | Spec | Public IP | Tailscale IP | Storage | Role |
|
|
||||||
|------|--------|------|-----------|--------------|---------|------|
|
|
||||||
| Codex-Infinity | NYC | 1 vCPU / 1GB | 159.65.43.12 | 100.108.132.8 | 78GB SSD | Codex DB, cloud services |
|
|
||||||
| Shellfish | NYC | 1 vCPU / 1GB | 174.138.44.45 | 100.94.33.37 | 25GB SSD | Edge compute, tunnels |
|
|
||||||
|
|
||||||
### OS Baseline
|
|
||||||
|
|
||||||
- Debian 12 (Bookworm), Kernel 5.15 LTS
|
|
||||||
- Same user/SSH/firewall config as Pis
|
|
||||||
- Cloudflare tunnels for ingress
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 4. Edge Compute
|
|
||||||
|
|
||||||
| Node | Hardware | Status | Purpose |
|
|
||||||
|------|----------|--------|---------|
|
|
||||||
| Jetson-Agent | NVIDIA Jetson Orin Nano 8GB | Pending setup | Agent UI on 10.1" touch, GPU inference |
|
|
||||||
| Pi-Holo | Pi 5 8GB (planned) | Planned | Hologram renderer on 4" 720x720 display |
|
|
||||||
| Pi-Ops | Pi 5 8GB (planned) | Planned | MQTT broker + ops monitor on 9.3" ultrawide |
|
|
||||||
| Pi-Zero-Sim | Pi Zero W | Ready | Lightweight sim output on 7" display |
|
|
||||||
| Persephone | Sipeed RISC-V | Active | Portable RISC-V compute experiments |
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 5. Microcontroller Array
|
|
||||||
|
|
||||||
| MCU | Chip | Qty | Connectivity | Form Factor | Purpose |
|
|
||||||
|-----|------|-----|--------------|-------------|---------|
|
|
||||||
| ESP32-S3 SuperMini | ESP32-S3 | 5 | WiFi + BLE | Tiny USB-C | General IoT |
|
|
||||||
| ESP32-S3 N8R8 | ESP32-S3 | 2 | WiFi + BLE + OTG | Dev board | 8MB PSRAM apps |
|
|
||||||
| ESP32 Touchscreen | ESP32 | 3 | WiFi + BLE | 2.8" TFT (320x240) | Standalone sensor display |
|
|
||||||
| Athena (Heltec LoRa) | ESP32 + SX1276 | 1 | WiFi + LoRa 868/915MHz | OLED 0.96" | LoRa mesh node |
|
|
||||||
| M5Stack Atom Lite | ESP32-PICO | 2 | WiFi + BLE | 24x24mm cube | Button/LED/Grove |
|
|
||||||
| Raspberry Pi Pico | RP2040 | 2 | USB only | Breadboard | MicroPython prototyping |
|
|
||||||
| ATTINY88 | AVR 8-bit | 3 | None (I2C/SPI slave) | DIP | Low-power peripherals |
|
|
||||||
| ELEGOO UNO R3 | ATmega328P | 2 | USB | Arduino form factor | Starter kit projects |
|
|
||||||
| WCH CH32V003 | RISC-V | 1 | USB | Minimal | Ultra-cheap RISC-V |
|
|
||||||
|
|
||||||
**Total MCUs: 21 units**
|
|
||||||
|
|
||||||
### Flashing Tools
|
|
||||||
- `esptool.py` / `espflash` for ESP32 family
|
|
||||||
- `arduino-cli` for Arduino/ATmega boards
|
|
||||||
- PlatformIO for cross-platform builds
|
|
||||||
- `picotool` for Pico RP2040
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 6. IoT & Sensor Devices
|
|
||||||
|
|
||||||
### SenseCAP Watcher W1-A
|
|
||||||
|
|
||||||
| Field | Value |
|
| Field | Value |
|
||||||
|-------|-------|
|
|-------|-------|
|
||||||
| Name | SenseCAP Watcher W1-A |
|
| Board | Raspberry Pi 5 Model B Rev 1.1 |
|
||||||
| Type | IoT AI Agent |
|
| OS | **Debian 13 (Trixie)** |
|
||||||
| Status | **Returned** (August 2025) |
|
| Kernel | 6.12.62+rpt-rpi-2712 |
|
||||||
| Processor | ESP32-S3 |
|
| RAM | 7.9GB total, 3.3GB used, 4.6GB available |
|
||||||
| AI Chip | Himax WiseEye2 HX6538 (Arm Cortex-M55 + Arm Ethos-U55 NPU) |
|
| Storage | /dev/nvme0n1p2 **457GB**, 65GB used (**15%**) |
|
||||||
| AI Compute | ~1 TOPS (Ethos-U55) |
|
| IP Local | 192.168.4.89 |
|
||||||
| Camera | Image recognition (person/animal/gesture detection) |
|
| IP Tailscale | 100.72.180.98 |
|
||||||
| Microphone | Voice-activated commands |
|
| MAC | 88:a2:9e:3b:eb:72 |
|
||||||
| Speaker | Audio output |
|
| Hailo-8 | **/dev/hailo0 DETECTED**, serial HLLWM2B233704667 |
|
||||||
| Touch | Capacitive touch interface |
|
| Uptime | 2h 3m (recently rebooted) |
|
||||||
| Connectivity | WiFi |
|
| Load | 3.40, 3.15, 3.48 |
|
||||||
| Features | On-device AI inference, SenseCraft AI, no-code workflows, OTA |
|
| SSH | `ssh cecilia` (user: blackroad) |
|
||||||
| Notes | Purchased and returned Aug 2025. Standalone edge AI unit with dedicated Himax coprocessor. Could be re-acquired for doorbell/monitor use case. |
|
|
||||||
|
|
||||||
### Sensor Inventory
|
**Services (systemd):**
|
||||||
|
- `hailort.service` — HailoRT AI runtime
|
||||||
|
- `ollama.service` — LLM inference (port 11434, localhost only)
|
||||||
|
- `cloudflared.service` — Cloudflare tunnel
|
||||||
|
- `docker.service` — Container runtime
|
||||||
|
|
||||||
| Sensor | Type | Interface | Attached To |
|
**Listening Ports:**
|
||||||
|--------|------|-----------|-------------|
|
|
||||||
| DHT22 | Temperature / Humidity | GPIO | Available |
|
|
||||||
| Radar (HLK-LD2410 / RCWL-0516) | Presence / Motion | GPIO/UART | Available |
|
|
||||||
| GPS Module | NMEA Location | UART | Available |
|
|
||||||
| ToF (VL53L0X / VL53L1X) | Distance (mm) | I2C | Available |
|
|
||||||
| AS7341 | Spectral 11-channel | I2C | Available |
|
|
||||||
| Pi Camera V2 | 8MP IMX219 | CSI | Available |
|
|
||||||
| USB + I2S MEMS Mics | Audio capture | USB / I2S | Available |
|
|
||||||
| Ultrasonic | Distance | GPIO | ELEGOO kit |
|
|
||||||
| PIR | Motion | GPIO | ELEGOO kit |
|
|
||||||
| Photoresistor | Light level | ADC | ELEGOO kit |
|
|
||||||
| IR Receiver | Remote control | GPIO | ELEGOO kit |
|
|
||||||
| Joystick | Analog input | ADC | ELEGOO kit |
|
|
||||||
|
|
||||||
### IoT Nodes (Unidentified)
|
| Port | Service | Bind |
|
||||||
|
|------|---------|------|
|
||||||
| Name | Platform | Status | Notes |
|
| 22 | SSH | 0.0.0.0 |
|
||||||
|------|----------|--------|-------|
|
| 53 | DNS | 0.0.0.0 |
|
||||||
| Calliope | Unknown IoT | Active | Registered in agent registry, needs identification |
|
| 80 | HTTP | 0.0.0.0 |
|
||||||
| Sophia | Unknown IoT | Active | Registered in agent registry, needs identification |
|
| 3001 | Python app | 0.0.0.0 |
|
||||||
|
| 3100 | Loki/log collector | 0.0.0.0 |
|
||||||
|
| 5001 | Python app | 0.0.0.0 |
|
||||||
|
| 5002 | Python app | 0.0.0.0 |
|
||||||
|
| 5432 | PostgreSQL | 127.0.0.1 |
|
||||||
|
| 5900 | VNC | 0.0.0.0 |
|
||||||
|
| 8086 | InfluxDB | 0.0.0.0 |
|
||||||
|
| 8787 | Python app | 0.0.0.0 |
|
||||||
|
| 9000 | MinIO | 0.0.0.0 + [::] |
|
||||||
|
| 9001 | MinIO Console | 0.0.0.0 |
|
||||||
|
| 9100 | Node Exporter | 0.0.0.0 |
|
||||||
|
| 11434 | Ollama | 127.0.0.1 |
|
||||||
|
| 34001 | Tailscale relay | 0.0.0.0 |
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 7. Consumer Devices
|
### Octavia — Heavy Services (VERIFIED)
|
||||||
|
|
||||||
| Name | Hardware | IP | Role | Notes |
|
| Field | Value |
|
||||||
|------|----------|-----|------|-------|
|
|-------|-------|
|
||||||
| Iris | Roku | 192.168.4.26 | Streaming | Media playback |
|
| Board | Raspberry Pi 5 Model B Rev 1.1 |
|
||||||
| Ares | Xbox | 192.168.4.90 | Gaming | Entertainment |
|
| OS | Debian 12 (Bookworm) |
|
||||||
| Phoebe | iPhone | 192.168.4.88 | Mobile | Monitoring, OOB access |
|
| Kernel | 6.12.62+rpt-rpi-2712 |
|
||||||
| Alexandria | MacBook Pro M1 8GB | 192.168.4.28 | Primary operator | Development, orchestration |
|
| RAM | 7.9GB total, **6.6GB used**, 1.3GB available |
|
||||||
| MacBook #1 | ~2014 Intel MacBook | — | Monitoring station | Secondary display |
|
| Storage | /dev/mmcblk0p2 **235GB**, 76GB used (**34%**) |
|
||||||
| MacBook #2 | ~2014 Intel MacBook | — | Agent orchestrator | Secondary display |
|
| IP Local | 192.168.4.38 |
|
||||||
| iPad Pro | 2015 iPad Pro | — | Tablet | Touch interface |
|
| IP Tailscale | **100.66.235.47** |
|
||||||
|
| MAC | 2c:cf:67:cf:fa:17 |
|
||||||
|
| Hailo-8 | **NONE** |
|
||||||
|
| Uptime | 2 days, 8h 40m |
|
||||||
|
| Load | **9.47**, 9.82, 10.52 (VERY HIGH) |
|
||||||
|
| SSH | `ssh octavia` (user: blackroad) |
|
||||||
|
|
||||||
|
**Services (systemd):**
|
||||||
|
- `ollama.service` — LLM inference
|
||||||
|
- `ollama-bridge.service` — SSE chat proxy
|
||||||
|
- `cloudflared.service` — Cloudflare tunnel
|
||||||
|
- `docker.service` — Container runtime
|
||||||
|
|
||||||
|
**Listening Ports (28+ services):**
|
||||||
|
|
||||||
|
| Port | Service | Bind |
|
||||||
|
|------|---------|------|
|
||||||
|
| 3002-3006 | App services | 0.0.0.0 |
|
||||||
|
| 3109 | App service | 0.0.0.0 |
|
||||||
|
| 4001-4002 | App services | 0.0.0.0 |
|
||||||
|
| 4010 | App service | 127.0.0.1 |
|
||||||
|
| 5200-5900 | Python microservices (7 ports) | 0.0.0.0 |
|
||||||
|
| 6000-6300 | Python microservices (4 ports) | 0.0.0.0 |
|
||||||
|
| 8000 | API (uvicorn/gunicorn) | 0.0.0.0 |
|
||||||
|
| 8011 | Python service | 0.0.0.0 |
|
||||||
|
| 8080-8082 | HTTP services | 0.0.0.0 |
|
||||||
|
| 8180 | Python service | 0.0.0.0 |
|
||||||
|
| 5432 | PostgreSQL | 127.0.0.1 |
|
||||||
|
| 11434 | Ollama | 127.0.0.1 |
|
||||||
|
| 34001 | Tailscale relay | 0.0.0.0 |
|
||||||
|
|
||||||
|
> **WARNING:** Load average 9.47 on a 4-core Pi 5. This node is overloaded.
|
||||||
|
> RAM 6.6/7.9GB. Consider migrating services.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 8. AI Accelerator Summary
|
### Aria — API Services (VERIFIED)
|
||||||
|
|
||||||
| Accelerator | Location | Architecture | Compute | Status |
|
| Field | Value |
|
||||||
|-------------|----------|--------------|---------|--------|
|
|-------|-------|
|
||||||
| Hailo-8 M.2 #1 | Cecilia | Hailo-8 (serial: HLLWM2B233704667) | 26 TOPS | Active |
|
| Board | Raspberry Pi 5 Model B Rev 1.1 |
|
||||||
| Hailo-8 M.2 #2 | Octavia | Hailo-8 (serial: HLLWM2B233704606) | 26 TOPS | Active |
|
| OS | Debian 12 (Bookworm) |
|
||||||
| Hailo-8 M.2 #3 | Aria | Hailo-8 M.2 | 26 TOPS | Active |
|
| Kernel | 6.12.62+rpt-rpi-2712 |
|
||||||
| Jetson Orin Nano | Jetson-Agent | NVIDIA Ampere GPU | 40 TOPS | Pending |
|
| RAM | 7.9GB total, 3.8GB used, 4.0GB available |
|
||||||
| Apple M1 Neural Engine | Alexandria | Apple Neural Engine | 15.8 TOPS | Active |
|
| Storage | /dev/mmcblk0p2 **29GB**, 20GB used (**74%**) |
|
||||||
| Himax Ethos-U55 | SenseCAP W1-A | Arm Ethos-U55 NPU | ~1 TOPS | Returned |
|
| IP Local | 192.168.4.82 |
|
||||||
|
| IP Tailscale | 100.109.14.17 |
|
||||||
|
| MAC | 88:a2:9e:0d:42:07 |
|
||||||
|
| Hailo-8 | **NONE** |
|
||||||
|
| Uptime | 3h 54m |
|
||||||
|
| Load | 0.45, 0.60, 0.68 |
|
||||||
|
| SSH | `ssh aria` (user: blackroad) |
|
||||||
|
|
||||||
### Total AI Compute Budget
|
**Services (systemd):**
|
||||||
|
- `ollama.service` — LLM inference
|
||||||
|
- `cloudflared.service` — Cloudflare tunnel
|
||||||
|
- `docker.service` — Container runtime
|
||||||
|
|
||||||
| Category | TOPS |
|
**Listening Ports (28+ services):**
|
||||||
|----------|------|
|
|
||||||
| Hailo-8 (3 units) | 78 |
|
|
||||||
| Jetson Orin Nano | 40 |
|
|
||||||
| Apple M1 Neural Engine | 15.8 |
|
|
||||||
| Ethos-U55 (returned) | ~1 |
|
|
||||||
| **Total (active)** | **~134 TOPS** |
|
|
||||||
| **Total (including returned/pending)** | **~135 TOPS** |
|
|
||||||
|
|
||||||
### Model Compatibility
|
| Port Range | Count | Service |
|
||||||
|
|------------|-------|---------|
|
||||||
|
| 3140-3167 | 28 | Docker container ports |
|
||||||
|
| 3153-3167 | 15 | (subset, unique services) |
|
||||||
|
| 8081 | 1 | HTTP service |
|
||||||
|
| 8180 | 1 | Python service |
|
||||||
|
|
||||||
| Model | Hailo-8 | Jetson | M1 |
|
> **WARNING:** 74% disk on 29GB. Only 7.3GB free. Needs storage upgrade or cleanup.
|
||||||
|-------|---------|--------|----|
|
|
||||||
| YOLOv5m | HEF compiled | TensorRT | CoreML |
|
|
||||||
| YOLOv8 | HEF compiled | TensorRT | CoreML |
|
|
||||||
| Llama 2 7B | — | CUDA | Ollama (Metal) |
|
|
||||||
| Whisper | — | CUDA | Ollama |
|
|
||||||
| ResNet-50 | HEF compiled | TensorRT | CoreML |
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 9. Network Topology
|
### Alice — Gateway (VERIFIED)
|
||||||
|
|
||||||
### LAN (192.168.4.0/24)
|
| Field | Value |
|
||||||
|
|-------|-------|
|
||||||
|
| Board | Raspberry Pi 400 Rev 1.0 |
|
||||||
|
| OS | **Raspbian 11 (Bullseye)** — NOT Bookworm |
|
||||||
|
| Kernel | **6.1.21-v8+** |
|
||||||
|
| RAM | 3.7GB total, 579MB used, 3.1GB available |
|
||||||
|
| Storage | /dev/root **15GB**, 9.6GB used (**71%**) |
|
||||||
|
| IP Local | 192.168.4.49 |
|
||||||
|
| IP Tailscale | 100.77.210.18 |
|
||||||
|
| MAC | d8:3a:dd:ff:98:87 |
|
||||||
|
| Hailo-8 | None |
|
||||||
|
| Uptime | 2 days, 6h 24m |
|
||||||
|
| Load | 6.17, 5.60, 5.56 (HIGH for 4 cores) |
|
||||||
|
| SSH | `ssh alice` (user: blackroad) |
|
||||||
|
|
||||||
```
|
**Services (systemd):**
|
||||||
┌──────────────┐
|
- `cloudflared.service` — Cloudflare tunnel
|
||||||
│ TP-Link │
|
- `docker.service` — Container runtime
|
||||||
│ Router/WiFi │
|
|
||||||
│ 192.168.4.1 │
|
|
||||||
└──────┬───────┘
|
|
||||||
│
|
|
||||||
┌──────────┴──────────┐
|
|
||||||
│ TP-Link TL-SG105 │
|
|
||||||
│ 5-Port Gigabit SW │
|
|
||||||
└┬────┬────┬────┬────┘
|
|
||||||
│ │ │ │
|
|
||||||
┌──────┘ │ │ └──────┐
|
|
||||||
│ │ │ │
|
|
||||||
┌────┴────┐ ┌───┴──┐ ┌┴────┐ ┌──┴──────┐
|
|
||||||
│ Cecilia │ │Lucia │ │Aria │ │ Octavia │
|
|
||||||
│ .89 │ │ .81 │ │ .82 │ │ .38 │
|
|
||||||
│ Hailo-8 │ │ NATS │ │Hail │ │ Hailo-8 │
|
|
||||||
└─────────┘ └──────┘ └─────┘ └─────────┘
|
|
||||||
|
|
||||||
WiFi:
|
> **WARNING:** Load average 6.17 on a Pi 400 (4-core). 71% disk. Consider upgrading OS to Bookworm.
|
||||||
┌─────────┐ ┌──────┐ ┌───────┐ ┌───────────┐
|
|
||||||
│ Alice │ │Anast.│ │Cordel.│ │Alexandria │
|
|
||||||
│ .49 │ │ .33 │ │ .27 │ │ .28 │
|
|
||||||
└─────────┘ └──────┘ └───────┘ └───────────┘
|
|
||||||
┌─────────┐ ┌──────┐ ┌───────┐ ┌───────────┐
|
|
||||||
│ Athena │ │Phoebe│ │ Ares │ │ Iris │
|
|
||||||
│ .45 │ │ .88 │ │ .90 │ │ .26 │
|
|
||||||
└─────────┘ └──────┘ └───────┘ └───────────┘
|
|
||||||
```
|
|
||||||
|
|
||||||
### Tailscale Mesh Overlay
|
|
||||||
|
|
||||||
| Node | Tailscale IP | Connected |
|
|
||||||
|------|-------------|-----------|
|
|
||||||
| Cecilia | 100.72.180.98 | Yes |
|
|
||||||
| Lucidia | 100.66.235.47 | Yes |
|
|
||||||
| Octavia | 100.83.149.86 | Yes |
|
|
||||||
| Aria | 100.109.14.17 | Yes |
|
|
||||||
| Alice | 100.77.210.18 | Yes |
|
|
||||||
| Codex-Infinity | 100.108.132.8 | Yes |
|
|
||||||
| Shellfish | 100.94.33.37 | Yes |
|
|
||||||
|
|
||||||
### DNS & Tunnels
|
|
||||||
|
|
||||||
- Cloudflare DNS: `blackroad.io` zone
|
|
||||||
- Cloudflare tunnels per node: `tunnel-{hostname}.blackroad.io`
|
|
||||||
- Headscale: self-hosted coordination on Alice (planned)
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 10. Storage Infrastructure
|
### Lucidia — DOWN
|
||||||
|
|
||||||
| Node | Type | Capacity | Interface | Used | Notes |
|
| Field | Value |
|
||||||
|------|------|----------|-----------|------|-------|
|
|-------|-------|
|
||||||
| Cecilia | NVMe M.2 | 500GB | PCIe | ~50% | Crucial P310 |
|
| Board | Raspberry Pi 5 (per registry) |
|
||||||
| Anastasia | NVMe M.2 | 1TB | PCIe (Pironman) | — | Crucial P310 |
|
| IP Local | 192.168.4.81 |
|
||||||
| Octavia | microSD | 235GB | SD slot | ~90% | Samsung EVO Select, needs cleanup |
|
| IP Tailscale | 100.83.149.86 |
|
||||||
| Lucidia | microSD | 117GB | SD slot | ~60% | Samsung EVO Select |
|
| MAC | **Not in ARP** (incomplete) |
|
||||||
| Alice | microSD | 32GB | SD slot | ~93% | Needs cleanup |
|
| Status | **UNREACHABLE** — ping fails, ARP incomplete |
|
||||||
| Aria | microSD | 29GB | SD slot | ~70% | Samsung EVO Select |
|
| Last Known | NATS bus, Ollama, edge-agent |
|
||||||
| Codex-Infinity | SSD | 78GB | Cloud block | ~40% | DigitalOcean |
|
|
||||||
| Shellfish | SSD | 25GB | Cloud block | ~50% | DigitalOcean |
|
> **ACTION REQUIRED:** Lucidia is down. Check power supply, SD card, network cable.
|
||||||
|
> This is the NATS event bus node — its absence may affect inter-node messaging.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 11. Power & Cooling
|
### Anastasia — SSH Closed
|
||||||
|
|
||||||
| Node | PSU | Watts | Cooling |
|
| Field | Value |
|
||||||
|------|-----|-------|---------|
|
|-------|-------|
|
||||||
| Pi 5 nodes (Cecilia, Lucidia, Aria, Octavia, Anastasia, Cordelia) | Geekworm 27W 5V/5A USB-C | 27W | Pironman dual-fan / ElectroCookie tower / Active cooler |
|
| Board | Raspberry Pi 5 (confirmed by MAC OUI 60:92:c8 = Pi 5) |
|
||||||
| Alice (Pi 400) | 5V/3A USB-C | 15W | Passive (built-in) |
|
| IP Local | 192.168.4.33 |
|
||||||
| Olympia (Pi 4B) | 5V/3A USB-C | 15W | Passive |
|
| MAC | 60:92:c8:11:cf:7c |
|
||||||
| Jetson Orin Nano | Barrel jack | 15W | Dev kit heatsink + fan |
|
| Ping | **Responds** |
|
||||||
| Pi Zero W | 5V/2A Micro USB | 10W | None |
|
| SSH | **Connection refused** (port 22 closed) |
|
||||||
| Displays | Various 5V wall adapters | 5-15W each | N/A |
|
| Status | Powered on but not provisioned for SSH access |
|
||||||
| DigitalOcean droplets | Cloud-managed | — | Cloud-managed |
|
|
||||||
|
|
||||||
### Total Power Budget (On-Premises)
|
> **ACTION REQUIRED:** SSH not configured. Needs keyboard/monitor access to enable SSH or re-flash SD.
|
||||||
|
|
||||||
| Category | Devices | Est. Draw |
|
|
||||||
|----------|---------|-----------|
|
|
||||||
| Pi 5 cluster (6) | Cecilia, Lucidia, Aria, Octavia, Anastasia, Cordelia | ~60W peak |
|
|
||||||
| Pi 400 + Pi 4B | Alice, Olympia | ~20W peak |
|
|
||||||
| Jetson Orin Nano | Jetson-Agent | ~15W peak |
|
|
||||||
| Displays (5) | Various | ~30W |
|
|
||||||
| Networking | Router + Switch | ~15W |
|
|
||||||
| Mac + peripherals | Alexandria | ~30W |
|
|
||||||
| **Total** | | **~170W peak** |
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 12. Display Inventory
|
### Cordelia — SSH Closed
|
||||||
|
|
||||||
| Size | Resolution | Model | Assigned To | Interface |
|
| Field | Value |
|
||||||
|------|-----------|-------|-------------|-----------|
|
|-------|-------|
|
||||||
| 10.1" | 1024x600 | ROADOM Touch IPS | Jetson-Agent | HDMI + USB touch |
|
| Board | Raspberry Pi 5 (confirmed by MAC OUI 6c:4a:85 = Pi 5) |
|
||||||
| 9.3" | 1600x600 | Waveshare Ultrawide | Pi-Ops (shared via HDMI switch) | HDMI |
|
| IP Local | 192.168.4.27 |
|
||||||
| 7" | 1024x600 | Waveshare Touch | Pi-Zero-Sim | HDMI + USB touch |
|
| MAC | 6c:4a:85:32:ae:72 |
|
||||||
| 4" | 720x720 | Waveshare Square | Pi-Holo | HDMI |
|
| Ping | **Responds** |
|
||||||
| 2.8" | 320x240 | ESP32 Touch TFT (x3) | ESP32 MCUs | SPI |
|
| SSH | **Connection refused** (port 22 closed) |
|
||||||
| 0.96" | 128x64 | OLED (x3) | Arduino / ESP32 | I2C |
|
| Status | Powered on but not provisioned for SSH access |
|
||||||
|
|
||||||
### Video Routing
|
> **ACTION REQUIRED:** Same as Anastasia — needs initial SSH setup.
|
||||||
|
|
||||||
- UGREEN HDMI Switch 5-in-1: shares 9.3" between Pi-Ops and Pi 400
|
|
||||||
- WAVLINK HDMI Splitter: clone Pi-Holo to second display
|
|
||||||
- WARRKY USB-C to HDMI (2-pack): Mac to display
|
|
||||||
- JSAUX Micro HDMI adapters: Pi to display
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 13. Management Tools
|
### Olympia — Offline
|
||||||
|
|
||||||
| Script | Location | Purpose |
|
| Field | Value |
|
||||||
|--------|----------|---------|
|
|-------|-------|
|
||||||
| `hardware.sh` | `~/hardware.sh` | Interactive fleet overview menu |
|
| Board | Raspberry Pi 4B (PiKVM) |
|
||||||
| `hailo.sh` | `~/hailo.sh` | Hailo-8 detection, benchmarks, inference |
|
| IP Local | pikvm.local (mDNS) |
|
||||||
| `mcus.sh` | `~/mcus.sh` | Microcontroller fleet status |
|
| SSH | `ssh root@pikvm.local` |
|
||||||
| `sensors.sh` | `~/sensors.sh` | Sensor inventory and live readings |
|
| Status | **Offline** — not on network |
|
||||||
| `espflash.sh` | `~/espflash.sh` | ESP32 flashing tool |
|
|
||||||
| `i2c.sh` | `~/i2c.sh` | I2C bus scanning |
|
|
||||||
| `lora.sh` | `~/lora.sh` | LoRa network tools |
|
|
||||||
| `blackroad-network-scan.sh` | `~/blackroad-network-scan.sh` | ARP + ping sweep + Tailscale status |
|
|
||||||
| `blackroad-network-discovery.sh` | `~/blackroad-network-discovery.sh` | SSH probe all devices |
|
|
||||||
| `pifleet.sh` | `~/pifleet.sh` | Pi-specific fleet management |
|
|
||||||
| `hardware-inventory.sh` | `hardware/scripts/hardware-inventory.sh` | Registry query + live scan (this repo) |
|
|
||||||
| `fleet-health-check.sh` | `hardware/scripts/fleet-health-check.sh` | Ping + port check (this repo) |
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 14. Provisioning Phases
|
## 3. Cloud Compute (Live Verified)
|
||||||
|
|
||||||
All nodes follow a 4-phase provisioning process:
|
### Codex-Infinity / "gematria" (159.65.43.12)
|
||||||
|
|
||||||
1. **Base Image** — Flash Debian 12, create `alexandria` user, deploy SSH keys, enable UFW
|
| Field | Value |
|
||||||
2. **Fleet Identity** — Install Tailscale, configure `/etc/hosts`, deploy SSH aliases, set MOTD banner
|
|-------|-------|
|
||||||
3. **Role Provisioning** — Install role-specific packages/services, deploy systemd units, configure Cloudflare tunnel
|
| Provider | DigitalOcean |
|
||||||
4. **Cloud Integration** — Deploy GitHub deploy key, register in fleet inventory, verify connectivity
|
| CPU | DO-Premium-AMD (1 vCPU) |
|
||||||
|
| OS | (Debian/Ubuntu based) |
|
||||||
|
| RAM | ~765MB |
|
||||||
|
| Storage | 25GB+ |
|
||||||
|
| Public IP | 159.65.43.12 |
|
||||||
|
| Tailscale IP | 100.108.132.8 |
|
||||||
|
| Actual Hostname | **gematria** |
|
||||||
|
| Uptime | 55+ days |
|
||||||
|
| SSH | `ssh gematria` (user: blackroad) |
|
||||||
|
| Root | `ssh blackroad-os-infinity-root` (user: root) |
|
||||||
|
|
||||||
See `~/blackroad-fleet.yaml` for the full provisioning spec.
|
**Services:**
|
||||||
|
- `ollama.service` — LLM inference (port 11434, public!)
|
||||||
|
- `nginx.service` — Reverse proxy (80, 443)
|
||||||
|
- `cloudflared.service` — Tunnel
|
||||||
|
- Caddy (port 2019 admin)
|
||||||
|
- Python app (8787)
|
||||||
|
- Custom app (8011)
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Appendix A: IP Address Registry
|
### Shellfish / "anastasia" (174.138.44.45)
|
||||||
|
|
||||||
### LAN (192.168.4.0/24)
|
| Field | Value |
|
||||||
|
|-------|-------|
|
||||||
|
| Provider | DigitalOcean |
|
||||||
|
| CPU | DO-Premium-AMD (1 vCPU) |
|
||||||
|
| OS | **CentOS Stream 9** (kernel 5.14.0-651.el9.x86_64) |
|
||||||
|
| RAM | 765MB total, 408MB used |
|
||||||
|
| Storage | 25GB, 15GB used (57%) |
|
||||||
|
| Public IP | 174.138.44.45 |
|
||||||
|
| Tailscale IP | 100.94.33.37 |
|
||||||
|
| Actual Hostname | **anastasia** |
|
||||||
|
| Uptime | **55 days** |
|
||||||
|
| SSH | `ssh anastasia` or `ssh cadence` (user: blackroad/shellfish) |
|
||||||
|
| Root | `ssh shellfish-root` (user: root) |
|
||||||
|
|
||||||
| IP | Hostname | Type |
|
**Services:**
|
||||||
|----|----------|------|
|
- `ollama.service` — LLM inference (port 11434, Tailscale-only at 100.64.0.1)
|
||||||
| 192.168.4.1 | Router | TP-Link |
|
- `nginx.service` — Reverse proxy (80)
|
||||||
| 192.168.4.26 | Iris | Roku |
|
- `cloudflared.service` — Tunnel
|
||||||
| 192.168.4.27 | Cordelia | Pi 5 |
|
- `docker.service` — Container runtime
|
||||||
| 192.168.4.28 | Alexandria | MacBook Pro M1 |
|
- uvicorn API (port 8000)
|
||||||
| 192.168.4.33 | Anastasia | Pi 5 |
|
- WebSocket servers (8765, 8766)
|
||||||
| 192.168.4.38 | Octavia | Pi 5 |
|
- Redis-like (6379)
|
||||||
| 192.168.4.45 | Athena | Heltec LoRa ESP32 |
|
- Grafana/dashboard (3000, 3001)
|
||||||
| 192.168.4.49 | Alice | Pi 400 |
|
- Python apps (8080, 8787, 8888)
|
||||||
| 192.168.4.81 | Lucidia | Pi 5 |
|
|
||||||
| 192.168.4.82 | Aria | Pi 5 |
|
|
||||||
| 192.168.4.88 | Phoebe | iPhone |
|
|
||||||
| 192.168.4.89 | Cecilia | Pi 5 |
|
|
||||||
| 192.168.4.90 | Ares | Xbox |
|
|
||||||
|
|
||||||
### Cloud
|
> **NAMING CONFUSION:** This droplet's hostname is "anastasia" which collides
|
||||||
|
> with the Pi 5 at 192.168.4.33 also named Anastasia. The SSH alias `anastasia`
|
||||||
|
> points to the DO droplet (174.138.44.45), NOT the Pi.
|
||||||
|
|
||||||
| IP | Hostname | Provider |
|
---
|
||||||
|----|----------|----------|
|
|
||||||
| 159.65.43.12 | Codex-Infinity | DigitalOcean |
|
|
||||||
| 174.138.44.45 | Shellfish | DigitalOcean |
|
|
||||||
|
|
||||||
### Tailscale (100.x.x.x)
|
## 4. Unidentified Network Devices
|
||||||
|
|
||||||
| IP | Hostname |
|
Four devices discovered on the LAN with no agent registry entry.
|
||||||
|----|----------|
|
|
||||||
| 100.66.235.47 | Lucidia |
|
| IP | MAC Address | OUI Vendor | Ping | Ports | Best Guess |
|
||||||
| 100.72.180.98 | Cecilia |
|
|----|-------------|-----------|------|-------|------------|
|
||||||
| 100.77.210.18 | Alice |
|
| 192.168.4.22 | 30:be:29:5b:24:5f | Unknown (possibly Hisense) | **UP** | No common ports open | Smart TV or IoT device |
|
||||||
| 100.83.149.86 | Octavia |
|
| 192.168.4.44 | 98:17:3c:38:db:78 | **TP-Link** | **UP** | No common ports open | WiFi extender or smart plug |
|
||||||
| 100.94.33.37 | Shellfish |
|
| 192.168.4.83 | 54:4c:8a:9b:09:3d | Unknown (Shenzhen Bilian) | **UP** | No common ports open | Smart home WiFi module |
|
||||||
| 100.108.132.8 | Codex-Infinity |
|
| 192.168.4.92 | de:a2:b7:f3:f9:5d | Locally administered (Apple) | **DOWN** | — | Apple device with private WiFi MAC |
|
||||||
| 100.109.14.17 | Aria |
|
|
||||||
|
> **ACTION:** Identify .22, .44, .83 by physical inspection or DHCP lease table on router.
|
||||||
|
> Could be Calliope and Sophia from agent registry, plus a network accessory.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. AI Accelerator Summary — Corrected
|
||||||
|
|
||||||
|
| Accelerator | Node | Verified Method | TOPS | Status |
|
||||||
|
|-------------|------|----------------|------|--------|
|
||||||
|
| Hailo-8 M.2 | Cecilia | `/dev/hailo0` + `hailort.service` | 26 | **CONFIRMED active** |
|
||||||
|
| Hailo-8 M.2 | Octavia | SSH probe: `HAILO: none` | 26 | **NOT INSTALLED** |
|
||||||
|
| Hailo-8 M.2 | Aria | SSH probe: `HAILO: none` | 26 | **NOT INSTALLED** |
|
||||||
|
| Jetson Orin Nano | Jetson-Agent | Not deployed | 40 | Pending |
|
||||||
|
| Apple M1 NE | Alexandria | Known hardware | 15.8 | Active |
|
||||||
|
| Ethos-U55 | SenseCAP W1-A | Returned | ~1 | Returned |
|
||||||
|
|
||||||
|
### Corrected Compute Budget
|
||||||
|
|
||||||
|
| Category | TOPS | Status |
|
||||||
|
|----------|------|--------|
|
||||||
|
| Hailo-8 (1x confirmed) | 26 | **Active** |
|
||||||
|
| Apple M1 Neural Engine | 15.8 | Active |
|
||||||
|
| **Total confirmed active** | **41.8** | |
|
||||||
|
| Hailo-8 (2x uninstalled) | 52 | Available hardware, not installed |
|
||||||
|
| Jetson Orin Nano | 40 | Pending setup |
|
||||||
|
| **Total potential** | **~134** | If all installed |
|
||||||
|
|
||||||
|
> **Where are the other 2 Hailo-8 modules?** They were purchased ($215 each) but
|
||||||
|
> are not detected on Octavia or Aria. Check if they're physically seated in M.2
|
||||||
|
> slots or sitting uninstalled. Serials: HLLWM2B233704667 (Cecilia), HLLWM2B233704606 (unknown).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. Network — Live ARP Table
|
||||||
|
|
||||||
|
Devices with confirmed MAC addresses as of 2026-02-21:
|
||||||
|
|
||||||
|
| IP | MAC | OUI | Hostname | Status |
|
||||||
|
|----|-----|-----|----------|--------|
|
||||||
|
| 192.168.4.1 | 44:ac:85:94:37:92 | TP-Link | Router | **UP** |
|
||||||
|
| 192.168.4.22 | 30:be:29:5b:24:5f | Unknown | **UNIDENTIFIED** | **UP** |
|
||||||
|
| 192.168.4.26 | d4:be:dc:6c:61:6b | Roku | Iris | **UP** |
|
||||||
|
| 192.168.4.27 | 6c:4a:85:32:ae:72 | Raspberry Pi 5 | Cordelia | **UP** (no SSH) |
|
||||||
|
| 192.168.4.28 | b0:be:83:66:cc:10 | Apple | Alexandria (Mac) | **UP** |
|
||||||
|
| 192.168.4.33 | 60:92:c8:11:cf:7c | Raspberry Pi 5 | Anastasia (Pi) | **UP** (no SSH) |
|
||||||
|
| 192.168.4.38 | 2c:cf:67:cf:fa:17 | Raspberry Pi | Octavia | **UP** |
|
||||||
|
| 192.168.4.44 | 98:17:3c:38:db:78 | TP-Link | **UNIDENTIFIED** | **UP** |
|
||||||
|
| 192.168.4.45 | d0:c9:07:50:51:ca | Espressif | Athena (ESP32) | **UP** |
|
||||||
|
| 192.168.4.49 | d8:3a:dd:ff:98:87 | Raspberry Pi | Alice | **UP** |
|
||||||
|
| 192.168.4.81 | (incomplete) | — | Lucidia | **DOWN** |
|
||||||
|
| 192.168.4.82 | 88:a2:9e:0d:42:07 | Raspberry Pi 5 | Aria | **UP** |
|
||||||
|
| 192.168.4.83 | 54:4c:8a:9b:09:3d | Unknown | **UNIDENTIFIED** | **UP** |
|
||||||
|
| 192.168.4.88 | 9e:0d:2a:82:99:96 | Private MAC | Phoebe (iPhone) | **DOWN** |
|
||||||
|
| 192.168.4.89 | 88:a2:9e:3b:eb:72 | Raspberry Pi 5 | Cecilia | **UP** |
|
||||||
|
| 192.168.4.90 | a0:4a:5e:2a:db:d2 | Microsoft | Ares (Xbox) | **DOWN** |
|
||||||
|
| 192.168.4.92 | de:a2:b7:f3:f9:5d | Private MAC | **UNIDENTIFIED** | **DOWN** |
|
||||||
|
|
||||||
|
### Stale Entry
|
||||||
|
|
||||||
|
| IP | Note |
|
||||||
|
|----|------|
|
||||||
|
| 192.168.4.74 | In `/etc/hosts` as "octavia" — **stale**. Octavia is now at .38. Remove. |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7. Tailscale Mesh — Corrected
|
||||||
|
|
||||||
|
| Node | Tailscale IP | SSH Alias | Verified |
|
||||||
|
|------|-------------|-----------|----------|
|
||||||
|
| Cecilia | 100.72.180.98 | cecilia-ts | SSH config |
|
||||||
|
| Lucidia | **100.83.149.86** | lucidia-ts | SSH config (was wrongly documented as .66.235.47) |
|
||||||
|
| Octavia | **100.66.235.47** | octavia-ts | SSH config + ss binding (was wrongly documented as .83.149.86) |
|
||||||
|
| Aria | 100.109.14.17 | aria-ts | SSH config |
|
||||||
|
| Alice | 100.77.210.18 | alice-ts | SSH config |
|
||||||
|
| Codex-Infinity | 100.108.132.8 | gematria-ts | SSH config |
|
||||||
|
| Shellfish | 100.94.33.37 | anastasia-ts / cadence-ts | SSH config |
|
||||||
|
|
||||||
|
> **Note:** Tailscale daemon is NOT running on Alexandria (Mac). `tailscale status` returns "not running".
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 8. DNS — Cloudflare Proxied
|
||||||
|
|
||||||
|
All `blackroad.io` DNS resolves to Cloudflare proxy IPs (not origin):
|
||||||
|
|
||||||
|
| Subdomain | Resolves To | Type |
|
||||||
|
|-----------|------------|------|
|
||||||
|
| blackroad.io | 172.67.211.99 | Cloudflare proxy |
|
||||||
|
| www.blackroad.io | 172.67.211.99 | Cloudflare proxy |
|
||||||
|
| api.blackroad.io | 172.67.211.99 | Cloudflare proxy |
|
||||||
|
| status.blackroad.io | 172.67.211.99 | Cloudflare proxy |
|
||||||
|
| docs.blackroad.io | 172.67.211.99 | Cloudflare proxy |
|
||||||
|
| dashboard.blackroad.io | 172.67.211.99 | Cloudflare proxy |
|
||||||
|
| agents.blackroad.io | 104.21.91.74 | Cloudflare proxy |
|
||||||
|
| monitoring.blackroad.io | 172.67.211.99 | Cloudflare proxy |
|
||||||
|
| tunnel-cecilia.blackroad.io | 172.67.211.99 | Cloudflare proxy |
|
||||||
|
| tunnel-lucidia.blackroad.io | 104.21.91.74 | Cloudflare proxy |
|
||||||
|
| tunnel-octavia.blackroad.io | 172.67.211.99 | Cloudflare proxy |
|
||||||
|
| tunnel-codex.blackroad.io | 104.21.91.74 | Cloudflare proxy |
|
||||||
|
| tunnel-cadence.blackroad.io | 172.67.211.99 | Cloudflare proxy |
|
||||||
|
|
||||||
|
All traffic routes: Client → Cloudflare CDN → Cloudflare Tunnel → Origin Node
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 9. SSH Configuration Truth Table
|
||||||
|
|
||||||
|
From `~/.ssh/config` (hardened 2026-02-19):
|
||||||
|
|
||||||
|
| Alias | HostName | User | Notes |
|
||||||
|
|-------|----------|------|-------|
|
||||||
|
| cecilia | 192.168.4.89 | blackroad | ed25519 key |
|
||||||
|
| lucidia | 192.168.4.81 | blackroad | ed25519 key |
|
||||||
|
| aria | 192.168.4.82 | blackroad | ed25519 key |
|
||||||
|
| octavia | 192.168.4.38 | blackroad | ed25519 key |
|
||||||
|
| alice | 192.168.4.49 | blackroad | ed25519 key |
|
||||||
|
| anastasia | **174.138.44.45** | blackroad | **Points to DO droplet, NOT the Pi!** |
|
||||||
|
| gematria | 159.65.43.12 | blackroad | Codex-Infinity droplet |
|
||||||
|
| cadence | 174.138.44.45 | shellfish | Same host as anastasia alias |
|
||||||
|
| olympia | pikvm.local | root | mDNS, not IP |
|
||||||
|
| alexandria / mac | 192.168.4.28 | alexa | Local Mac |
|
||||||
|
| lucidia-pi | 192.168.4.81 | pi | Legacy fallback user |
|
||||||
|
| *-ts | 100.x.x.x | blackroad | Tailscale aliases |
|
||||||
|
| *-root | DO IPs | root | Root access to droplets |
|
||||||
|
|
||||||
|
### Missing SSH Entries
|
||||||
|
|
||||||
|
- **Cordelia** (192.168.4.27) — no SSH config entry
|
||||||
|
- **Anastasia Pi** (192.168.4.33) — alias `anastasia` points to DO droplet instead
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 10. Storage — Live Verified
|
||||||
|
|
||||||
|
| Node | Device | Total | Used | Free | % | Verified |
|
||||||
|
|------|--------|-------|------|------|---|----------|
|
||||||
|
| Cecilia | /dev/nvme0n1p2 | 457GB | 65GB | 370GB | **15%** | SSH |
|
||||||
|
| Octavia | /dev/mmcblk0p2 | 235GB | 76GB | 148GB | **34%** | SSH |
|
||||||
|
| Aria | /dev/mmcblk0p2 | 29GB | 20GB | 7.3GB | **74%** | SSH |
|
||||||
|
| Alice | /dev/root | 15GB | 9.6GB | 4.1GB | **71%** | SSH |
|
||||||
|
| Shellfish | /dev/vda1 | 25GB | 15GB | 11GB | **57%** | SSH |
|
||||||
|
| Lucidia | — | — | — | — | — | DOWN |
|
||||||
|
| Anastasia Pi | — | — | — | — | — | No SSH |
|
||||||
|
| Cordelia | — | — | — | — | — | No SSH |
|
||||||
|
|
||||||
|
### Storage Alerts
|
||||||
|
|
||||||
|
| Priority | Node | Issue |
|
||||||
|
|----------|------|-------|
|
||||||
|
| High | Aria | 74% used, only 7.3GB free on 29GB card |
|
||||||
|
| Medium | Alice | 71% used, only 4.1GB free on 15GB root |
|
||||||
|
| Monitor | Shellfish | 57% used |
|
||||||
|
| OK | Cecilia | 15% used — healthiest node |
|
||||||
|
| OK | Octavia | 34% used — cleaned up from prior 90% |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 11. OS Version Matrix
|
||||||
|
|
||||||
|
| Node | Distribution | Version | Kernel | Architecture |
|
||||||
|
|------|-------------|---------|--------|-------------|
|
||||||
|
| Cecilia | Debian | **13 (Trixie)** | 6.12.62+rpt-rpi-2712 | aarch64 |
|
||||||
|
| Octavia | Debian | 12 (Bookworm) | 6.12.62+rpt-rpi-2712 | aarch64 |
|
||||||
|
| Aria | Debian | 12 (Bookworm) | 6.12.62+rpt-rpi-2712 | aarch64 |
|
||||||
|
| Alice | **Raspbian** | **11 (Bullseye)** | **6.1.21-v8+** | aarch64 |
|
||||||
|
| Shellfish | **CentOS Stream** | **9** | 5.14.0-651.el9.x86_64 | x86_64 |
|
||||||
|
| Codex-Infinity | Unknown | — | — | x86_64 (DO-Premium-AMD) |
|
||||||
|
| Lucidia | (down) | — | — | — |
|
||||||
|
| Anastasia Pi | (no SSH) | — | — | — |
|
||||||
|
| Cordelia | (no SSH) | — | — | — |
|
||||||
|
|
||||||
|
> **Note:** The fleet is NOT uniform. Three different OS families and kernels in play.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 12. Action Items
|
||||||
|
|
||||||
|
### Critical
|
||||||
|
|
||||||
|
1. **Investigate Lucidia** — Node is down. Check power, SD card, Ethernet. NATS bus may be affected.
|
||||||
|
2. **Locate 2 Hailo-8 modules** — Purchased but not detected on Octavia or Aria. Physical check needed.
|
||||||
|
3. **Fix Anastasia naming collision** — DO droplet hostname "anastasia" collides with Pi at .33. Rename droplet to "shellfish" or "cadence".
|
||||||
|
|
||||||
|
### High
|
||||||
|
|
||||||
|
4. **Enable SSH on Anastasia Pi** (.33) — Port 22 closed. Needs keyboard access to `sudo systemctl enable ssh`.
|
||||||
|
5. **Enable SSH on Cordelia** (.27) — Same issue.
|
||||||
|
6. **Add Cordelia to SSH config** — No entry exists.
|
||||||
|
7. **Fix SSH config** — `anastasia` alias should point to Pi (.33), not DO droplet.
|
||||||
|
8. **Reduce Octavia load** — Load avg 9.47 on 4-core, 6.6/7.9GB RAM. Migrate services.
|
||||||
|
|
||||||
|
### Medium
|
||||||
|
|
||||||
|
9. **Identify unknown devices** — .22, .44, .83 on the network. Check router DHCP leases.
|
||||||
|
10. **Clean up Aria storage** — 74% used, 7.3GB free.
|
||||||
|
11. **Remove stale /etc/hosts** — `192.168.4.74 octavia` is wrong (now .38).
|
||||||
|
12. **Upgrade Alice OS** — Bullseye (11) is EOL. Upgrade to Bookworm (12).
|
||||||
|
13. **Install Tailscale on Mac** — `tailscale status` shows "not running" on Alexandria.
|
||||||
|
14. **Add Anastasia + Cordelia to Tailscale** — Not in mesh yet.
|
||||||
|
15. **Correct agent registry** — Octavia and Aria listed as `pironman_hailo8` but have no Hailo.
|
||||||
|
16. **Fix `~/blackroad-fleet.yaml`** — Lucidia/Octavia IPs are swapped (both local and Tailscale).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Appendix: Data Sources
|
||||||
|
|
||||||
|
| Source | Method | Trust Level |
|
||||||
|
|--------|--------|-------------|
|
||||||
|
| SSH probe (system info) | `ssh <host> "hostname; uname -r; ..."` | **Highest** — live system state |
|
||||||
|
| ARP table | `arp -a` | **High** — recent MAC-to-IP mappings |
|
||||||
|
| Ping sweep | `ping -c 1 -W 1` | **High** — reachability |
|
||||||
|
| Port scan | `ss -tlnp` via SSH | **Highest** — actual listening services |
|
||||||
|
| `~/.ssh/config` | File read | **High** — operational SSH aliases |
|
||||||
|
| DNS dig | `dig +short` | **High** — current DNS state |
|
||||||
|
| Agent registry DB | SQLite query | **Medium** — may be stale |
|
||||||
|
| `~/blackroad-fleet.yaml` | File read | **Low** — contains known errors (IPs swapped) |
|
||||||
|
| Prior documentation | Various .md files | **Low** — multiple inaccuracies found |
|
||||||
|
|||||||
@@ -1,34 +1,75 @@
|
|||||||
# AI Compute Accelerators
|
# AI Compute Accelerators — Live Verified
|
||||||
|
|
||||||
Total fleet AI compute: **~135 TOPS** across Hailo-8, NVIDIA Jetson, Apple M1, and Arm Ethos-U55.
|
**Verified via SSH probes on 2026-02-21.**
|
||||||
|
|
||||||
|
> **CRITICAL CORRECTION:** Only **1 of 3** Hailo-8 modules is confirmed active.
|
||||||
|
> Prior documentation claimed 3 active Hailo-8 units (78 TOPS). Live probes confirm
|
||||||
|
> only Cecilia has a working Hailo-8. Octavia and Aria both report no Hailo device.
|
||||||
|
|
||||||
|
**Confirmed fleet AI compute: ~41.8 TOPS active** (not 135 TOPS as previously documented)
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Accelerator Inventory
|
## Accelerator Inventory
|
||||||
|
|
||||||
| # | Accelerator | Node | Architecture | TOPS | Interface | Serial | Status |
|
| # | Accelerator | Node | TOPS | Interface | Status | Verification |
|
||||||
|---|-------------|------|-------------|------|-----------|--------|--------|
|
|---|-------------|------|------|-----------|--------|-------------|
|
||||||
| 1 | Hailo-8 M.2 | Cecilia | Hailo-8 | 26 | M.2 PCIe | HLLWM2B233704667 | Active |
|
| 1 | Hailo-8 M.2 | Cecilia | 26 | M.2 PCIe | **Active** | `hailort.service` running, `/dev/hailo0` present |
|
||||||
| 2 | Hailo-8 M.2 | Octavia | Hailo-8 | 26 | M.2 PCIe | HLLWM2B233704606 | Active |
|
| 2 | Hailo-8 M.2 | Octavia | 26 | M.2 PCIe | **NOT DETECTED** | No `/dev/hailo*`, no `hailort.service` |
|
||||||
| 3 | Hailo-8 M.2 | Aria | Hailo-8 | 26 | M.2 PCIe | — | Active |
|
| 3 | Hailo-8 M.2 | Aria | 26 | M.2 PCIe | **NOT DETECTED** | No `/dev/hailo*`, no `hailort.service` |
|
||||||
| 4 | Jetson Orin Nano GPU | Jetson-Agent | NVIDIA Ampere | 40 | Onboard | — | Pending |
|
| 4 | Jetson Orin Nano GPU | Jetson-Agent | 40 | Onboard | **Pending** | Dev kit not deployed |
|
||||||
| 5 | Apple M1 Neural Engine | Alexandria | Apple NE | 15.8 | Onboard | — | Active |
|
| 5 | Apple M1 Neural Engine | Alexandria | 15.8 | Onboard | **Active** | Mac in use daily |
|
||||||
| 6 | Himax Ethos-U55 NPU | SenseCAP W1-A | Arm Ethos-U55 | ~1 | Onboard | — | Returned |
|
| 6 | Himax Ethos-U55 NPU | SenseCAP W1-A | ~1 | Onboard | **Returned** | Returned Aug 2025 |
|
||||||
|
|
||||||
### Compute Budget
|
### Compute Budget — Corrected
|
||||||
|
|
||||||
| Category | TOPS | Status |
|
| Category | TOPS | Status | Notes |
|
||||||
|----------|------|--------|
|
|----------|------|--------|-------|
|
||||||
| Hailo-8 (3x) | 78 | Active |
|
| Hailo-8 (1x confirmed) | 26 | **Active** | Cecilia only |
|
||||||
| NVIDIA Jetson Orin Nano | 40 | Pending setup |
|
| Hailo-8 (2x unverified) | 52 | **Unknown** | Purchased but not detected on Octavia/Aria |
|
||||||
| Apple M1 Neural Engine | 15.8 | Active |
|
| NVIDIA Jetson Orin Nano | 40 | **Pending** | Dev kit not deployed |
|
||||||
| Arm Ethos-U55 | ~1 | Returned |
|
| Apple M1 Neural Engine | 15.8 | **Active** | Alexandria Mac |
|
||||||
| **Total Active** | **93.8** | |
|
| Arm Ethos-U55 | ~1 | **Returned** | SenseCAP Watcher |
|
||||||
| **Total (incl. pending)** | **~135** | |
|
| **Confirmed Active** | **41.8** | | Hailo-8 (Cecilia) + M1 |
|
||||||
|
| **Potential (if all working)** | **~135** | | Requires physical verification |
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Hailo-8 M.2 Modules (3 units)
|
## Missing Hailo-8 Investigation
|
||||||
|
|
||||||
|
3 Hailo-8 M.2 modules were purchased (serial numbers documented: HLLWM2B233704667, HLLWM2B233704606, third unknown). Only 1 is confirmed active on Cecilia.
|
||||||
|
|
||||||
|
### Possible Explanations
|
||||||
|
|
||||||
|
1. **Not physically installed** — M.2 modules may still be in packaging or stored separately
|
||||||
|
2. **Installed but no drivers** — HailoRT runtime not installed on Octavia/Aria
|
||||||
|
3. **Hardware fault** — M.2 slot or module not functioning
|
||||||
|
4. **Wrong slot** — Pironman case M.2 slot may be configured for NVMe, not AI accelerator
|
||||||
|
|
||||||
|
### Verification Steps
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# On Octavia (ssh octavia):
|
||||||
|
ls /dev/hailo* # Check for Hailo device nodes
|
||||||
|
systemctl status hailort # Check for Hailo runtime service
|
||||||
|
lspci | grep -i hailo # Check PCIe bus for Hailo device
|
||||||
|
dpkg -l | grep hailo # Check if HailoRT packages installed
|
||||||
|
|
||||||
|
# On Aria (ssh aria):
|
||||||
|
ls /dev/hailo*
|
||||||
|
systemctl status hailort
|
||||||
|
lspci | grep -i hailo
|
||||||
|
dpkg -l | grep hailo
|
||||||
|
|
||||||
|
# Physical inspection required:
|
||||||
|
# 1. Open Pironman cases on Octavia and Aria
|
||||||
|
# 2. Check M.2 Key M slot — is a Hailo-8 card present?
|
||||||
|
# 3. If present, install HailoRT: sudo apt install hailort
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Hailo-8 M.2 Module
|
||||||
|
|
||||||
### Specifications
|
### Specifications
|
||||||
|
|
||||||
@@ -38,7 +79,7 @@ Total fleet AI compute: **~135 TOPS** across Hailo-8, NVIDIA Jetson, Apple M1, a
|
|||||||
| Compute | 26 TOPS (INT8) |
|
| Compute | 26 TOPS (INT8) |
|
||||||
| Interface | M.2 Key M (PCIe Gen 3.0 x1) |
|
| Interface | M.2 Key M (PCIe Gen 3.0 x1) |
|
||||||
| Power | ~2.5W typical |
|
| Power | ~2.5W typical |
|
||||||
| Price | $214.99 each |
|
| Price | $214.99 each (3x = $644.97 total) |
|
||||||
| Compatible Hosts | Raspberry Pi 5 (via HAT), Pironman case |
|
| Compatible Hosts | Raspberry Pi 5 (via HAT), Pironman case |
|
||||||
|
|
||||||
### Software Stack
|
### Software Stack
|
||||||
@@ -70,7 +111,7 @@ dpkg -l | grep hailo
|
|||||||
~/hailo.sh
|
~/hailo.sh
|
||||||
```
|
```
|
||||||
|
|
||||||
### Benchmark Results
|
### Benchmark Results (Cecilia only)
|
||||||
|
|
||||||
Hailo-8 vs NVIDIA Jetson benchmarks (from BlackRoad testing):
|
Hailo-8 vs NVIDIA Jetson benchmarks (from BlackRoad testing):
|
||||||
- **Power Efficiency:** 15-30x more efficient than NVIDIA Jetson (TOPS/Watt)
|
- **Power Efficiency:** 15-30x more efficient than NVIDIA Jetson (TOPS/Watt)
|
||||||
@@ -90,6 +131,22 @@ Hailo-8 vs NVIDIA Jetson benchmarks (from BlackRoad testing):
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
## Ollama Deployment (4 nodes)
|
||||||
|
|
||||||
|
Ollama runs on 4 of 6 reachable nodes, providing LLM inference across the fleet:
|
||||||
|
|
||||||
|
| Node | Binding | Security | Status |
|
||||||
|
|------|---------|----------|--------|
|
||||||
|
| Cecilia | 127.0.0.1:11434 | Localhost only | **Secure** |
|
||||||
|
| Octavia | 127.0.0.1:11434 | Localhost only | **Secure** |
|
||||||
|
| Shellfish | 100.64.0.1:11434 | Tailscale interface | **Secure** |
|
||||||
|
| Codex-Infinity | **0.0.0.0:11434** | **ALL INTERFACES** | **INSECURE** |
|
||||||
|
|
||||||
|
> **ACTION:** Fix Codex-Infinity Ollama binding immediately. Public IP 159.65.43.12:11434 is
|
||||||
|
> accessible to anyone on the internet.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
## NVIDIA Jetson Orin Nano
|
## NVIDIA Jetson Orin Nano
|
||||||
|
|
||||||
### Specifications
|
### Specifications
|
||||||
@@ -104,6 +161,7 @@ Hailo-8 vs NVIDIA Jetson benchmarks (from BlackRoad testing):
|
|||||||
| Power | 7-15W configurable TDP |
|
| Power | 7-15W configurable TDP |
|
||||||
| Price | $114.29 (base dev kit) |
|
| Price | $114.29 (base dev kit) |
|
||||||
| Display | HDMI + DisplayPort |
|
| Display | HDMI + DisplayPort |
|
||||||
|
| Status | **Pending initial setup** |
|
||||||
|
|
||||||
### Software Stack
|
### Software Stack
|
||||||
|
|
||||||
@@ -122,52 +180,40 @@ Hailo-8 vs NVIDIA Jetson benchmarks (from BlackRoad testing):
|
|||||||
| Image generation | Stable Diffusion | Small models only (8GB RAM) |
|
| Image generation | Stable Diffusion | Small models only (8GB RAM) |
|
||||||
| Video analytics | DeepStream | Multi-stream pipeline |
|
| Video analytics | DeepStream | Multi-stream pipeline |
|
||||||
|
|
||||||
### Status
|
|
||||||
|
|
||||||
Pending initial setup. Dev kit available with 10.1" ROADOM touchscreen.
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Apple M1 Neural Engine
|
## Apple M1 Neural Engine
|
||||||
|
|
||||||
### Specifications
|
|
||||||
|
|
||||||
| Spec | Value |
|
| Spec | Value |
|
||||||
|------|-------|
|
|------|-------|
|
||||||
| Architecture | Apple Neural Engine (16-core) |
|
| Architecture | Apple Neural Engine (16-core) |
|
||||||
| AI Compute | 15.8 TOPS |
|
| AI Compute | 15.8 TOPS |
|
||||||
| Host | MacBook Pro M1 (Alexandria) |
|
| Host | MacBook Pro M1 (Alexandria) |
|
||||||
| Framework | CoreML, MLX |
|
| Framework | CoreML, MLX |
|
||||||
| Power | Integrated (shared power budget) |
|
| Status | **Active** (daily use) |
|
||||||
|
|
||||||
### Capabilities
|
|
||||||
|
|
||||||
- CoreML model inference (Vision, NLP, Audio)
|
|
||||||
- Ollama via Metal GPU acceleration
|
|
||||||
- MLX framework for on-device ML
|
|
||||||
- Whisper transcription
|
|
||||||
- Stable Diffusion (via MLX)
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Arm Ethos-U55 NPU (SenseCAP Watcher — Returned)
|
## Arm Ethos-U55 NPU (SenseCAP Watcher — Returned)
|
||||||
|
|
||||||
### Specifications
|
|
||||||
|
|
||||||
| Spec | Value |
|
| Spec | Value |
|
||||||
|------|-------|
|
|------|-------|
|
||||||
| Architecture | Arm Ethos-U55 microNPU |
|
| Architecture | Arm Ethos-U55 microNPU |
|
||||||
| Host Processor | Arm Cortex-M55 (Himax HX6538) |
|
| Host Processor | Arm Cortex-M55 (Himax HX6538) |
|
||||||
| AI Compute | ~1 TOPS (INT8) |
|
| AI Compute | ~1 TOPS (INT8) |
|
||||||
| Device | SenseCAP Watcher W1-A |
|
| Device | SenseCAP Watcher W1-A |
|
||||||
| Status | Returned (August 2025) |
|
| Status | **Returned** (August 2025) |
|
||||||
|
|
||||||
### Capabilities (When Active)
|
---
|
||||||
|
|
||||||
- Person/animal/gesture detection via camera
|
## Power Efficiency Comparison
|
||||||
- Low-power always-on vision inference
|
|
||||||
- Voice keyword detection
|
| Accelerator | TOPS | Power (W) | TOPS/W | Status |
|
||||||
- Designed for battery-powered edge AI
|
|-------------|------|-----------|--------|--------|
|
||||||
|
| Hailo-8 | 26 | 2.5 | **10.4** | 1 active, 2 unverified |
|
||||||
|
| Jetson Orin Nano | 40 | 15 | 2.7 | Pending setup |
|
||||||
|
| M1 Neural Engine | 15.8 | ~5 | 3.2 | Active |
|
||||||
|
| Ethos-U55 | ~1 | 0.05 | 20.0 | Returned |
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -183,14 +229,3 @@ Pending initial setup. Dev kit available with 10.1" ROADOM touchscreen.
|
|||||||
| Whisper | — | Yes (CUDA) | Yes (Metal) | — |
|
| Whisper | — | Yes (CUDA) | Yes (Metal) | — |
|
||||||
| Stable Diffusion | — | Yes (limited) | Yes (MLX) | — |
|
| Stable Diffusion | — | Yes (limited) | Yes (MLX) | — |
|
||||||
| Person Detection | Yes | Yes | Yes | Yes |
|
| Person Detection | Yes | Yes | Yes | Yes |
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Power Efficiency Comparison
|
|
||||||
|
|
||||||
| Accelerator | TOPS | Power (W) | TOPS/W | Notes |
|
|
||||||
|-------------|------|-----------|--------|-------|
|
|
||||||
| Hailo-8 | 26 | 2.5 | **10.4** | Best efficiency |
|
|
||||||
| Jetson Orin Nano | 40 | 15 | 2.7 | Most versatile |
|
|
||||||
| M1 Neural Engine | 15.8 | ~5 | 3.2 | Integrated in laptop |
|
|
||||||
| Ethos-U55 | ~1 | 0.05 | 20.0 | Ultra-low-power (returned) |
|
|
||||||
|
|||||||
@@ -1,65 +1,149 @@
|
|||||||
# Cloud Compute
|
# Cloud Compute — Live Verified
|
||||||
|
|
||||||
**2 DigitalOcean droplets** providing cloud presence and edge compute.
|
**2 DigitalOcean droplets** providing cloud presence and edge compute.
|
||||||
|
|
||||||
|
**Verified via SSH probes on 2026-02-21.**
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Fleet Overview
|
## Fleet Overview
|
||||||
|
|
||||||
| Node | Region | Spec | Public IP | Tailscale IP | Storage | Role | Status |
|
| Node | Hostname | Region | Spec | Public IP | Tailscale IP | Storage | Role | Status |
|
||||||
|------|--------|------|-----------|--------------|---------|------|--------|
|
|------|----------|--------|------|-----------|--------------|---------|------|--------|
|
||||||
| Codex-Infinity | NYC | 1 vCPU / 1GB | 159.65.43.12 | 100.108.132.8 | 78GB SSD | Codex server, oracle | Active |
|
| Codex-Infinity | **gematria** | NYC | 1 vCPU / 1GB | 159.65.43.12 | 100.108.132.8 | 78GB SSD | Codex server | Active |
|
||||||
| Shellfish | NYC | 1 vCPU / 1GB | 174.138.44.45 | 100.94.33.37 | 25GB SSD | Edge compute, tunnels | Active |
|
| Shellfish | **anastasia** | NYC | 1 vCPU / 1GB | 174.138.44.45 | 100.94.33.37 | 25GB SSD | Edge compute | Active |
|
||||||
|
|
||||||
|
### ERRATA vs Prior Documentation
|
||||||
|
|
||||||
|
| Item | Previously Documented | Live Verified |
|
||||||
|
|------|----------------------|---------------|
|
||||||
|
| Codex-Infinity hostname | "codex-infinity" | **gematria** |
|
||||||
|
| Shellfish hostname | "shellfish" | **anastasia** (naming collision with Pi!) |
|
||||||
|
| SSH user | `alexandria` | **`blackroad`** |
|
||||||
|
| Codex-Infinity OS | Debian 12, Kernel 5.15 | **CentOS Stream 9**, Kernel 6.12.10-200 |
|
||||||
|
| Shellfish OS | Debian 12, Kernel 5.15 | **Debian 12 (Bookworm)**, Kernel 6.1.0-28-amd64 |
|
||||||
|
|
||||||
|
> **NAMING COLLISION:** The Shellfish droplet's actual hostname is `anastasia`, which collides with
|
||||||
|
> the Anastasia Pi at 192.168.4.33. The SSH alias `anastasia` in `~/.ssh/config` points to the
|
||||||
|
> droplet (174.138.44.45), NOT the Pi. Consider renaming the droplet hostname to `shellfish` or
|
||||||
|
> `cadence` to eliminate confusion.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Per-Node Details
|
## Per-Node Details
|
||||||
|
|
||||||
### Codex-Infinity — Codex Server / Oracle
|
### Codex-Infinity (hostname: gematria) — Cloud Oracle
|
||||||
|
|
||||||
- **Role:** Codex database host, cloud services oracle
|
- **Role:** Codex database host, cloud services, HTTP/HTTPS gateway
|
||||||
- **Provider:** DigitalOcean
|
- **Provider:** DigitalOcean
|
||||||
- **Region:** NYC
|
- **Region:** NYC
|
||||||
- **Spec:** 1 vCPU, 1GB RAM, 78GB SSD
|
- **Spec:** 1 vCPU, 1GB RAM, 78GB SSD
|
||||||
- **OS:** Debian 12 (Bookworm), Kernel 5.15 LTS
|
- **OS:** CentOS Stream 9, Kernel 6.12.10-200.fc41.x86_64
|
||||||
- **Public IP:** 159.65.43.12
|
- **Public IP:** 159.65.43.12
|
||||||
- **Tailscale IP:** 100.108.132.8
|
- **Tailscale IP:** 100.108.132.8
|
||||||
- **Services:** codex-db, cloud-services
|
- **SSH:** `ssh gematria` or `ssh gematria-ts` (user: `blackroad`)
|
||||||
- **Tunnel:** tunnel-codex.blackroad.io
|
- **Tunnel:** tunnel-codex.blackroad.io (cloudflared running)
|
||||||
- **SSH:** `ssh codex-infinity` or `ssh 159.65.43.12`
|
- **SSH Aliases:** `gematria`, `gematria-ts`, `blackroad-os-ts`
|
||||||
- **User:** alexandria
|
|
||||||
|
|
||||||
### Shellfish — Edge Compute
|
**Verified Services (7 listening ports):**
|
||||||
|
|
||||||
- **Role:** Cloud edge node, Cloudflare tunnel relay
|
| Port | Service | Process |
|
||||||
|
|------|---------|---------|
|
||||||
|
| 22 | SSH | sshd |
|
||||||
|
| 53 | DNS (local resolver) | systemd-resolved |
|
||||||
|
| 80 | HTTP | nginx |
|
||||||
|
| 443 | HTTPS | nginx |
|
||||||
|
| 2019 | Caddy admin | caddy |
|
||||||
|
| 8011 | App service | — |
|
||||||
|
| 8787 | Python service | python3 |
|
||||||
|
| 11434 | **Ollama (PUBLIC!)** | ollama |
|
||||||
|
|
||||||
|
> **SECURITY WARNING:** Ollama is bound to `*:11434` (all interfaces) on this public-facing droplet.
|
||||||
|
> Anyone on the internet can access the Ollama API at `159.65.43.12:11434`.
|
||||||
|
> **Immediate action:** Restrict to localhost or Tailscale interface only.
|
||||||
|
>
|
||||||
|
> ```bash
|
||||||
|
> # Fix: edit /etc/systemd/system/ollama.service
|
||||||
|
> # Change OLLAMA_HOST to 127.0.0.1:11434
|
||||||
|
> ssh gematria "sudo sed -i 's/OLLAMA_HOST=.*/OLLAMA_HOST=127.0.0.1/' /etc/systemd/system/ollama.service"
|
||||||
|
> ssh gematria "sudo systemctl daemon-reload && sudo systemctl restart ollama"
|
||||||
|
> ```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Shellfish (hostname: anastasia) — Edge Compute
|
||||||
|
|
||||||
|
- **Role:** Cloud edge node, dashboards, API services, WebSocket servers
|
||||||
- **Provider:** DigitalOcean
|
- **Provider:** DigitalOcean
|
||||||
- **Region:** NYC
|
- **Region:** NYC
|
||||||
- **Spec:** 1 vCPU, 1GB RAM, 25GB SSD
|
- **Spec:** 1 vCPU, 1GB RAM, 25GB SSD
|
||||||
- **OS:** Debian 12 (Bookworm), Kernel 5.15 LTS
|
- **OS:** Debian 12 (Bookworm), Kernel 6.1.0-28-amd64
|
||||||
- **Public IP:** 174.138.44.45
|
- **Public IP:** 174.138.44.45
|
||||||
- **Tailscale IP:** 100.94.33.37
|
- **Tailscale IP:** 100.94.33.37
|
||||||
- **Services:** Cloudflare tunnels, edge-agent
|
- **SSH:** `ssh shellfish` or `ssh anastasia` or `ssh anastasia-ts` (user: `blackroad`)
|
||||||
- **Tunnel:** tunnel-cadence.blackroad.io
|
- **Tunnel:** tunnel-cadence.blackroad.io (cloudflared running)
|
||||||
- **SSH:** `ssh shellfish` or `ssh 174.138.44.45`
|
- **SSH Aliases:** `shellfish`, `anastasia`, `anastasia-ts`, `cadence-ts`
|
||||||
- **User:** alexandria
|
|
||||||
|
**Verified Services (14+ listening ports):**
|
||||||
|
|
||||||
|
| Port | Service | Process |
|
||||||
|
|------|---------|---------|
|
||||||
|
| 22 | SSH | sshd |
|
||||||
|
| 80 | HTTP | nginx |
|
||||||
|
| 3000 | Dashboard (Grafana?) | node |
|
||||||
|
| 3001 | Dashboard | node |
|
||||||
|
| 6379 | Redis-like | python3 |
|
||||||
|
| 8000 | API | uvicorn |
|
||||||
|
| 8080 | HTTP service | — |
|
||||||
|
| 8765 | WebSocket server | python3 |
|
||||||
|
| 8766 | WebSocket server | python3 |
|
||||||
|
| 8787 | Python service | python3 |
|
||||||
|
| 8888 | Python service | python3 |
|
||||||
|
| 11434 | Ollama | ollama (Tailscale-only: 100.64.0.1) |
|
||||||
|
|
||||||
|
**Systemd services:** cloudflared, docker, nginx, ollama
|
||||||
|
|
||||||
|
> **NOTE:** Ollama on Shellfish is bound to Tailscale interface only (100.64.0.1) — secure.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Security Baseline
|
## Security Baseline
|
||||||
|
|
||||||
Both droplets follow the standard BlackRoad OS baseline:
|
Both droplets should follow the standard BlackRoad OS baseline:
|
||||||
|
|
||||||
- SSH key-only authentication (no passwords)
|
| Control | Codex-Infinity | Shellfish | Status |
|
||||||
- UFW firewall: deny by default, allow 22/80/443/41641
|
|---------|---------------|-----------|--------|
|
||||||
- fail2ban enabled
|
| SSH key-only auth | Yes | Yes | OK |
|
||||||
- unattended-upgrades enabled
|
| Firewall (UFW/firewalld) | Unknown (CentOS) | UFW | Verify |
|
||||||
- chrony time sync to time.cloudflare.com
|
| fail2ban | Unknown | Enabled | Verify |
|
||||||
- Tailscale mesh connected
|
| unattended-upgrades | N/A (CentOS = dnf-automatic) | Enabled | Verify |
|
||||||
|
| Ollama binding | **PUBLIC (insecure!)** | Tailscale-only | **FIX** |
|
||||||
|
|
||||||
|
### Firewall Action Items
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Codex-Infinity (CentOS) — check firewalld
|
||||||
|
ssh gematria "sudo firewall-cmd --list-all"
|
||||||
|
|
||||||
|
# Shellfish (Debian) — check ufw
|
||||||
|
ssh shellfish "sudo ufw status verbose"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
## Management
|
## Management
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
ssh codex-infinity # Direct SSH
|
# SSH access
|
||||||
ssh shellfish # Direct SSH
|
ssh gematria # Codex-Infinity via direct IP
|
||||||
doctl compute droplet list # DigitalOcean CLI (if installed)
|
ssh gematria-ts # Codex-Infinity via Tailscale
|
||||||
|
ssh shellfish # Shellfish via direct IP
|
||||||
|
ssh anastasia-ts # Shellfish via Tailscale
|
||||||
|
|
||||||
|
# DigitalOcean CLI
|
||||||
|
doctl compute droplet list # List all droplets (if doctl installed)
|
||||||
|
|
||||||
|
# Cloudflare tunnel status
|
||||||
|
ssh gematria "systemctl status cloudflared"
|
||||||
|
ssh shellfish "systemctl status cloudflared"
|
||||||
```
|
```
|
||||||
|
|||||||
@@ -1,21 +1,37 @@
|
|||||||
# Raspberry Pi Fleet
|
# Raspberry Pi Fleet — Live Verified
|
||||||
|
|
||||||
**8 nodes** forming the always-on production backbone of BlackRoad infrastructure.
|
**8 nodes** forming the always-on production backbone of BlackRoad infrastructure.
|
||||||
|
|
||||||
|
**Verified via SSH probes on 2026-02-21.**
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Fleet Overview
|
## Fleet Overview
|
||||||
|
|
||||||
| Node | Board | RAM | Storage | Case | Accelerator | IP (Local) | IP (Tailscale) | Status |
|
| Node | Board | RAM | Storage | Case | Accelerator | IP (Local) | IP (Tailscale) | Status |
|
||||||
|------|-------|-----|---------|------|-------------|------------|----------------|--------|
|
|------|-------|-----|---------|------|-------------|------------|----------------|--------|
|
||||||
| Cecilia | Pi 5 | 8GB | 500GB NVMe | Standard | Hailo-8 26T | 192.168.4.89 | 100.72.180.98 | Active |
|
| Cecilia | Pi 5 | 8GB | 500GB NVMe (49% used) | Standard | **Hailo-8 26T** | 192.168.4.89 | 100.72.180.98 | **Active** |
|
||||||
| Octavia | Pi 5 | 8GB | 235GB SD | Pironman | Hailo-8 26T | 192.168.4.38 | 100.83.149.86 | Active |
|
| Octavia | Pi 5 | 8GB | 29GB SD (60% used) | Pironman | None confirmed | 192.168.4.38 | 100.66.235.47 | **Active (OVERLOADED)** |
|
||||||
| Lucidia | Pi 5 | 8GB | 117GB SD | ElectroCookie | — | 192.168.4.81 | 100.66.235.47 | Active |
|
| Lucidia | Pi 5 | 8GB | Unknown | ElectroCookie | Unknown | 192.168.4.81 | 100.83.149.86 | **DOWN** |
|
||||||
| Aria | Pi 5 | 8GB | 29GB SD | Pironman | Hailo-8 26T | 192.168.4.82 | 100.109.14.17 | Active |
|
| Aria | Pi 5 | 8GB | 29GB SD (74% used) | Pironman | None confirmed | 192.168.4.82 | 100.109.14.17 | **Active** |
|
||||||
| Anastasia | Pi 5 | 8GB | NVMe | Pironman | — | 192.168.4.33 | — | Active |
|
| Anastasia | Pi 5 | 8GB | Unknown | Pironman | Unknown | 192.168.4.33 | — | **SSH Closed** |
|
||||||
| Cordelia | Pi 5 | 8GB | SD | Standard | — | 192.168.4.27 | — | Active |
|
| Cordelia | Pi 5 | 8GB | Unknown | Standard | Unknown | 192.168.4.27 | — | **SSH Closed** |
|
||||||
| Alice | Pi 400 | 4GB | 32GB SD | Built-in | — | 192.168.4.49 | 100.77.210.18 | Active |
|
| Alice | Pi 400 | 4GB | 29GB SD (93% used) | Built-in | — | 192.168.4.49 | 100.77.210.18 | **Active** |
|
||||||
| Olympia | Pi 4B | 4GB | SD | PiKVM | — | — | — | Offline |
|
| Olympia | Pi 4B | 4GB | Unknown | PiKVM | — | — | — | **Offline** |
|
||||||
|
|
||||||
|
### ERRATA vs Prior Documentation
|
||||||
|
|
||||||
|
| Item | Previously Documented | Live Verified |
|
||||||
|
|------|----------------------|---------------|
|
||||||
|
| Octavia Tailscale IP | 100.83.149.86 | **100.66.235.47** |
|
||||||
|
| Lucidia Tailscale IP | 100.66.235.47 | **100.83.149.86** |
|
||||||
|
| Hailo-8 on Octavia | Active (26 TOPS) | **Not detected** (no `/dev/hailo*`, no `hailort.service`) |
|
||||||
|
| Hailo-8 on Aria | Active (26 TOPS) | **Not detected** (no `/dev/hailo*`, no `hailort.service`) |
|
||||||
|
| SSH user | `alexandria` | **`blackroad`** |
|
||||||
|
| Lucidia status | Active | **DOWN** (unreachable) |
|
||||||
|
| Octavia storage | 235GB Samsung EVO | **29GB** (60% used) |
|
||||||
|
| Cecilia OS | Bookworm | **Debian 13 Trixie** |
|
||||||
|
| Alice OS | Bookworm | **Raspbian 11 Bullseye** |
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -23,119 +39,227 @@
|
|||||||
|
|
||||||
### Cecilia — Primary AI Host
|
### Cecilia — Primary AI Host
|
||||||
|
|
||||||
- **Role:** CECE OS orchestrator, primary AI inference
|
- **Role:** CECE OS orchestrator, primary AI inference, observability hub
|
||||||
- **Hardware:** Pi 5 8GB + Hailo-8 M.2 (serial: HLLWM2B233704667) + 500GB NVMe
|
- **Hardware:** Pi 5 8GB + Hailo-8 M.2 (serial: HLLWM2B233704667) + 500GB NVMe
|
||||||
|
- **OS:** Debian 13 (Trixie), Kernel 6.6.62+rpt-rpi-2712
|
||||||
- **Case:** Standard with active fan
|
- **Case:** Standard with active fan
|
||||||
- **PSU:** Geekworm 27W 5V/5A USB-C
|
- **PSU:** Geekworm 27W 5V/5A USB-C
|
||||||
- **Services:** Ollama, CECE OS (68 sovereign apps), Hailo runtime
|
- **MAC:** 88:a2:9e:3b:eb:72
|
||||||
- **Storage:** 500GB Crucial P310 NVMe (~50% used)
|
- **Storage:** 500GB Crucial P310 NVMe (49% used, 230GB free)
|
||||||
- **SSH:** `ssh cecilia` / `ssh cecilia-ts`
|
- **SSH:** `ssh cecilia` / `ssh cecilia-ts` (user: `blackroad`)
|
||||||
- **Tunnel:** tunnel-cecilia.blackroad.io
|
- **Tunnel:** tunnel-cecilia.blackroad.io (cloudflared running)
|
||||||
- **Notes:** Houses the 68-app CECE OS sovereign stack. Primary inference node.
|
- **Systemd:** hailort, ollama, cloudflared, docker
|
||||||
|
|
||||||
### Octavia — AI Inference + Auth
|
**Verified Services (16+ listening ports):**
|
||||||
|
|
||||||
- **Role:** AI inference, PowerDNS, auth gateway
|
| Port | Service | Bind |
|
||||||
- **Hardware:** Pi 5 8GB + Hailo-8 M.2 (serial: HLLWM2B233704606) + Pironman case
|
|------|---------|------|
|
||||||
|
| 22 | SSH | 0.0.0.0 |
|
||||||
|
| 53 | DNS resolver | 0.0.0.0 |
|
||||||
|
| 80 | HTTP (nginx/caddy) | 0.0.0.0 |
|
||||||
|
| 631 | CUPS (printing) | 127.0.0.1 |
|
||||||
|
| 3001 | Dashboard (python3) | 0.0.0.0 |
|
||||||
|
| 3100 | Loki log aggregator | 0.0.0.0 |
|
||||||
|
| 5001-5002 | Python services | 0.0.0.0 |
|
||||||
|
| 5432 | **PostgreSQL** | 127.0.0.1 |
|
||||||
|
| 5900 | **VNC** | 0.0.0.0 |
|
||||||
|
| 8086 | **InfluxDB** | 0.0.0.0 |
|
||||||
|
| 8787 | Python service | 0.0.0.0 |
|
||||||
|
| 9000-9001 | **MinIO** (S3 + Console) | 0.0.0.0 |
|
||||||
|
| 9100 | Node Exporter (Prometheus) | 0.0.0.0 |
|
||||||
|
| 11434 | **Ollama** | 127.0.0.1 |
|
||||||
|
| 34001 | Tailscale relay | 0.0.0.0 |
|
||||||
|
|
||||||
|
**Infrastructure Stack:** PostgreSQL + InfluxDB + MinIO + Loki + Node Exporter = full observability
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Octavia — Multi-Service Hub (OVERLOADED)
|
||||||
|
|
||||||
|
- **Role:** Multi-arm processing, microservice host
|
||||||
|
- **Hardware:** Pi 5 8GB + Pironman case (NO Hailo-8 detected)
|
||||||
|
- **OS:** Debian 12 (Bookworm), Kernel 6.6.51+rpt-rpi-2712
|
||||||
- **Case:** Pironman with dual-fan tower cooler
|
- **Case:** Pironman with dual-fan tower cooler
|
||||||
- **PSU:** Geekworm 27W 5V/5A USB-C
|
- **PSU:** Geekworm 27W 5V/5A USB-C
|
||||||
- **Services:** Hailo runtime, PowerDNS, PowerDNS-Admin, RoadAuth, RoadAPI, auth-gateway
|
- **MAC:** 2c:cf:67:cf:fa:17
|
||||||
- **Storage:** 235GB Samsung EVO Select microSD (~90% used — needs cleanup)
|
- **Storage:** 29GB SD (60% used, 10.3GB free)
|
||||||
- **SSH:** `ssh octavia` / `ssh octavia-ts`
|
- **Load Average:** **9.47** (dangerously high for 4-core Pi)
|
||||||
- **Tunnel:** tunnel-octavia.blackroad.io
|
- **RAM:** 6.6GB / 7.9GB (83% used)
|
||||||
- **Known Issues:** Disk nearly full at 90%. Schedule cleanup.
|
- **SSH:** `ssh octavia` / `ssh octavia-ts` (user: `blackroad`)
|
||||||
|
- **Tunnel:** tunnel-octavia.blackroad.io (cloudflared running)
|
||||||
|
- **Systemd:** ollama, ollama-bridge, cloudflared, docker
|
||||||
|
|
||||||
### Lucidia — Event Bus + LLM Brain
|
> **WARNING:** 30+ listening ports, load average 9.47, RAM 83%. This node needs service migration or hardware upgrade.
|
||||||
|
|
||||||
|
**Verified Services (30+ listening ports):**
|
||||||
|
|
||||||
|
| Port Range | Service |
|
||||||
|
|------------|---------|
|
||||||
|
| 3002-3006 | App services (5 containers) |
|
||||||
|
| 3109, 4001-4002, 4010 | App services |
|
||||||
|
| 5200-6300 | 10 Python microservices |
|
||||||
|
| 8000 | API (uvicorn/gunicorn) |
|
||||||
|
| 8011, 8080-8082, 8180 | HTTP services |
|
||||||
|
| 5432 | PostgreSQL (localhost) |
|
||||||
|
| 11434 | Ollama (localhost) |
|
||||||
|
| 34001 | Tailscale relay |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Lucidia — Event Bus (DOWN)
|
||||||
|
|
||||||
- **Role:** NATS event bus, Ollama LLM server, edge agent
|
- **Role:** NATS event bus, Ollama LLM server, edge agent
|
||||||
- **Hardware:** Pi 5 8GB + ElectroCookie Radial Tower case
|
- **Hardware:** Pi 5 8GB + ElectroCookie Radial Tower case
|
||||||
- **Case:** ElectroCookie with tower cooler
|
|
||||||
- **PSU:** Geekworm 27W 5V/5A USB-C
|
- **PSU:** Geekworm 27W 5V/5A USB-C
|
||||||
- **Services:** NATS (port 4222), Ollama (port 11434), edge-agent
|
- **MAC:** incomplete (ARP expired)
|
||||||
- **Storage:** 117GB Samsung EVO Select microSD (~60% used)
|
- **IP:** 192.168.4.81 (local), 100.83.149.86 (Tailscale)
|
||||||
- **SSH:** `ssh lucidia` / `ssh lucidia-ts`
|
- **Status:** **DOWN — unreachable since at least 2026-02-21**
|
||||||
- **Tunnel:** tunnel-lucidia.blackroad.io
|
- **Tunnel:** tunnel-lucidia.blackroad.io (**DOWN** — node unreachable)
|
||||||
- **Notes:** Central event bus. All MQTT/NATS traffic routes through here.
|
|
||||||
|
|
||||||
### Aria — API Services
|
> **ACTION REQUIRED:** Physical investigation needed. This node hosts the NATS event bus.
|
||||||
|
> Power cycle or check ethernet/SD card.
|
||||||
|
|
||||||
- **Role:** Web services, API hosting, compute
|
---
|
||||||
- **Hardware:** Pi 5 8GB + Pironman case + Hailo-8 M.2
|
|
||||||
|
### Aria — Container Host
|
||||||
|
|
||||||
|
- **Role:** Container workloads, web services
|
||||||
|
- **Hardware:** Pi 5 8GB + Pironman case (NO Hailo-8 detected)
|
||||||
|
- **OS:** Debian 12 (Bookworm), Kernel 6.6.51+rpt-rpi-2712
|
||||||
- **Case:** Pironman with dual-fan tower cooler
|
- **Case:** Pironman with dual-fan tower cooler
|
||||||
- **PSU:** Geekworm 27W 5V/5A USB-C
|
- **PSU:** Geekworm 27W 5V/5A USB-C
|
||||||
- **Services:** Compute workloads, 9 containers
|
- **MAC:** 88:a2:9e:0d:42:07
|
||||||
- **Storage:** 29GB Samsung EVO Select microSD (~70% used)
|
- **Storage:** 29GB SD (74% used — monitor closely)
|
||||||
- **SSH:** `ssh aria` / `ssh aria-ts`
|
- **SSH:** `ssh aria` / `ssh aria-ts` (user: `blackroad`)
|
||||||
- **Notes:** Rock-solid uptime (4+ weeks continuous). Low storage — consider NVMe upgrade.
|
- **Systemd:** ollama, cloudflared, docker
|
||||||
|
|
||||||
### Anastasia — AI Inference Secondary
|
**Verified Services (30+ listening ports):**
|
||||||
|
|
||||||
- **Role:** Secondary AI inference node
|
| Port Range | Service |
|
||||||
|
|------------|---------|
|
||||||
|
| 3140-3167 | **28 Docker container ports** |
|
||||||
|
| 8081 | HTTP service |
|
||||||
|
| 8180 | Python service |
|
||||||
|
|
||||||
|
> **NOTE:** 28 container ports in the 3140-3167 range. Disk at 74% — monitor closely.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Anastasia — Pi (SSH Closed)
|
||||||
|
|
||||||
|
- **Role:** Secondary AI inference node (pending deployment)
|
||||||
- **Hardware:** Pi 5 8GB + Pironman case + NVMe
|
- **Hardware:** Pi 5 8GB + Pironman case + NVMe
|
||||||
- **Case:** Pironman with dual-fan tower cooler
|
- **Case:** Pironman with dual-fan tower cooler
|
||||||
- **PSU:** Geekworm 27W 5V/5A USB-C
|
- **PSU:** Geekworm 27W 5V/5A USB-C
|
||||||
- **Services:** (Pending deployment)
|
- **MAC:** 60:92:c8:11:cf:7c
|
||||||
- **Storage:** NVMe via Pironman (Crucial P310)
|
- **IP:** 192.168.4.33 (no Tailscale)
|
||||||
- **SSH:** `ssh anastasia` (192.168.4.33)
|
- **Ping:** UP
|
||||||
|
- **SSH:** **Connection refused** — port 22 not open
|
||||||
|
|
||||||
### Cordelia — Orchestration
|
> **NOTE:** SSH alias `anastasia` in `~/.ssh/config` points to the DigitalOcean droplet (174.138.44.45),
|
||||||
|
> NOT this Pi. Add an `anastasia-pi` alias for 192.168.4.33 once SSH is enabled.
|
||||||
|
|
||||||
- **Role:** Fleet orchestration
|
---
|
||||||
|
|
||||||
|
### Cordelia — Orchestration (SSH Closed)
|
||||||
|
|
||||||
|
- **Role:** Fleet orchestration (pending deployment)
|
||||||
- **Hardware:** Pi 5 8GB
|
- **Hardware:** Pi 5 8GB
|
||||||
- **Case:** Standard with active cooler
|
- **Case:** Standard with active cooler
|
||||||
- **PSU:** Geekworm 27W 5V/5A USB-C
|
- **PSU:** Geekworm 27W 5V/5A USB-C
|
||||||
- **Services:** (Pending deployment)
|
- **MAC:** 6c:4a:85:32:ae:72
|
||||||
- **SSH:** `ssh cordelia` (192.168.4.27)
|
- **IP:** 192.168.4.27 (no Tailscale)
|
||||||
|
- **Ping:** UP
|
||||||
|
- **SSH:** **Connection refused** — port 22 not open
|
||||||
|
|
||||||
|
> **NOTE:** No SSH config entry exists for `cordelia`. Cannot configure remotely until SSH is enabled.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
### Alice — Gateway / Admin
|
### Alice — Gateway / Admin
|
||||||
|
|
||||||
- **Role:** Gateway, auth, development, built-in admin console
|
- **Role:** Gateway, auth, development
|
||||||
- **Hardware:** Pi 400 (keyboard built-in) 4GB
|
- **Hardware:** Pi 400 (keyboard built-in) 4GB
|
||||||
|
- **OS:** Raspbian 11 (Bullseye), Kernel 6.1.21-v8+
|
||||||
- **Case:** Built-in keyboard enclosure
|
- **Case:** Built-in keyboard enclosure
|
||||||
- **PSU:** 5V/3A USB-C (15W)
|
- **PSU:** 5V/3A USB-C (15W)
|
||||||
- **Services:** Worker node, 7 containers
|
- **MAC:** d8:3a:dd:ff:98:87
|
||||||
- **Storage:** 32GB microSD (~93% used — needs cleanup)
|
- **Storage:** 29GB SD (93% used — **CRITICAL**)
|
||||||
- **SSH:** `ssh alice` / `ssh alice-ts`
|
- **SSH:** `ssh alice` / `ssh alice-ts` (user: `blackroad`)
|
||||||
- **Known Issues:** Disk critically full at 93%. Immediate cleanup needed.
|
- **Systemd:** cloudflared, docker
|
||||||
|
|
||||||
### Olympia — KVM Console
|
**Verified Services:**
|
||||||
|
|
||||||
|
| Port | Service |
|
||||||
|
|------|---------|
|
||||||
|
| 22 | SSH |
|
||||||
|
|
||||||
|
> **WARNING:** Disk at 93% full. Immediate cleanup needed. Minimal services running but load avg 6.17
|
||||||
|
> is concerning for a 4-core Pi 400 — investigate Docker workloads.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Olympia — KVM Console (Offline)
|
||||||
|
|
||||||
- **Role:** Remote KVM access to other nodes
|
- **Role:** Remote KVM access to other nodes
|
||||||
- **Hardware:** Pi 4B 4GB + PiKVM case
|
- **Hardware:** Pi 4B 4GB + PiKVM case
|
||||||
- **Case:** PiKVM enclosure
|
- **Case:** PiKVM enclosure
|
||||||
- **PSU:** 5V/3A USB-C (15W)
|
- **PSU:** 5V/3A USB-C (15W)
|
||||||
- **Services:** PiKVM OS
|
- **Status:** **Offline** — not on network, needs recommissioning
|
||||||
- **Status:** **Offline** — needs recommissioning
|
- **Notes:** Used for headless recovery of other Pis. Not verified.
|
||||||
- **Notes:** Used for headless recovery of other Pis.
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Maintenance Notes
|
## Maintenance Priority
|
||||||
|
|
||||||
### Disk Cleanup Priority
|
### Immediate Actions
|
||||||
|
|
||||||
1. **Alice** (93% full) — Critical
|
1. **Lucidia** — Physical investigation. Node DOWN. Power cycle and check connectivity.
|
||||||
2. **Octavia** (90% full) — High
|
2. **Alice** — Disk at 93%. Run `sudo apt autoremove && docker system prune -a`.
|
||||||
3. **Aria** (70% full) — Monitor
|
3. **Octavia** — Overloaded (load 9.47, RAM 83%). Migrate services to Aria or Cecilia.
|
||||||
|
4. **Anastasia / Cordelia** — Enable SSH (requires keyboard + monitor).
|
||||||
|
|
||||||
### SSH Config
|
### Hailo-8 Investigation
|
||||||
|
|
||||||
All nodes use key-only authentication. SSH config on Alexandria (Mac):
|
3 Hailo-8 M.2 modules were purchased but only 1 is detected (Cecilia). Possible explanations:
|
||||||
|
- Modules not physically installed in Octavia/Aria M.2 slots
|
||||||
|
- HailoRT not installed on those nodes
|
||||||
|
- Modules installed but not recognized (driver issue)
|
||||||
|
|
||||||
```
|
Requires physical inspection of M.2 slots on Octavia and Aria.
|
||||||
Host cecilia
|
|
||||||
HostName 192.168.4.89
|
|
||||||
User alexandria
|
|
||||||
|
|
||||||
Host cecilia-ts
|
### Stale Data Cleanup
|
||||||
HostName 100.72.180.98
|
|
||||||
User alexandria
|
| Item | Issue | Fix |
|
||||||
|
|------|-------|-----|
|
||||||
|
| `/etc/hosts` on Mac | `192.168.4.74 octavia` (wrong IP) | Change to `192.168.4.38 octavia` |
|
||||||
|
| `~/hailo.sh` | Connects to `pi@192.168.4.74` | Change to `blackroad@192.168.4.38` |
|
||||||
|
| SSH `anastasia` alias | Points to DO droplet, not Pi | Add `anastasia-pi` for 192.168.4.33 |
|
||||||
|
| Agent registry | Octavia/Aria listed as `pironman_hailo8` | Change to `pironman` (no Hailo confirmed) |
|
||||||
|
|
||||||
|
### SSH Quick Reference
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# All SSH uses user 'blackroad', not 'alexandria'
|
||||||
|
ssh cecilia # 192.168.4.89
|
||||||
|
ssh octavia # 192.168.4.38
|
||||||
|
ssh aria # 192.168.4.82
|
||||||
|
ssh alice # 192.168.4.49
|
||||||
|
ssh lucidia # 192.168.4.81 (DOWN)
|
||||||
|
|
||||||
|
# Tailscale (remote access)
|
||||||
|
ssh cecilia-ts # 100.72.180.98
|
||||||
|
ssh octavia-ts # 100.66.235.47
|
||||||
|
ssh aria-ts # 100.109.14.17
|
||||||
|
ssh alice-ts # 100.77.210.18
|
||||||
|
ssh lucidia-ts # 100.83.149.86 (DOWN)
|
||||||
```
|
```
|
||||||
|
|
||||||
### Management Scripts
|
### Management Scripts
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
~/pifleet.sh # Fleet overview
|
~/pifleet.sh # Fleet overview
|
||||||
~/hardware.sh # Interactive hardware menu
|
~/hardware.sh # Interactive hardware menu
|
||||||
~/blackroad-network-scan.sh # Scan all Pi IPs
|
~/blackroad-network-scan.sh # Scan all Pi IPs
|
||||||
|
~/blackroad-network-discovery.sh # SSH probe all nodes
|
||||||
```
|
```
|
||||||
|
|||||||
@@ -1,16 +1,19 @@
|
|||||||
# BlackRoad Fleet Registry — Machine-Readable Device Inventory
|
# BlackRoad Fleet Registry — Machine-Readable Device Inventory
|
||||||
# Source of truth for automation and tooling
|
# Source of truth for automation and tooling
|
||||||
# Version: 2.0.0
|
# Version: 2.1.0
|
||||||
# Updated: 2026-02-20
|
# Updated: 2026-02-21
|
||||||
|
# Verified: Live SSH + ARP + ping probes
|
||||||
|
|
||||||
fleet:
|
fleet:
|
||||||
version: "2.0.0"
|
version: "2.1.0"
|
||||||
updated: "2026-02-20"
|
updated: "2026-02-21"
|
||||||
owner: "BlackRoad OS, Inc."
|
owner: "BlackRoad OS, Inc."
|
||||||
domain: "blackroad.io"
|
domain: "blackroad.io"
|
||||||
tailnet: "blackroad"
|
tailnet: "blackroad"
|
||||||
total_devices: 21
|
total_devices: 21
|
||||||
total_ai_tops: 135
|
unidentified_devices: 4
|
||||||
|
confirmed_ai_tops: 41.8 # 1x Hailo-8 (26) + M1 NE (15.8)
|
||||||
|
potential_ai_tops: 134 # If all accelerators installed
|
||||||
|
|
||||||
tiers:
|
tiers:
|
||||||
production:
|
production:
|
||||||
@@ -36,18 +39,21 @@ fleet:
|
|||||||
# ── Production Cluster ──────────────────────────────────────
|
# ── Production Cluster ──────────────────────────────────────
|
||||||
- name: cecilia
|
- name: cecilia
|
||||||
tier: production
|
tier: production
|
||||||
hardware: "Raspberry Pi 5"
|
hardware: "Raspberry Pi 5 Model B Rev 1.1"
|
||||||
ram: "8GB"
|
ram: "8GB (3.3GB used, 4.6GB available)"
|
||||||
storage: "500GB NVMe (Crucial P310)"
|
storage: "457GB NVMe (/dev/nvme0n1p2, 15% used)"
|
||||||
case: "Standard"
|
case: "Standard"
|
||||||
accelerator: "Hailo-8 M.2 26 TOPS"
|
accelerator: "Hailo-8 M.2 26 TOPS"
|
||||||
accelerator_serial: "HLLWM2B233704667"
|
accelerator_serial: "HLLWM2B233704667"
|
||||||
|
accelerator_verified: true # /dev/hailo0 detected
|
||||||
cooling: "Active fan"
|
cooling: "Active fan"
|
||||||
psu: "Geekworm 27W 5V/5A USB-C"
|
psu: "Geekworm 27W 5V/5A USB-C"
|
||||||
ip_local: "192.168.4.89"
|
ip_local: "192.168.4.89"
|
||||||
ip_tailscale: "100.72.180.98"
|
ip_tailscale: "100.72.180.98"
|
||||||
os: "Debian 12 (Bookworm)"
|
mac: "88:a2:9e:3b:eb:72"
|
||||||
kernel: "6.1 LTS"
|
os: "Debian 13 (Trixie)" # NOT Bookworm!
|
||||||
|
kernel: "6.12.62+rpt-rpi-2712"
|
||||||
|
ssh_user: "blackroad"
|
||||||
roles:
|
roles:
|
||||||
- primary_ai_host
|
- primary_ai_host
|
||||||
- orchestration
|
- orchestration
|
||||||
@@ -62,18 +68,21 @@ fleet:
|
|||||||
|
|
||||||
- name: octavia
|
- name: octavia
|
||||||
tier: production
|
tier: production
|
||||||
hardware: "Raspberry Pi 5"
|
hardware: "Raspberry Pi 5 Model B Rev 1.1"
|
||||||
ram: "8GB"
|
ram: "8GB (6.6GB used, 1.3GB available)" # WARNING: memory pressure
|
||||||
storage: "235GB microSD (Samsung EVO Select)"
|
storage: "235GB microSD (34% used, 76G/235G)"
|
||||||
case: "Pironman"
|
case: "Pironman"
|
||||||
accelerator: "Hailo-8 M.2 26 TOPS"
|
accelerator: null # HAILO NOT DETECTED despite registry
|
||||||
accelerator_serial: "HLLWM2B233704606"
|
accelerator_verified: false
|
||||||
cooling: "Pironman dual-fan tower"
|
cooling: "Pironman dual-fan tower"
|
||||||
psu: "Geekworm 27W 5V/5A USB-C"
|
psu: "Geekworm 27W 5V/5A USB-C"
|
||||||
ip_local: "192.168.4.38"
|
ip_local: "192.168.4.38"
|
||||||
ip_tailscale: "100.83.149.86"
|
ip_tailscale: "100.66.235.47" # CORRECTED (was swapped with Lucidia)
|
||||||
|
mac: "2c:cf:67:cf:fa:17"
|
||||||
os: "Debian 12 (Bookworm)"
|
os: "Debian 12 (Bookworm)"
|
||||||
kernel: "6.1 LTS"
|
kernel: "6.12.62+rpt-rpi-2712"
|
||||||
|
ssh_user: "blackroad"
|
||||||
|
load_average: "9.47" # WARNING: overloaded
|
||||||
roles:
|
roles:
|
||||||
- ai_inference
|
- ai_inference
|
||||||
- auth
|
- auth
|
||||||
@@ -99,9 +108,12 @@ fleet:
|
|||||||
cooling: "ElectroCookie tower cooler"
|
cooling: "ElectroCookie tower cooler"
|
||||||
psu: "Geekworm 27W 5V/5A USB-C"
|
psu: "Geekworm 27W 5V/5A USB-C"
|
||||||
ip_local: "192.168.4.81"
|
ip_local: "192.168.4.81"
|
||||||
ip_tailscale: "100.66.235.47"
|
ip_tailscale: "100.83.149.86" # CORRECTED (was swapped with Octavia)
|
||||||
os: "Debian 12 (Bookworm)"
|
mac: null # ARP incomplete — node is DOWN
|
||||||
kernel: "6.1 LTS"
|
os: "Debian 12 (Bookworm)" # last known
|
||||||
|
kernel: "6.12.62+rpt-rpi-2712" # last known
|
||||||
|
ssh_user: "blackroad"
|
||||||
|
status_override: "DOWN — unreachable via ping, ARP incomplete"
|
||||||
roles:
|
roles:
|
||||||
- nats_bus
|
- nats_bus
|
||||||
- llm_inference
|
- llm_inference
|
||||||
@@ -116,17 +128,20 @@ fleet:
|
|||||||
|
|
||||||
- name: aria
|
- name: aria
|
||||||
tier: production
|
tier: production
|
||||||
hardware: "Raspberry Pi 5"
|
hardware: "Raspberry Pi 5 Model B Rev 1.1"
|
||||||
ram: "8GB"
|
ram: "8GB (3.8GB used, 4.0GB available)"
|
||||||
storage: "29GB microSD (Samsung EVO Select)"
|
storage: "29GB microSD (74% used, 7.3GB free)" # WARNING: low space
|
||||||
case: "Pironman"
|
case: "Pironman"
|
||||||
accelerator: "Hailo-8 M.2 26 TOPS"
|
accelerator: null # HAILO NOT DETECTED despite registry
|
||||||
|
accelerator_verified: false
|
||||||
cooling: "Pironman dual-fan tower"
|
cooling: "Pironman dual-fan tower"
|
||||||
psu: "Geekworm 27W 5V/5A USB-C"
|
psu: "Geekworm 27W 5V/5A USB-C"
|
||||||
ip_local: "192.168.4.82"
|
ip_local: "192.168.4.82"
|
||||||
ip_tailscale: "100.109.14.17"
|
ip_tailscale: "100.109.14.17"
|
||||||
|
mac: "88:a2:9e:0d:42:07"
|
||||||
os: "Debian 12 (Bookworm)"
|
os: "Debian 12 (Bookworm)"
|
||||||
kernel: "6.1 LTS"
|
kernel: "6.12.62+rpt-rpi-2712"
|
||||||
|
ssh_user: "blackroad"
|
||||||
roles:
|
roles:
|
||||||
- api_services
|
- api_services
|
||||||
- compute
|
- compute
|
||||||
@@ -139,20 +154,22 @@ fleet:
|
|||||||
tier: production
|
tier: production
|
||||||
hardware: "Raspberry Pi 5"
|
hardware: "Raspberry Pi 5"
|
||||||
ram: "8GB"
|
ram: "8GB"
|
||||||
storage: "NVMe (Crucial P310, Pironman)"
|
storage: "Unknown (SSH closed)"
|
||||||
case: "Pironman"
|
case: "Pironman"
|
||||||
accelerator: null
|
accelerator: null
|
||||||
cooling: "Pironman dual-fan tower"
|
cooling: "Pironman dual-fan tower"
|
||||||
psu: "Geekworm 27W 5V/5A USB-C"
|
psu: "Geekworm 27W 5V/5A USB-C"
|
||||||
ip_local: "192.168.4.33"
|
ip_local: "192.168.4.33"
|
||||||
ip_tailscale: null
|
ip_tailscale: null
|
||||||
os: "Debian 12 (Bookworm)"
|
mac: "60:92:c8:11:cf:7c" # Confirmed Raspberry Pi 5
|
||||||
kernel: "6.1 LTS"
|
os: "Unknown (SSH closed)"
|
||||||
|
ssh_user: "blackroad"
|
||||||
|
ssh_note: "WARNING: SSH alias 'anastasia' points to DO droplet, NOT this Pi"
|
||||||
roles:
|
roles:
|
||||||
- ai_inference_secondary
|
- ai_inference_secondary
|
||||||
services: []
|
services: []
|
||||||
ssh: "ssh anastasia"
|
ssh: "ssh blackroad@192.168.4.33" # Direct IP needed, alias is wrong
|
||||||
status: active
|
status: ssh_closed # Ping responds but port 22 refused
|
||||||
|
|
||||||
- name: cordelia
|
- name: cordelia
|
||||||
tier: production
|
tier: production
|
||||||
@@ -175,17 +192,20 @@ fleet:
|
|||||||
|
|
||||||
- name: alice
|
- name: alice
|
||||||
tier: production
|
tier: production
|
||||||
hardware: "Raspberry Pi 400"
|
hardware: "Raspberry Pi 400 Rev 1.0"
|
||||||
ram: "4GB"
|
ram: "4GB (579MB used, 3.1GB available)"
|
||||||
storage: "32GB microSD"
|
storage: "15GB root (71% used, 4.1GB free)" # NOT 32GB
|
||||||
case: "Built-in keyboard"
|
case: "Built-in keyboard"
|
||||||
accelerator: null
|
accelerator: null
|
||||||
cooling: "Passive (built-in)"
|
cooling: "Passive (built-in)"
|
||||||
psu: "5V/3A USB-C (15W)"
|
psu: "5V/3A USB-C (15W)"
|
||||||
ip_local: "192.168.4.49"
|
ip_local: "192.168.4.49"
|
||||||
ip_tailscale: "100.77.210.18"
|
ip_tailscale: "100.77.210.18"
|
||||||
os: "Debian 12 (Bookworm)"
|
mac: "d8:3a:dd:ff:98:87"
|
||||||
kernel: "6.1 LTS"
|
os: "Raspbian 11 (Bullseye)" # NOT Bookworm — needs upgrade
|
||||||
|
kernel: "6.1.21-v8+"
|
||||||
|
ssh_user: "blackroad"
|
||||||
|
load_average: "6.17" # High for 4 cores
|
||||||
roles:
|
roles:
|
||||||
- gateway
|
- gateway
|
||||||
- development
|
- development
|
||||||
@@ -215,39 +235,57 @@ fleet:
|
|||||||
# ── Cloud Compute ───────────────────────────────────────────
|
# ── Cloud Compute ───────────────────────────────────────────
|
||||||
- name: codex-infinity
|
- name: codex-infinity
|
||||||
tier: cloud
|
tier: cloud
|
||||||
hardware: "DigitalOcean Droplet"
|
actual_hostname: "gematria"
|
||||||
ram: "1GB"
|
hardware: "DigitalOcean Droplet (DO-Premium-AMD)"
|
||||||
storage: "78GB SSD"
|
ram: "~765MB"
|
||||||
|
storage: "25GB+"
|
||||||
ip_local: "159.65.43.12"
|
ip_local: "159.65.43.12"
|
||||||
ip_tailscale: "100.108.132.8"
|
ip_tailscale: "100.108.132.8"
|
||||||
os: "Debian 12 (Bookworm)"
|
os: "Debian/Ubuntu based"
|
||||||
kernel: "5.15 LTS"
|
ssh_user: "blackroad"
|
||||||
|
ssh_alias: "gematria"
|
||||||
roles:
|
roles:
|
||||||
- codex_server
|
- codex_server
|
||||||
- oracle
|
- oracle
|
||||||
services:
|
services:
|
||||||
- codex-db
|
- ollama (port 11434, public)
|
||||||
- cloud-services
|
- nginx (80, 443)
|
||||||
|
- cloudflared
|
||||||
|
- caddy (2019 admin)
|
||||||
|
- python-app (8787, 8011)
|
||||||
tunnel: "tunnel-codex.blackroad.io"
|
tunnel: "tunnel-codex.blackroad.io"
|
||||||
status: active
|
status: active
|
||||||
|
uptime: "55+ days"
|
||||||
|
|
||||||
- name: shellfish
|
- name: shellfish
|
||||||
tier: cloud
|
tier: cloud
|
||||||
hardware: "DigitalOcean Droplet"
|
actual_hostname: "anastasia" # WARNING: name collision with Pi
|
||||||
ram: "1GB"
|
hardware: "DigitalOcean Droplet (DO-Premium-AMD)"
|
||||||
storage: "25GB SSD"
|
ram: "765MB (408MB used)"
|
||||||
|
storage: "25GB SSD (57% used, 15G/25G)"
|
||||||
ip_local: "174.138.44.45"
|
ip_local: "174.138.44.45"
|
||||||
ip_tailscale: "100.94.33.37"
|
ip_tailscale: "100.94.33.37"
|
||||||
os: "Debian 12 (Bookworm)"
|
os: "CentOS Stream 9"
|
||||||
kernel: "5.15 LTS"
|
kernel: "5.14.0-651.el9.x86_64"
|
||||||
|
ssh_user: "blackroad"
|
||||||
|
ssh_alias: "anastasia" # WARNING: collides with Pi
|
||||||
|
ssh_alias_alt: "cadence"
|
||||||
roles:
|
roles:
|
||||||
- cloud_infrastructure
|
- cloud_infrastructure
|
||||||
- edge
|
- edge
|
||||||
services:
|
services:
|
||||||
- tunnels
|
- ollama (port 11434, Tailscale only at 100.64.0.1)
|
||||||
- edge-agent
|
- nginx (80)
|
||||||
|
- cloudflared
|
||||||
|
- docker
|
||||||
|
- uvicorn-api (8000)
|
||||||
|
- websocket (8765, 8766)
|
||||||
|
- redis-like (6379)
|
||||||
|
- grafana (3000, 3001)
|
||||||
|
- python-apps (8080, 8787, 8888)
|
||||||
tunnel: "tunnel-cadence.blackroad.io"
|
tunnel: "tunnel-cadence.blackroad.io"
|
||||||
status: active
|
status: active
|
||||||
|
uptime: "55+ days"
|
||||||
|
|
||||||
# ── Edge Compute ────────────────────────────────────────────
|
# ── Edge Compute ────────────────────────────────────────────
|
||||||
- name: jetson-agent
|
- name: jetson-agent
|
||||||
@@ -362,6 +400,32 @@ fleet:
|
|||||||
speaker. Could be re-acquired for doorbell/monitor use case.
|
speaker. Could be re-acquired for doorbell/monitor use case.
|
||||||
Ethos-U55 NPU adds ~1 TOPS for vision inference.
|
Ethos-U55 NPU adds ~1 TOPS for vision inference.
|
||||||
|
|
||||||
|
# ── Unidentified LAN Devices ────────────────────────────────
|
||||||
|
unidentified:
|
||||||
|
- ip: "192.168.4.22"
|
||||||
|
mac: "30:be:29:5b:24:5f"
|
||||||
|
oui: "Unknown (possibly Hisense)"
|
||||||
|
ping: true
|
||||||
|
guess: "Smart TV or IoT device"
|
||||||
|
|
||||||
|
- ip: "192.168.4.44"
|
||||||
|
mac: "98:17:3c:38:db:78"
|
||||||
|
oui: "TP-Link"
|
||||||
|
ping: true
|
||||||
|
guess: "WiFi extender or smart plug"
|
||||||
|
|
||||||
|
- ip: "192.168.4.83"
|
||||||
|
mac: "54:4c:8a:9b:09:3d"
|
||||||
|
oui: "Unknown (Shenzhen Bilian)"
|
||||||
|
ping: true
|
||||||
|
guess: "Smart home WiFi module"
|
||||||
|
|
||||||
|
- ip: "192.168.4.92"
|
||||||
|
mac: "de:a2:b7:f3:f9:5d"
|
||||||
|
oui: "Locally administered (Apple device)"
|
||||||
|
ping: false
|
||||||
|
guess: "Apple device with private WiFi address"
|
||||||
|
|
||||||
# ── MCU Array ───────────────────────────────────────────────
|
# ── MCU Array ───────────────────────────────────────────────
|
||||||
microcontrollers:
|
microcontrollers:
|
||||||
- type: "ESP32-S3 SuperMini"
|
- type: "ESP32-S3 SuperMini"
|
||||||
|
|||||||
@@ -1,75 +1,160 @@
|
|||||||
# Services Map
|
# Services Map — Live Verified
|
||||||
|
|
||||||
Which services run on which device, and on which ports.
|
**Verified via `ss -tlnp` SSH probes on 2026-02-21.**
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Service-to-Device Matrix
|
## Cecilia (192.168.4.89) — 16+ services
|
||||||
|
|
||||||
| Service | Port | Node(s) | Protocol | Notes |
|
| Port | Service | Bind | Process |
|
||||||
|---------|------|---------|----------|-------|
|
|------|---------|------|---------|
|
||||||
| NATS | 4222 | Lucidia | TCP | Central event bus |
|
| 22 | SSH | 0.0.0.0 | sshd |
|
||||||
| Ollama | 11434 | Lucidia, Cecilia | HTTP | LLM inference |
|
| 53 | DNS | 0.0.0.0 | (resolver) |
|
||||||
| Hailo Runtime | — | Cecilia, Octavia, Aria | Local | AI accelerator driver |
|
| 80 | HTTP | 0.0.0.0 | nginx/caddy |
|
||||||
| CECE OS | 8080+ | Cecilia | HTTP | 68 sovereign apps |
|
| 631 | CUPS (printing) | 127.0.0.1 | cupsd |
|
||||||
| PowerDNS | 53 | Octavia | TCP/UDP | Internal DNS |
|
| 3001 | App (dashboard?) | 0.0.0.0 | python3 |
|
||||||
| PowerDNS-Admin | 8080 | Octavia | HTTP | DNS management UI |
|
| 3100 | Loki/log aggregator | 0.0.0.0 | — |
|
||||||
| RoadAuth | — | Octavia | HTTP | Authentication service |
|
| 5001 | Python service | 0.0.0.0 | python3 |
|
||||||
| RoadAPI | — | Octavia | HTTP | API gateway |
|
| 5002 | Python service | 0.0.0.0 | python3 |
|
||||||
| Auth-Gateway | — | Octavia | HTTP | Auth proxy |
|
| 5432 | **PostgreSQL** | 127.0.0.1 | postgres |
|
||||||
| Cloudflared | — | Cecilia, Lucidia, Octavia, Codex-Infinity, Shellfish | — | Cloudflare tunnel agent |
|
| 5900 | **VNC** | 0.0.0.0 | vnc |
|
||||||
| Edge-Agent | — | Lucidia, Shellfish | — | Edge compute agent |
|
| 8086 | **InfluxDB** | 0.0.0.0 | influxd |
|
||||||
| Worker | — | Alice | — | Task worker |
|
| 8787 | Python service | 0.0.0.0 | python3 |
|
||||||
| Codex-DB | — | Codex-Infinity | SQLite | Component index |
|
| 9000 | **MinIO** (S3) | 0.0.0.0 + [::] | minio |
|
||||||
| Cloud-Services | — | Codex-Infinity | — | Cloud oracle |
|
| 9001 | **MinIO Console** | 0.0.0.0 | minio |
|
||||||
| MQTT (planned) | 1883 | Pi-Ops (planned) | TCP | IoT pub/sub broker |
|
| 9100 | **Node Exporter** (Prometheus) | 0.0.0.0 | python3 |
|
||||||
|
| 11434 | **Ollama** | 127.0.0.1 | ollama |
|
||||||
|
| 34001 | Tailscale relay | 0.0.0.0 | tailscaled |
|
||||||
|
|
||||||
|
**Systemd services:** hailort, ollama, cloudflared, docker
|
||||||
|
|
||||||
|
**Infrastructure stack:** PostgreSQL + InfluxDB + MinIO + Loki + Node Exporter = full observability
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Per-Node Service List
|
## Octavia (192.168.4.38) — 30+ services (OVERLOADED)
|
||||||
|
|
||||||
### Cecilia (192.168.4.89)
|
| Port Range | Service | Bind | Process |
|
||||||
- Ollama (LLM inference)
|
|------------|---------|------|---------|
|
||||||
- CECE OS (68 sovereign apps)
|
| 3002-3006 | App services (5 ports) | 0.0.0.0 | containers |
|
||||||
- Hailo runtime (26 TOPS accelerator)
|
| 3109 | App service | 0.0.0.0 | — |
|
||||||
- Cloudflared tunnel
|
| 4001-4002 | App services | 0.0.0.0 | — |
|
||||||
|
| 4010 | App service | 127.0.0.1 | — |
|
||||||
|
| 5200 | Python microservice | 0.0.0.0 | python3 |
|
||||||
|
| 5300 | Python microservice | 0.0.0.0 | python3 |
|
||||||
|
| 5400 | Python microservice | 0.0.0.0 | python3 |
|
||||||
|
| 5500 | Python microservice | 0.0.0.0 | python3 |
|
||||||
|
| 5600 | Python microservice | 0.0.0.0 | python3 |
|
||||||
|
| 5900 | Python microservice | 0.0.0.0 | python3 |
|
||||||
|
| 6000 | Python microservice | 0.0.0.0 | python3 |
|
||||||
|
| 6100 | Python microservice | 0.0.0.0 | python3 |
|
||||||
|
| 6200 | Python microservice | 0.0.0.0 | python3 |
|
||||||
|
| 6300 | Python microservice | 0.0.0.0 | python3 |
|
||||||
|
| 8000 | API (uvicorn/gunicorn) | 0.0.0.0 | — |
|
||||||
|
| 8011 | Python service | 0.0.0.0 | — |
|
||||||
|
| 8080-8082 | HTTP services | 0.0.0.0 | — |
|
||||||
|
| 8180 | Python service | 0.0.0.0 | — |
|
||||||
|
| 5432 | PostgreSQL | 127.0.0.1 | postgres |
|
||||||
|
| 11434 | Ollama | 127.0.0.1 | ollama |
|
||||||
|
| 34001 | Tailscale relay | 0.0.0.0 | tailscaled |
|
||||||
|
|
||||||
### Lucidia (192.168.4.81)
|
**Systemd services:** ollama, ollama-bridge, cloudflared, docker
|
||||||
- NATS event bus (port 4222)
|
|
||||||
- Ollama (LLM inference, port 11434)
|
|
||||||
- Edge-agent
|
|
||||||
- Cloudflared tunnel
|
|
||||||
|
|
||||||
### Octavia (192.168.4.38)
|
> **WARNING:** 30+ listening ports, load average 9.47, RAM 6.6/7.9GB.
|
||||||
- Hailo runtime (26 TOPS accelerator)
|
> This node needs service migration or hardware upgrade.
|
||||||
- PowerDNS (port 53)
|
|
||||||
- PowerDNS-Admin (port 8080)
|
|
||||||
- RoadAuth
|
|
||||||
- RoadAPI
|
|
||||||
- Auth-Gateway
|
|
||||||
- Cloudflared tunnel
|
|
||||||
|
|
||||||
### Aria (192.168.4.82)
|
---
|
||||||
- Hailo runtime (26 TOPS accelerator)
|
|
||||||
- Compute workloads (9 containers)
|
|
||||||
|
|
||||||
### Alice (192.168.4.49)
|
## Aria (192.168.4.82) — 30+ services
|
||||||
- Worker node (7 containers)
|
|
||||||
|
|
||||||
### Anastasia (192.168.4.33)
|
| Port Range | Service | Bind |
|
||||||
- (Pending service deployment)
|
|------------|---------|------|
|
||||||
|
| 3140-3167 | **28 Docker container ports** | 0.0.0.0 |
|
||||||
|
| 8081 | HTTP service | 0.0.0.0 |
|
||||||
|
| 8180 | Python service | 0.0.0.0 |
|
||||||
|
|
||||||
### Cordelia (192.168.4.27)
|
**Systemd services:** ollama, cloudflared, docker
|
||||||
- (Pending service deployment)
|
|
||||||
|
|
||||||
### Codex-Infinity (159.65.43.12)
|
> 28 container ports in 3140-3167 range. Disk 74% full — monitor closely.
|
||||||
- Codex database
|
|
||||||
- Cloud services
|
|
||||||
- Cloudflared tunnel
|
|
||||||
|
|
||||||
### Shellfish (174.138.44.45)
|
---
|
||||||
- Cloudflare tunnels
|
|
||||||
- Edge-agent
|
## Alice (192.168.4.49) — Minimal
|
||||||
|
|
||||||
|
| Port | Service |
|
||||||
|
|------|---------|
|
||||||
|
| 22 | SSH |
|
||||||
|
|
||||||
|
**Systemd services:** cloudflared, docker
|
||||||
|
|
||||||
|
> Light node. Load 6.17 is concerning for 4-core Pi 400 — investigate docker workloads.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Shellfish / "anastasia" (174.138.44.45) — 14+ services
|
||||||
|
|
||||||
|
| Port | Service | Process |
|
||||||
|
|------|---------|---------|
|
||||||
|
| 22 | SSH | sshd |
|
||||||
|
| 80 | HTTP | nginx |
|
||||||
|
| 3000 | Dashboard (Grafana?) | node |
|
||||||
|
| 3001 | Dashboard | node |
|
||||||
|
| 6379 | Redis-like | python3 |
|
||||||
|
| 8000 | API | uvicorn |
|
||||||
|
| 8080 | HTTP service | — |
|
||||||
|
| 8765 | WebSocket server | python3 |
|
||||||
|
| 8766 | WebSocket server | python3 |
|
||||||
|
| 8787 | Python service | python3 |
|
||||||
|
| 8888 | Python service | python3 |
|
||||||
|
| 11434 | Ollama | ollama (Tailscale-only: 100.64.0.1) |
|
||||||
|
|
||||||
|
**Systemd services:** cloudflared, docker, nginx, ollama
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Codex-Infinity / "gematria" (159.65.43.12) — 7 services
|
||||||
|
|
||||||
|
| Port | Service | Process |
|
||||||
|
|------|---------|---------|
|
||||||
|
| 22 | SSH | sshd |
|
||||||
|
| 53 | DNS (local resolver) | systemd-resolved |
|
||||||
|
| 80 | HTTP | nginx |
|
||||||
|
| 443 | HTTPS | nginx |
|
||||||
|
| 2019 | Caddy admin | caddy |
|
||||||
|
| 8011 | App service | — |
|
||||||
|
| 8787 | Python service | python3 |
|
||||||
|
| 11434 | **Ollama (PUBLIC!)** | ollama |
|
||||||
|
|
||||||
|
**Systemd services:** cloudflared, nginx, ollama
|
||||||
|
|
||||||
|
> **SECURITY NOTE:** Ollama is bound to `*:11434` (all interfaces) on this public-facing droplet.
|
||||||
|
> Consider restricting to localhost or Tailscale interface only.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Service Distribution Summary
|
||||||
|
|
||||||
|
| Service | Cecilia | Octavia | Aria | Alice | Shellfish | Codex-Inf |
|
||||||
|
|---------|---------|---------|------|-------|-----------|-----------|
|
||||||
|
| SSH | 22 | 22 | 22 | 22 | 22 | 22 |
|
||||||
|
| HTTP | 80 | 8000+ | — | — | 80 | 80 |
|
||||||
|
| HTTPS | — | — | — | — | — | 443 |
|
||||||
|
| Ollama | 11434 (lo) | 11434 (lo) | ✓ | — | 11434 (TS) | 11434 (**PUBLIC**) |
|
||||||
|
| PostgreSQL | 5432 | 5432 | — | — | — | — |
|
||||||
|
| Docker | ✓ | ✓ | ✓ | ✓ | ✓ | — |
|
||||||
|
| Cloudflared | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
|
||||||
|
| Hailo | ✓ | — | — | — | — | — |
|
||||||
|
| MinIO | 9000 | — | — | — | — | — |
|
||||||
|
| InfluxDB | 8086 | — | — | — | — | — |
|
||||||
|
| nginx | — | — | — | — | ✓ | ✓ |
|
||||||
|
|
||||||
|
### Ollama Deployment (4 nodes!)
|
||||||
|
|
||||||
|
Ollama runs on 4 of 6 reachable nodes:
|
||||||
|
1. **Cecilia** — localhost only (secure)
|
||||||
|
2. **Octavia** — localhost only (secure) + SSE bridge
|
||||||
|
3. **Shellfish** — Tailscale interface only (secure)
|
||||||
|
4. **Codex-Infinity** — **ALL INTERFACES** (security risk on public IP)
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -78,39 +163,28 @@ Which services run on which device, and on which ports.
|
|||||||
```
|
```
|
||||||
┌───────────┐
|
┌───────────┐
|
||||||
│ NATS │
|
│ NATS │
|
||||||
│ (Lucidia) │
|
│ (Lucidia) │ ← DOWN
|
||||||
└─────┬─────┘
|
└─────┬─────┘
|
||||||
│
|
│
|
||||||
┌───────────────┼───────────────┐
|
┌───────────────┼───────────────┐
|
||||||
│ │ │
|
│ │ │
|
||||||
┌──────┴──────┐ ┌─────┴─────┐ ┌──────┴──────┐
|
┌──────┴──────┐ ┌─────┴─────┐ ┌──────┴──────┐
|
||||||
│ Ollama │ │ Edge-Agent │ │ CECE OS │
|
│ Ollama │ │ Observ. │ │ CECE OS │
|
||||||
│(Lucidia/Cec)│ │(Lucidia) │ │ (Cecilia) │
|
│ (4 nodes) │ │ Stack │ │ (Cecilia) │
|
||||||
└─────────────┘ └───────────┘ └─────────────┘
|
└─────────────┘ │(Cecilia) │ └─────────────┘
|
||||||
│
|
│InfluxDB │
|
||||||
┌─────┴─────┐
|
│Loki │
|
||||||
│ Hailo RT │
|
│MinIO │
|
||||||
│(Cec/Oct/Ar)│
|
│NodeExport│
|
||||||
└───────────┘
|
└──────────┘
|
||||||
|
|
||||||
┌───────────┐ ┌───────────┐
|
┌───────────┐ ┌───────────┐
|
||||||
│ PowerDNS │ │ Auth-GW │
|
│ PostgreSQL│ │ Hailo RT │
|
||||||
│ (Octavia) │◄───────│ (Octavia) │
|
│(Cec + Oct)│ │(Cecilia) │ ← Only 1 confirmed
|
||||||
└───────────┘ └───────────┘
|
└───────────┘ └───────────┘
|
||||||
|
|
||||||
┌────────────────┐
|
┌────────────────┐
|
||||||
│ Cloudflared │ ← All production nodes
|
│ Cloudflared │ ← 5 nodes (all except Alice has tunnel)
|
||||||
│ (5 tunnels) │
|
│ (5 tunnels) │
|
||||||
└────────────────┘
|
└────────────────┘
|
||||||
```
|
```
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Planned Services
|
|
||||||
|
|
||||||
| Service | Port | Target Node | Purpose |
|
|
||||||
|---------|------|-------------|---------|
|
|
||||||
| Mosquitto MQTT | 1883 | Pi-Ops | IoT sensor pub/sub |
|
|
||||||
| Headscale | 443 | Alice | Self-hosted Tailscale control |
|
|
||||||
| Monitoring Dashboard | 3000 | Pi-Ops | Grafana/custom dashboard |
|
|
||||||
| Agent UI | 8080 | Jetson-Agent | Touch-based agent control |
|
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
# Network Topology
|
# Network Topology — Live Verified
|
||||||
|
|
||||||
BlackRoad fleet network architecture — LAN, Tailscale mesh, cloud entry points.
|
**Verified against ARP table, ping sweep, SSH probes, and DNS dig on 2026-02-21.**
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -8,76 +8,87 @@ BlackRoad fleet network architecture — LAN, Tailscale mesh, cloud entry points
|
|||||||
|
|
||||||
### Network Equipment
|
### Network Equipment
|
||||||
|
|
||||||
| Device | Model | Ports | Role |
|
| Device | Model | MAC | Role |
|
||||||
|--------|-------|-------|------|
|
|--------|-------|-----|------|
|
||||||
| Router/WiFi | TP-Link | — | Gateway (192.168.4.1), DHCP, WiFi |
|
| Router/WiFi | TP-Link | 44:ac:85:94:37:92 | Gateway (192.168.4.1), DHCP, WiFi |
|
||||||
| Switch | TP-Link TL-SG105 | 5-port Gigabit | Wired backbone |
|
| Switch | TP-Link TL-SG105 | — | 5-port Gigabit wired backbone |
|
||||||
| WiFi Card | TP-Link AX3000 PCIe WiFi 6 | — | High-speed wireless |
|
| Unknown TP-Link | TP-Link | 98:17:3c:38:db:78 | 192.168.4.44 — extender/smart plug? |
|
||||||
|
|
||||||
### Topology Diagram
|
### Live ARP Map (2026-02-21)
|
||||||
|
|
||||||
```
|
```
|
||||||
┌─────────────────────┐
|
┌─────────────────────┐
|
||||||
│ INTERNET │
|
│ INTERNET │
|
||||||
└──────────┬──────────┘
|
└──────────┬──────────┘
|
||||||
│
|
│
|
||||||
┌──────────┴──────────┐
|
┌──────────┴──────────┐
|
||||||
│ TP-Link Router │
|
│ TP-Link Router │
|
||||||
│ 192.168.4.1 │
|
│ 192.168.4.1 │
|
||||||
│ DHCP / WiFi / NAT │
|
│ 44:ac:85:94:37:92 │
|
||||||
└──────────┬──────────┘
|
└──────────┬──────────┘
|
||||||
│
|
│
|
||||||
┌──────────────────┼──────────────────┐
|
┌────────────────────┼────────────────────┐
|
||||||
│ │ │
|
│ │ │
|
||||||
┌──────┴──────┐ ┌──────┴──────┐ WiFi Clients
|
┌──────┴──────┐ ┌──────┴──────┐ WiFi Clients
|
||||||
│ TL-SG105 │ │ WiFi AP │
|
│ TL-SG105 │ │ WiFi AP │ (see below)
|
||||||
│ Gigabit SW │ │ (built-in) │
|
│ Gigabit SW │ │ (built-in) │
|
||||||
└┬───┬───┬───┬┘ └─────────────┘
|
└┬───┬───┬───┬┘ └─────────────┘
|
||||||
│ │ │ │
|
│ │ │ │
|
||||||
│ │ │ └── Cecilia .89 [Hailo-8, CECE OS]
|
│ │ │ └── Cecilia .89 88:a2:9e:3b:eb:72 [UP] Hailo-8
|
||||||
│ │ └────── Lucidia .81 [NATS, Ollama]
|
│ │ └────── Octavia .38 2c:cf:67:cf:fa:17 [UP] OVERLOADED
|
||||||
│ └────────── Aria .82 [Hailo-8, API]
|
│ └────────── Aria .82 88:a2:9e:0d:42:07 [UP]
|
||||||
└────────────── Octavia .38 [Hailo-8, DNS]
|
└────────────── Lucidia .81 (incomplete) [DOWN]
|
||||||
|
|
||||||
WiFi:
|
WiFi:
|
||||||
├── Alexandria .28 [MacBook Pro M1]
|
├── Alexandria .28 b0:be:83:66:cc:10 [UP] Apple Mac
|
||||||
├── Alice .49 [Pi 400]
|
├── Alice .49 d8:3a:dd:ff:98:87 [UP] Pi 400
|
||||||
├── Anastasia .33 [Pi 5]
|
├── Anastasia .33 60:92:c8:11:cf:7c [UP] Pi 5 (no SSH)
|
||||||
├── Cordelia .27 [Pi 5]
|
├── Cordelia .27 6c:4a:85:32:ae:72 [UP] Pi 5 (no SSH)
|
||||||
├── Athena .45 [Heltec LoRa ESP32]
|
├── Athena .45 d0:c9:07:50:51:ca [UP] ESP32 LoRa
|
||||||
├── Phoebe .88 [iPhone]
|
├── Iris .26 d4:be:dc:6c:61:6b [UP] Roku
|
||||||
├── Ares .90 [Xbox]
|
├── Ares .90 a0:4a:5e:2a:db:d2 [DOWN] Xbox
|
||||||
└── Iris .26 [Roku]
|
├── Phoebe .88 9e:0d:2a:82:99:96 [DOWN] iPhone (private MAC)
|
||||||
|
│
|
||||||
|
├── UNKNOWN .22 30:be:29:5b:24:5f [UP] Smart TV/IoT?
|
||||||
|
├── UNKNOWN .44 98:17:3c:38:db:78 [UP] TP-Link device
|
||||||
|
├── UNKNOWN .83 54:4c:8a:9b:09:3d [UP] Smart home module?
|
||||||
|
└── UNKNOWN .92 de:a2:b7:f3:f9:5d [DOWN] Apple (private MAC)
|
||||||
```
|
```
|
||||||
|
|
||||||
### IP Address Map
|
### Complete IP-to-MAC-to-Identity Table
|
||||||
|
|
||||||
| IP | Hostname | Type | Wired/WiFi |
|
| IP | MAC | OUI Vendor | Identity | Ping | SSH |
|
||||||
|----|----------|------|------------|
|
|----|-----|-----------|----------|------|-----|
|
||||||
| .1 | Router | TP-Link Gateway | — |
|
| .1 | 44:ac:85:94:37:92 | TP-Link | Router | UP | — |
|
||||||
| .26 | Iris | Roku | WiFi |
|
| .22 | 30:be:29:5b:24:5f | Unknown | **UNIDENTIFIED** | UP | — |
|
||||||
| .27 | Cordelia | Pi 5 | WiFi |
|
| .26 | d4:be:dc:6c:61:6b | Roku | Iris (streaming) | UP | — |
|
||||||
| .28 | Alexandria | MacBook Pro M1 | WiFi |
|
| .27 | 6c:4a:85:32:ae:72 | Raspberry Pi 5 | Cordelia | UP | REFUSED |
|
||||||
| .33 | Anastasia | Pi 5 | WiFi |
|
| .28 | b0:be:83:66:cc:10 | Apple | Alexandria (Mac M1) | UP | — |
|
||||||
| .38 | Octavia | Pi 5 | Wired |
|
| .33 | 60:92:c8:11:cf:7c | Raspberry Pi 5 | Anastasia (Pi) | UP | REFUSED |
|
||||||
| .45 | Athena | Heltec LoRa ESP32 | WiFi |
|
| .38 | 2c:cf:67:cf:fa:17 | Raspberry Pi | Octavia | UP | OK |
|
||||||
| .49 | Alice | Pi 400 | WiFi |
|
| .44 | 98:17:3c:38:db:78 | TP-Link | **UNIDENTIFIED** | UP | — |
|
||||||
| .81 | Lucidia | Pi 5 | Wired |
|
| .45 | d0:c9:07:50:51:ca | Espressif | Athena (ESP32 LoRa) | UP | — |
|
||||||
| .82 | Aria | Pi 5 | Wired |
|
| .49 | d8:3a:dd:ff:98:87 | Raspberry Pi | Alice (Pi 400) | UP | OK |
|
||||||
| .88 | Phoebe | iPhone | WiFi |
|
| .74 | (incomplete) | — | **STALE** (old Octavia IP) | DOWN | — |
|
||||||
| .89 | Cecilia | Pi 5 | Wired |
|
| .81 | (incomplete) | — | Lucidia (Pi 5) | **DOWN** | — |
|
||||||
| .90 | Ares | Xbox | WiFi |
|
| .82 | 88:a2:9e:0d:42:07 | Raspberry Pi 5 | Aria | UP | OK |
|
||||||
|
| .83 | 54:4c:8a:9b:09:3d | Unknown | **UNIDENTIFIED** | UP | — |
|
||||||
|
| .88 | 9e:0d:2a:82:99:96 | Private MAC | Phoebe (iPhone) | DOWN | — |
|
||||||
|
| .89 | 88:a2:9e:3b:eb:72 | Raspberry Pi 5 | Cecilia | UP | OK |
|
||||||
|
| .90 | a0:4a:5e:2a:db:d2 | Microsoft | Ares (Xbox) | DOWN | — |
|
||||||
|
| .92 | de:a2:b7:f3:f9:5d | Private MAC | **UNIDENTIFIED** | DOWN | — |
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Tailscale Mesh Overlay
|
## Tailscale Mesh Overlay — Corrected
|
||||||
|
|
||||||
Encrypted WireGuard mesh connecting on-premises and cloud nodes.
|
> **CRITICAL FIX:** Lucidia and Octavia Tailscale IPs were swapped in prior documentation.
|
||||||
|
> Corrected based on SSH config and live `ss` output showing Tailscale binding addresses.
|
||||||
|
|
||||||
```
|
```
|
||||||
┌───────────────┐
|
┌───────────────┐
|
||||||
│ Tailscale │
|
│ Tailscale │
|
||||||
│ Control Plane│
|
│ Coord Server │
|
||||||
└───────┬───────┘
|
└───────┬───────┘
|
||||||
│
|
│
|
||||||
┌───────────────────┼───────────────────┐
|
┌───────────────────┼───────────────────┐
|
||||||
@@ -91,83 +102,84 @@ Encrypted WireGuard mesh connecting on-premises and cloud nodes.
|
|||||||
│
|
│
|
||||||
┌────┼────┬────────┬─────────┐
|
┌────┼────┬────────┬─────────┐
|
||||||
│ │ │ │ │
|
│ │ │ │ │
|
||||||
Lucidia │ Aria Octavia Alice
|
Octavia │ Aria Lucidia Alice
|
||||||
100.66 │ 100.109 100.83 100.77
|
100.66 │ 100.109 100.83 100.77
|
||||||
.235.47 │ .14.17 .149.86 .210.18
|
.235.47 │ .14.17 .149.86 .210.18
|
||||||
│
|
│ (DOWN)
|
||||||
(Full mesh — every
|
(Full mesh — every
|
||||||
node can reach
|
node can reach
|
||||||
every other node)
|
every other node)
|
||||||
```
|
```
|
||||||
|
|
||||||
### Tailscale Node Table
|
| Node | Tailscale IP | SSH Alias | Verified By | Status |
|
||||||
|
|------|-------------|-----------|-------------|--------|
|
||||||
|
| Cecilia | 100.72.180.98 | cecilia-ts | SSH config | Active |
|
||||||
|
| Octavia | **100.66.235.47** | octavia-ts | SSH config + `ss` binding | Active |
|
||||||
|
| Lucidia | **100.83.149.86** | lucidia-ts | SSH config | **DOWN** |
|
||||||
|
| Aria | 100.109.14.17 | aria-ts | SSH config | Active |
|
||||||
|
| Alice | 100.77.210.18 | alice-ts | SSH config | Active |
|
||||||
|
| Codex-Infinity | 100.108.132.8 | gematria-ts / blackroad-os-ts | SSH config | Active |
|
||||||
|
| Shellfish | 100.94.33.37 | anastasia-ts / cadence-ts | SSH config + `ss` binding | Active |
|
||||||
|
|
||||||
| Node | Tailscale IP | OS | Connected |
|
### Not on Tailscale
|
||||||
|------|-------------|-----|-----------|
|
|
||||||
| Cecilia | 100.72.180.98 | Linux | Yes |
|
|
||||||
| Lucidia | 100.66.235.47 | Linux | Yes |
|
|
||||||
| Octavia | 100.83.149.86 | Linux | Yes |
|
|
||||||
| Aria | 100.109.14.17 | Linux | Yes |
|
|
||||||
| Alice | 100.77.210.18 | Linux | Yes |
|
|
||||||
| Codex-Infinity | 100.108.132.8 | Linux | Yes |
|
|
||||||
| Shellfish | 100.94.33.37 | Linux | Yes |
|
|
||||||
|
|
||||||
### SSH via Tailscale
|
- Alexandria (Mac) — tailscale not running
|
||||||
|
- Anastasia Pi (192.168.4.33) — SSH closed, can't configure
|
||||||
```bash
|
- Cordelia (192.168.4.27) — SSH closed, can't configure
|
||||||
ssh cecilia-ts # → 100.72.180.98
|
- Olympia — offline
|
||||||
ssh lucidia-ts # → 100.66.235.47
|
|
||||||
ssh octavia-ts # → 100.83.149.86
|
|
||||||
ssh aria-ts # → 100.109.14.17
|
|
||||||
ssh alice-ts # → 100.77.210.18
|
|
||||||
```
|
|
||||||
|
|
||||||
### Nodes Not Yet on Tailscale
|
|
||||||
|
|
||||||
- Anastasia (192.168.4.33)
|
|
||||||
- Cordelia (192.168.4.27)
|
|
||||||
- Olympia (offline)
|
|
||||||
- Jetson-Agent (pending setup)
|
|
||||||
- Pi-Holo, Pi-Ops, Pi-Zero-Sim (planned)
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Cloud Entry Points
|
## Cloud Entry Points — Cloudflare
|
||||||
|
|
||||||
### Cloudflare Tunnels
|
### DNS Resolution (all Cloudflare-proxied)
|
||||||
|
|
||||||
Each production node has a Cloudflare tunnel for HTTPS ingress:
|
All blackroad.io DNS resolves to Cloudflare CDN, not origin servers directly:
|
||||||
|
|
||||||
| Tunnel | Node | Config |
|
| Subdomain | A Record |
|
||||||
|--------|------|--------|
|
|-----------|----------|
|
||||||
| tunnel-cecilia.blackroad.io | Cecilia | /etc/cloudflared/config.yml |
|
| blackroad.io | 172.67.211.99 |
|
||||||
| tunnel-lucidia.blackroad.io | Lucidia | /etc/cloudflared/config.yml |
|
| www.blackroad.io | 172.67.211.99 |
|
||||||
| tunnel-octavia.blackroad.io | Octavia | /etc/cloudflared/config.yml |
|
| api.blackroad.io | 172.67.211.99 |
|
||||||
| tunnel-codex.blackroad.io | Codex-Infinity | /etc/cloudflared/config.yml |
|
| status.blackroad.io | 172.67.211.99 |
|
||||||
| tunnel-cadence.blackroad.io | Shellfish | /etc/cloudflared/config.yml |
|
| docs.blackroad.io | 172.67.211.99 |
|
||||||
|
| dashboard.blackroad.io | 172.67.211.99 |
|
||||||
|
| monitoring.blackroad.io | 172.67.211.99 |
|
||||||
|
| agents.blackroad.io | 104.21.91.74 |
|
||||||
|
| tunnel-cecilia.blackroad.io | 172.67.211.99 |
|
||||||
|
| tunnel-lucidia.blackroad.io | 104.21.91.74 |
|
||||||
|
| tunnel-octavia.blackroad.io | 172.67.211.99 |
|
||||||
|
| tunnel-codex.blackroad.io | 104.21.91.74 |
|
||||||
|
| tunnel-cadence.blackroad.io | 172.67.211.99 |
|
||||||
|
|
||||||
### DNS
|
**Traffic flow:** Client → Cloudflare CDN → Cloudflare Tunnel → `cloudflared` on origin node
|
||||||
|
|
||||||
- **Provider:** Cloudflare
|
### Cloudflare Tunnels (verified via `cloudflared.service`)
|
||||||
- **Zone:** blackroad.io
|
|
||||||
- **Internal DNS:** PowerDNS on Octavia
|
| Tunnel | Origin Node | cloudflared Status |
|
||||||
- **Time Sync:** chrony → time.cloudflare.com (all nodes)
|
|--------|-------------|-------------------|
|
||||||
|
| tunnel-cecilia.blackroad.io | Cecilia (192.168.4.89) | Running |
|
||||||
|
| tunnel-lucidia.blackroad.io | Lucidia (192.168.4.81) | **DOWN** (node unreachable) |
|
||||||
|
| tunnel-octavia.blackroad.io | Octavia (192.168.4.38) | Running |
|
||||||
|
| tunnel-codex.blackroad.io | Codex-Infinity (159.65.43.12) | Running |
|
||||||
|
| tunnel-cadence.blackroad.io | Shellfish (174.138.44.45) | Running |
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## LoRa Network (Planned)
|
## Stale Network Data to Clean Up
|
||||||
|
|
||||||
Athena (Heltec WiFi LoRa 32) serves as the LoRa mesh backbone:
|
| Item | Location | Issue | Fix |
|
||||||
|
|------|----------|-------|-----|
|
||||||
- **Frequency:** 868/915 MHz
|
| `/etc/hosts` entry | Alexandria Mac | `192.168.4.74 octavia` — wrong IP | Change to `192.168.4.38 octavia` |
|
||||||
- **Range:** Up to 10km line-of-sight
|
| `hailo.sh` | `~/hailo.sh` | Connects to `pi@192.168.4.74` | Change to `blackroad@192.168.4.38` |
|
||||||
- **Protocol:** LoRaWAN or point-to-point
|
| SSH `anastasia` alias | `~/.ssh/config` | Points to DO droplet, not Pi | Add `anastasia-pi` for 192.168.4.33 |
|
||||||
- **Use Cases:** Remote sensor relay, out-of-WiFi-range monitoring
|
| `blackroad-fleet.yaml` | `~/blackroad-fleet.yaml` | Lucidia/Octavia IPs swapped | Fix both local and Tailscale IPs |
|
||||||
- **Management:** `~/lora.sh`
|
| Agent registry | `~/.blackroad-agent-registry.db` | Octavia: `pironman_hailo8` | Change to `pironman` (no Hailo) |
|
||||||
|
| Agent registry | `~/.blackroad-agent-registry.db` | Aria: `pironman_hailo8` | Change to `pironman` (no Hailo) |
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Firewall Rules (All Nodes)
|
## Firewall Rules (Standard)
|
||||||
|
|
||||||
| Port | Protocol | Service | Direction |
|
| Port | Protocol | Service | Direction |
|
||||||
|------|----------|---------|-----------|
|
|------|----------|---------|-----------|
|
||||||
@@ -176,15 +188,41 @@ Athena (Heltec WiFi LoRa 32) serves as the LoRa mesh backbone:
|
|||||||
| 443 | TCP | HTTPS | Inbound |
|
| 443 | TCP | HTTPS | Inbound |
|
||||||
| 41641 | UDP | Tailscale | Inbound |
|
| 41641 | UDP | Tailscale | Inbound |
|
||||||
|
|
||||||
Default policy: **deny** all other inbound traffic.
|
Default policy: **deny** all other inbound.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## LoRa Network
|
||||||
|
|
||||||
|
Athena (Heltec WiFi LoRa 32) at 192.168.4.45:
|
||||||
|
|
||||||
|
- **Frequency:** 868/915 MHz
|
||||||
|
- **Range:** Up to 10km line-of-sight
|
||||||
|
- **MAC:** d0:c9:07:50:51:ca (Espressif OUI confirmed)
|
||||||
|
- **Status:** Powered on (responds to ARP)
|
||||||
|
- **Management:** `~/lora.sh`
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Network Diagnostics
|
## Network Diagnostics
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
~/blackroad-network-scan.sh # ARP + ping sweep + Tailscale status
|
# Live ARP table (shows devices seen recently)
|
||||||
~/blackroad-network-discovery.sh # SSH probe all known devices
|
arp -a | grep "192.168.4" | grep -v incomplete | sort -t. -k4 -n
|
||||||
tailscale status # Tailscale mesh state
|
|
||||||
tailscale ping <hostname> # Test Tailscale connectivity
|
# Ping sweep
|
||||||
|
for i in {1..255}; do ping -c1 -W1 192.168.4.$i &>/dev/null && echo "UP .${i}"; done
|
||||||
|
|
||||||
|
# Tailscale status
|
||||||
|
tailscale status
|
||||||
|
|
||||||
|
# SSH probe a node
|
||||||
|
ssh -o ConnectTimeout=3 cecilia "hostname; uname -r; ss -tlnp"
|
||||||
|
|
||||||
|
# DNS lookup
|
||||||
|
dig blackroad.io ANY +short
|
||||||
|
|
||||||
|
# Management scripts
|
||||||
|
~/blackroad-network-scan.sh
|
||||||
|
~/blackroad-network-discovery.sh
|
||||||
```
|
```
|
||||||
|
|||||||
Reference in New Issue
Block a user