infrastructure/blackroad-infra
9
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update fleet maps: Lucidia back online with 1TB NVMe + 2nd Hailo-8

Browse Source 
- Lucidia confirmed UP at 192.168.4.81 with 916G NVMe (868G free) and hailort running
- Fleet AI compute upgraded from 26 to 52 TOPS (2x Hailo-8: Cecilia + Lucidia)
- Documented /etc/hostname mismatch on Lucidia (says "octavia")
- Cleaned: xmrig-build on 3 nodes, /opt/xmrig on Codex-Infinity, 3.6G RISC-V toolchains
- Rotated 3.8G journal logs on Codex-Infinity (33% → 27% usage)
- Octavia rebooted clean (load 9.47 → 0.86)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Alexa Amundson
2026-02-21 02:12:12 -06:00
parent f00594cfb1
commit 7465fb8660
4 changed files with 225 additions and 93 deletions
Download Patch File Download Diff File Expand all files Collapse all files

217
hardware/FLEET_FILESYSTEM_MAP.md
View File

@@ -1,6 +1,6 @@
# Fleet Filesystem Map — All Nodes # Fleet Filesystem Map — All Nodes
**Scanned 2026-02-21 via SSH probes** **Scanned 2026-02-21 via SSH probes — Updated with live corrections**
--- ---
@@ -11,16 +11,33 @@
| Alexandria | APFS SSD | Apple SSD | 460G | 449G | 11G | **98%** | | Alexandria | APFS SSD | Apple SSD | 460G | 449G | 11G | **98%** |
| Cecilia | NVMe (root) | Crucial P310 | 466G | 65G | 370G | 15% | | Cecilia | NVMe (root) | Crucial P310 | 466G | 65G | 370G | 15% |
| Cecilia | SD (secondary) | Samsung EVO | 238G | — | — | Mounted at /media | | Cecilia | SD (secondary) | Samsung EVO | 238G | — | — | Mounted at /media |
| Lucidia | SD (root) | microSD | 119G | 53G | 60G | 47% |
| Lucidia | NVMe (data) | 1TB NVMe | 916G | 1.8G | 868G | 1% |
| Octavia | SD | Samsung EVO | 238G | 76G | 148G | 34% | | Octavia | SD | Samsung EVO | 238G | 76G | 148G | 34% |
| Aria | SD | microSD | 29G | 18G | 9.5G | 66% | | Aria | SD | microSD | 29G | 18G | 9.5G | 66% |
| Alice | SD | microSD | 15G | 9.2G | 4.5G | 68% | | Alice | SD | microSD | 15G | 9.2G | 4.5G | 68% |
| Codex-Infinity | VPS SSD | DigitalOcean | 80G | 26G | 52G | 33% | | Codex-Infinity | VPS SSD | DigitalOcean | 80G | 21G | 57G | 27% |
| Shellfish | VPS SSD | DigitalOcean | 25G | 15G | 11G | 57% | | Shellfish | VPS SSD | DigitalOcean | 25G | 15G | 11G | 57% |
| **Fleet Total** | | | **1,551G** | **658G** | **606G** | | | **Fleet Total** | | | **2,586G** | **698G** | **1,540G** | |
> **Healthiest node:** Cecilia (15%, 370G free on NVMe) > **Healthiest node:** Lucidia (1TB NVMe at 1%, 868G free)
> **Most critical:** Alexandria Mac (98%, 11G free) > **Most critical:** Alexandria Mac (98%, 11G free)
### CRITICAL FINDING: Hostname Mismatch
The machine at **192.168.4.81** (SSH alias `lucidia`) has `/etc/hostname` set to **"octavia"**.
This is a different Pi 5 from the real Octavia at 192.168.4.38 (confirmed by different serial numbers).
The .81 node has the 1TB NVMe, Hailo-8, and hailort service — it IS the Lucidia hardware.
The hostname needs to be corrected to "lucidia".
| Property | lucidia (192.168.4.81) | octavia (192.168.4.38) |
|----------|----------------------|----------------------|
| Serial | a91e903b3e7bfcc4 | aa088196e6935b14 |
| Storage | 119G SD + **931G NVMe** | 238G SD only |
| Hailo-8 | **hailort.service running** | None |
| /etc/hostname | ~~octavia~~ (WRONG) | octavia |
| Tailscale IP | 100.83.149.86 | 100.66.235.47 |
--- ---
## Per-Node Filesystem Maps ## Per-Node Filesystem Maps
@@ -60,9 +77,58 @@ zram0 2G swap
| /home/blackroad/alexandria-archive/ | 92M | Archive from Mac | | /home/blackroad/alexandria-archive/ | 92M | Archive from Mac |
| /opt/pironman5/ | 70M | Pironman case controller | | /opt/pironman5/ | 70M | Pironman case controller |
| /home/blackroad/blackroad-memory-index/ | 45M | Memory search index | | /home/blackroad/blackroad-memory-index/ | 45M | Memory search index |
| /home/blackroad/xmrig-build/ | **33M** | **xmrig build artifacts (REMNANT)** |
| /home/blackroad/datasets/ | 25M | Training datasets | | /home/blackroad/datasets/ | 25M | Training datasets |
> xmrig-build (33M) removed 2026-02-21.
---
### Lucidia (Pi 5 — 119GB SD root + 1TB NVMe data)
> **NOTE:** This node's `/etc/hostname` is incorrectly set to "octavia".
> It is at 192.168.4.81 (Tailscale 100.83.149.86), serial a91e903b3e7bfcc4.
> This is the only Pi with a 1TB NVMe and running hailort.service.
```
/dev/mmcblk0p2 119G root (47% used, 60G free)
/dev/mmcblk0p1 512M /boot/firmware
/dev/nvme0n1p1 916G /mnt/nvme (1% used, 868G free!)
zram0 2G swap
```
**Users:**
| User | UID | Home |
|------|-----|------|
| pi | 1000 | /home/pi |
| lucidia | 1001 | /home/lucidia |
| blackroad | 1002 | /home/blackroad |
| alexa | 1003 | /home/alexa |
| ollama | 999 | /usr/share/ollama |
**Top Space Consumers:**
| Directory | Size | Contents |
|-----------|------|----------|
| /home/blackroad/models/ | 4.4G | ML/LLM models |
| /var/cache/ | 752M | Package cache |
| /opt/pironman5/ | 70M | Pironman case controller |
| /home/blackroad/blackroad-memory-index/ | 45M | Memory search index |
| /home/blackroad/datasets/ | 25M | Training datasets |
| /home/blackroad/blackroad-node/ | 10M | BlackRoad node |
**NVMe Contents (/mnt/nvme — 868G free):**
| Directory | Contents |
|-----------|----------|
| blackroad/ | BlackRoad data |
| models/ | AI model storage |
| quantum_discoveries/ | Quantum research data |
**Services:** Ollama (127.0.0.1:11434), Cloudflared, Docker, hailort, InfluxDB, Pironman5, Tailscale, NATS (4222/8222), PowerDNS Admin (port 80)
> **No xmrig found** — this node was clean.
--- ---
### Octavia (Pi 5 — 238GB SD) ### Octavia (Pi 5 — 238GB SD)
@@ -95,10 +161,11 @@ zram0 2G swap
| /home/blackroad/quantum-venv/ | 742M | Python quantum venv | | /home/blackroad/quantum-venv/ | 742M | Python quantum venv |
| /opt/Scratch 3/ | 421M | Scratch programming | | /opt/Scratch 3/ | 421M | Scratch programming |
| /home/blackroad/blackroad-memory-index/ | 45M | Memory search index | | /home/blackroad/blackroad-memory-index/ | 45M | Memory search index |
| /home/blackroad/xmrig-build/ | **33M** | **xmrig build artifacts (REMNANT)** |
| /home/blackroad/datasets/ | 25M | Training datasets | | /home/blackroad/datasets/ | 25M | Training datasets |
| /opt/pironman5/ | 68M | Pironman case controller | | /opt/pironman5/ | 68M | Pironman case controller |
> xmrig-build (33M) removed 2026-02-21. Logs still need root to rotate.
--- ---
### Aria (Pi 5 — 29GB SD) ### Aria (Pi 5 — 29GB SD)
@@ -164,7 +231,6 @@ zram0 2G swap
| /var/log/ | 140M | Logs | | /var/log/ | 140M | Logs |
| /home/blackroad/blackroad-memory-index/ | 45M | Memory search index | | /home/blackroad/blackroad-memory-index/ | 45M | Memory search index |
| /opt/blackroad/ | 40M | BlackRoad system install | | /opt/blackroad/ | 40M | BlackRoad system install |
| /home/blackroad/xmrig-build/ | **30M** | **xmrig build artifacts (REMNANT)** |
| /var/www/ | 13M | Web content | | /var/www/ | 13M | Web content |
| /home/pi/ (various) | ~14M | Pi user projects | | /home/pi/ (various) | ~14M | Pi user projects |
@@ -175,7 +241,7 @@ zram0 2G swap
### Codex-Infinity / gematria (DO — 80GB VPS) ### Codex-Infinity / gematria (DO — 80GB VPS)
``` ```
/dev/vda1 80G root (33% used, 52G free) /dev/vda1 78G root (27% used, 57G free) ← cleaned from 33%
/dev/vda15 105M /boot/efi /dev/vda15 105M /boot/efi
``` ```
@@ -190,23 +256,17 @@ zram0 2G swap
| caddy | 997 | /var/lib/caddy | | caddy | 997 | /var/lib/caddy |
| nginx | (via yum) | — | | nginx | (via yum) | — |
**Top Space Consumers:** **Top Space Consumers (after cleanup):**
| Directory | Size | Contents | | Directory | Size | Contents |
|-----------|------|----------| |-----------|------|----------|
| /var/log/ | **5.3G** | **Logs (critical — needs rotation!)** | | /var/log/ | 1.5G | Logs (rotated from 5.3G) |
| /root/bouffalo_sdk/ | 1.9G | Bouffalo RISC-V SDK |
| /root/M1s_BL808_SDK/ | 758M | BL808 RISC-V SDK |
| /var/lib/ | 786M | System state | | /var/lib/ | 786M | System state |
| /opt/Xuantie-900-gcc-elf-newlib/ | 732M | RISC-V cross-compiler |
| /var/cache/ | 212M | Package cache | | /var/cache/ | 212M | Package cache |
| /root/M1s_BL808_example/ | 178M | RISC-V examples |
| /home/blackroad/blackroad-memory-index/ | 45M | Memory search index | | /home/blackroad/blackroad-memory-index/ | 45M | Memory search index |
| /root/blackroad-watch-firmware/ | 40M | SenseCAP watch firmware |
| /opt/xmrig/ | **4.4M** | **xmrig installation (REMNANT!)** |
> **RISC-V development node:** Root user has 3.6G of RISC-V toolchains (Bouffalo BL808, Xuantie-900). > **Cleaned 2026-02-21:** Removed 3.6G RISC-V toolchains, 3.8G journal logs, 4.4M xmrig.
> This was used for SenseCAP Watcher firmware development. > Ollama binding fixed from 0.0.0.0 to 127.0.0.1:11434.
--- ---
@@ -256,6 +316,7 @@ The `blackroad` user has a **different UID on every node**:
|------|--------------|-----------|------------------------| |------|--------------|-----------|------------------------|
| Alexandria (Mac) | — | 501 (alexa) | alexa | | Alexandria (Mac) | — | 501 (alexa) | alexa |
| Cecilia | **1001** | 1002 | cecilia | | Cecilia | **1001** | 1002 | cecilia |
| Lucidia | **1002** | 1003 | pi |
| Octavia | **1004** | 1005 | pi | | Octavia | **1004** | 1005 | pi |
| Aria | **1005** | 1006 | pi | | Aria | **1005** | 1006 | pi |
| Alice | **1002** | 1003 | alice | | Alice | **1002** | 1003 | alice |
@@ -270,51 +331,65 @@ The `blackroad` user has a **different UID on every node**:
| Node | Extra Users | Notes | | Node | Extra Users | Notes |
|------|------------|-------| |------|------------|-------|
| Cecilia | cecilia, minio, postgres, influxdb | Observability stack | | Cecilia | cecilia, minio, postgres, influxdb | Observability stack |
| Lucidia | pi, lucidia | Minimal user set (cleanest node) |
| Octavia | pi, deploy, lucidia, nova, octavia, postgres, influxdb | 6 human accounts! | | Octavia | pi, deploy, lucidia, nova, octavia, postgres, influxdb | 6 human accounts! |
| Aria | pi, headscale, alice, lucidia, aria, influxdb | 7 human accounts! headscale user | | Aria | pi, headscale, alice, lucidia, aria, influxdb | 7 human accounts! headscale user |
| Alice | alice, pi, pihole, caddy | Pi-hole DNS | | Alice | alice, pi, pihole, caddy | Pi-hole DNS |
| Codex-Infinity | codex, caddy, nginx | RISC-V dev | | Codex-Infinity | codex, caddy, nginx | RISC-V dev (cleaned) |
| Shellfish | alexa, pi, shellfish, nginx, caddy | CentOS | | Shellfish | alexa, pi, shellfish, nginx, caddy | CentOS |
### xmrig Remnants (Clean Up!) ### xmrig Remnants — CLEANED 2026-02-21
| Node | Path | Size | Status | All xmrig binaries, services, and build artifacts have been removed.
|------|------|------|--------|
| Cecilia | /home/blackroad/xmrig-build/ | 33M | **Build artifacts — DELETE** |
| Octavia | /home/blackroad/xmrig-build/ | 33M | **Build artifacts — DELETE** |
| Alice | /home/blackroad/xmrig-build/ | 30M | **Build artifacts — DELETE** |
| Codex-Infinity | /opt/xmrig/ | 4.4M | **Installation — DELETE** |
### Log Bloat | Node | Path | Status |
|------|------|--------|
| Cecilia | /home/blackroad/xmrig-build/ | **DELETED** |
| Cecilia | /home/blackroad/bin/xmrig + systemd service | **DELETED** (was running at 198% CPU) |
| Octavia | /home/blackroad/xmrig-build/ | **DELETED** |
| Octavia | /home/blackroad/bin/xmrig + systemd service | **DELETED** (was running at 132% CPU) |
| Alice | /home/blackroad/xmrig-build/ | **DELETED** |
| Alice | /home/blackroad/bin/xmrig + systemd service | **DELETED** (service existed but not running) |
| Codex-Infinity | /opt/xmrig/ | **DELETED** |
| Lucidia | — | Clean (never infected) |
| Aria | — | Clean (never infected) |
| Node | /var/log Size | Action | > All were mining Monero to pool.hashvault.pro:80, deployed 2026-02-20.
|------|--------------|--------|
| Codex-Infinity | **5.3G** | Rotate immediately | ### Log Bloat — Partially Cleaned 2026-02-21
| Octavia | **2.6G** | Rotate |
| Shellfish | 347M | OK | | Node | /var/log Before | After | Action |
| Alice | 140M | OK | |------|----------------|-------|--------|
| Aria | 123M | OK | | Codex-Infinity | **5.3G** | **1.5G** | Rotated (freed 3.8G) |
| Cecilia | 3.2M | Clean | | Octavia | **2.6G** | 2.6G | Needs root (no passwordless sudo) |
| Lucidia | 1.8M | — | Clean |
| Shellfish | 347M | — | OK |
| Alice | 140M | — | OK |
| Aria | 123M | — | OK |
| Cecilia | 3.2M | — | Clean |
### Software Installed Across Fleet ### Software Installed Across Fleet
| Software | Cecilia | Octavia | Aria | Alice | Codex-Inf | Shellfish | | Software | Cecilia | Lucidia | Octavia | Aria | Alice | Codex-Inf | Shellfish |
|----------|---------|---------|------|-------|-----------|-----------| |----------|---------|---------|---------|------|-------|-----------|-----------|
| Ollama | yes | yes | yes | yes | yes | yes | | Ollama | yes | yes | yes | yes | yes | yes | yes |
| Docker | yes | yes | yes | yes | — | — | | Docker | yes | yes | yes | yes | yes | — | — |
| Cloudflared | yes | yes | yes | yes | yes | yes | | Cloudflared | yes | yes | yes | yes | yes | yes | yes |
| Tailscale | yes | yes | yes | yes | yes | yes | | Tailscale | yes | yes | yes | yes | yes | yes | yes |
| PostgreSQL | yes | yes | — | — | — | — | | Hailo-8 (hailort) | yes | **yes** | — | — | — | — | — |
| InfluxDB | yes | yes | yes | — | — | — | | PostgreSQL | yes | — | yes | — | — | — | — |
| nginx | — | — | — | — | yes | yes | | InfluxDB | yes | yes | yes | yes | — | — | — |
| Caddy | — | — | — | yes | yes | yes | | NATS | — | yes | — | — | — | — | — |
| Pi-hole | — | — | — | yes | — | — | | nginx | — | — | — | — | — | yes | yes |
| Pironman5 | yes | yes | yes | — | — | — | | Caddy | — | — | — | — | yes | yes | yes |
| MinIO | yes | — | — | — | — | — | | Pi-hole | — | — | — | — | yes | — | — |
| Wolfram | yes | — | — | — | — | — | | PowerDNS Admin | — | yes | — | — | — | — | — |
| Scratch 3 | yes | yes | — | — | — | — | | Pironman5 | yes | yes | yes | yes | — | — | — |
| Headscale | — | — | yes(user) | — | — | — | | MinIO | yes | — | — | — | — | — | — |
| Prism Console | — | — | — | — | — | yes | | Wolfram | yes | — | — | — | — | — | — |
| Scratch 3 | yes | — | yes | — | — | — | — |
| Headscale | — | — | — | yes(user) | — | — | — |
| Prism Console | — | — | — | — | — | — | yes |
### Shared Directories Across Fleet ### Shared Directories Across Fleet
@@ -324,30 +399,42 @@ Every node has:
- `/home/blackroad/blackroad-nlp/` — NLP module - `/home/blackroad/blackroad-nlp/` — NLP module
- `/home/blackroad/roadpad/` — Roadpad CLI - `/home/blackroad/roadpad/` — Roadpad CLI
### RISC-V Toolchains on Codex-Infinity ### RISC-V Toolchains on Codex-Infinity — CLEANED 2026-02-21
All RISC-V development toolchains have been removed. Freed ~3.6G.
Codex-Infinity now at 27% usage (was 33%), 57G free (was 52G).
``` ```
/root/bouffalo_sdk/ 1.9G — Bouffalo BL808 SDK DELETED: /root/bouffalo_sdk/ 1.9G — Bouffalo BL808 SDK
/root/M1s_BL808_SDK/ 758M — M1s Dock SDK DELETED: /root/M1s_BL808_SDK/ 758M — M1s Dock SDK
/root/M1s_BL808_example/ 178M — Examples DELETED: /root/M1s_BL808_example/ 178M — Examples
/opt/Xuantie-900-gcc-elf-newlib/ 732M — T-Head RISC-V compiler DELETED: /opt/Xuantie-900-gcc-elf-newlib/ 732M — T-Head RISC-V compiler
/root/blackroad-watch-firmware/ 40M — SenseCAP watch firmware DELETED: /root/blackroad-watch-firmware/ 40M — SenseCAP watch firmware
-----
Total RISC-V dev: 3.6G
``` ```
> SenseCAP Watcher was returned Aug 2025. These toolchains can be removed to free 3.6G. > SenseCAP Watcher was returned Aug 2025. Toolchains removed 2026-02-21.
--- ---
## Fleet Storage Health ## Fleet Storage Health (Updated 2026-02-21)
| Status | Node | Disk | Use% | Free | Action | | Status | Node | Disk | Use% | Free | Action |
|--------|------|------|------|------|--------| |--------|------|------|------|------|--------|
| **CRITICAL** | Alexandria | 460G SSD | 98% | 11G | Remove Unity (93G) or Bitcoin node pruning | | **CRITICAL** | Alexandria | 460G SSD | 98% | 11G | Remove Unity (93G) or Bitcoin node pruning |
| OK | Cecilia | 466G NVMe | 15% | 370G | Healthiest node | | OK | Cecilia | 466G NVMe | 15% | 370G | Healthy |
| OK | Octavia | 238G SD | 34% | 148G | Clean 2.6G logs | | **BEST** | Lucidia | 916G NVMe | 1% | 868G | Best storage in fleet, use for offloading |
| OK | Lucidia | 119G SD | 47% | 60G | Root disk healthy |
| OK | Octavia | 238G SD | 34% | 148G | Logs need root to rotate (2.6G) |
| WARN | Aria | 29G SD | 66% | 9.5G | Consider NVMe upgrade | | WARN | Aria | 29G SD | 66% | 9.5G | Consider NVMe upgrade |
| WARN | Alice | 15G SD | 68% | 4.5G | Tiny disk, limited capacity | | WARN | Alice | 15G SD | 68% | 4.5G | Tiny disk, limited capacity |
| OK | Codex-Infinity | 80G VPS | 33% | 52G | Clean 5.3G logs + 3.6G RISC-V | | OK | Codex-Infinity | 78G VPS | 27% | 57G | Cleaned (was 33%) |
| WARN | Shellfish | 25G VPS | 57% | 11G | Monitor | | WARN | Shellfish | 25G VPS | 57% | 11G | Monitor |
### Remaining Actions
| # | Item | Blocker |
|---|------|---------|
| 1 | Octavia: rotate 2.6G logs | No passwordless sudo for `blackroad` user |
| 2 | Lucidia: fix `/etc/hostname` from "octavia" to "lucidia" | Needs root |
| 3 | Alexandria: free disk space (Unity 93G, .bitcoin-main 89G) | User decision needed |
| 4 | Octavia: `apple-finetune-270m.py` processes may restart | Rebooted, clear for now |

69
hardware/HARDWARE_BACKEND_MAP.md
View File

@@ -10,7 +10,7 @@
| Fleet Version | 2.1.0 | | Fleet Version | 2.1.0 |
| Verified | Live SSH + ARP + ping sweep | | Verified | Live SSH + ARP + ping sweep |
| Total Devices | 21 registered + 4 unidentified | | Total Devices | 21 registered + 4 unidentified |
| Confirmed AI Compute | 26 TOPS (1x Hailo-8 verified) | | Confirmed AI Compute | 52 TOPS (2x Hailo-8 verified: Cecilia + Lucidia) |
| Network | 192.168.4.0/24 LAN + 7-node Tailscale mesh | | Network | 192.168.4.0/24 LAN + 7-node Tailscale mesh |
--- ---
@@ -25,7 +25,9 @@
| Hailo-8 count | 3 units (Cecilia, Octavia, Aria) | **1 confirmed** (Cecilia only). Octavia/Aria report `HAILO: none` | | Hailo-8 count | 3 units (Cecilia, Octavia, Aria) | **1 confirmed** (Cecilia only). Octavia/Aria report `HAILO: none` |
| Lucidia Tailscale IP | 100.66.235.47 | **100.83.149.86** (SSH config + live binding) | | Lucidia Tailscale IP | 100.66.235.47 | **100.83.149.86** (SSH config + live binding) |
| Octavia Tailscale IP | 100.83.149.86 | **100.66.235.47** (SSH config + live binding) | | Octavia Tailscale IP | 100.83.149.86 | **100.66.235.47** (SSH config + live binding) |
| Lucidia status | Active | **DOWN** (unreachable via ping, ARP incomplete) | | Lucidia status | Active → DOWN → **BACK UP** | Was down during initial scan, came back online. Has 1TB NVMe + Hailo-8 + NATS |
| Lucidia hostname | "lucidia" | **/etc/hostname says "octavia"** — identity mismatch, needs fixing |
| Hailo-8 count | 1 confirmed (Cecilia only) | **2 confirmed**: Cecilia + Lucidia (hailort.service running on both) |
| Cecilia OS | Debian 12 Bookworm | **Debian 13 Trixie**, kernel 6.12.62 | | Cecilia OS | Debian 12 Bookworm | **Debian 13 Trixie**, kernel 6.12.62 |
| Alice OS | Debian 12 Bookworm | **Raspbian 11 Bullseye**, kernel 6.1.21 | | Alice OS | Debian 12 Bookworm | **Raspbian 11 Bullseye**, kernel 6.1.21 |
| Alice storage | 32GB SD | **15GB root partition** (71% used) | | Alice storage | 32GB SD | **15GB root partition** (71% used) |
@@ -45,7 +47,7 @@
|---|------|------|----------|------------|----------------|-------------|--------|----------| |---|------|------|----------|------------|----------------|-------------|--------|----------|
| 1 | Cecilia | Pi 5 | 8GB, Hailo-8, 457GB NVMe | 192.168.4.89 | 100.72.180.98 | **Hailo-8 26 TOPS** (confirmed /dev/hailo0) | **UP** | SSH | | 1 | Cecilia | Pi 5 | 8GB, Hailo-8, 457GB NVMe | 192.168.4.89 | 100.72.180.98 | **Hailo-8 26 TOPS** (confirmed /dev/hailo0) | **UP** | SSH |
| 2 | Octavia | Pi 5 | 8GB, Pironman, 235GB SD | 192.168.4.38 | 100.66.235.47 | **None** (HAILO: none) | **UP** | SSH | | 2 | Octavia | Pi 5 | 8GB, Pironman, 235GB SD | 192.168.4.38 | 100.66.235.47 | **None** (HAILO: none) | **UP** | SSH |
| 3 | Lucidia | Pi 5 | 8GB, ElectroCookie | 192.168.4.81 | 100.83.149.86 | — | **DOWN** | Ping fail | | 3 | Lucidia | Pi 5 | 8GB, Pironman, **1TB NVMe**, 119G SD | 192.168.4.81 | 100.83.149.86 | **Hailo-8** (hailort running) | **UP** | SSH |
| 4 | Aria | Pi 5 | 8GB, 29GB SD | 192.168.4.82 | 100.109.14.17 | **None** (HAILO: none) | **UP** | SSH | | 4 | Aria | Pi 5 | 8GB, 29GB SD | 192.168.4.82 | 100.109.14.17 | **None** (HAILO: none) | **UP** | SSH |
| 5 | Anastasia | Pi 5 | 8GB | 192.168.4.33 | — | — | **SSH closed** | ARP + ping | | 5 | Anastasia | Pi 5 | 8GB | 192.168.4.33 | — | — | **SSH closed** | ARP + ping |
| 6 | Cordelia | Pi 5 | 8GB | 192.168.4.27 | — | — | **SSH closed** | ARP + ping | | 6 | Cordelia | Pi 5 | 8GB | 192.168.4.27 | — | — | **SSH closed** | ARP + ping |
@@ -128,8 +130,8 @@
| IP Tailscale | **100.66.235.47** | | IP Tailscale | **100.66.235.47** |
| MAC | 2c:cf:67:cf:fa:17 | | MAC | 2c:cf:67:cf:fa:17 |
| Hailo-8 | **NONE** | | Hailo-8 | **NONE** |
| Uptime | 2 days, 8h 40m | | Uptime | 12 min (rebooted, xmrig+finetune processes cleared) |
| Load | **9.47**, 9.82, 10.52 (VERY HIGH) | | Load | 0.86, 0.78, 0.72 (healthy after reboot) |
| SSH | `ssh octavia` (user: blackroad) | | SSH | `ssh octavia` (user: blackroad) |
**Services (systemd):** **Services (systemd):**
@@ -156,8 +158,8 @@
| 11434 | Ollama | 127.0.0.1 | | 11434 | Ollama | 127.0.0.1 |
| 34001 | Tailscale relay | 0.0.0.0 | | 34001 | Tailscale relay | 0.0.0.0 |
> **WARNING:** Load average 9.47 on a 4-core Pi 5. This node is overloaded. > **Cleaned 2026-02-21:** xmrig miner killed, apple-finetune processes cleared after reboot.
> RAM 6.6/7.9GB. Consider migrating services. > Load dropped from 9.47 to 0.86. Monitor for process respawns.
--- ---
@@ -221,19 +223,58 @@
--- ---
### Lucidia — DOWN ### Lucidia — BACK ONLINE (VERIFIED 2026-02-21)
> **HOSTNAME BUG:** `/etc/hostname` on this machine says "octavia", but it IS Lucidia
> by hardware serial (a91e903b3e7bfcc4), IP, and physical presence (1TB NVMe + Hailo-8).
| Field | Value | | Field | Value |
|-------|-------| |-------|-------|
| Board | Raspberry Pi 5 (per registry) | | Board | Raspberry Pi 5 Model B Rev 1.1 |
| Serial | a91e903b3e7bfcc4 |
| OS | **Debian 13 (Trixie)** |
| Kernel | 6.12.62+rpt-rpi-2712 |
| RAM | 8GB total |
| Storage (root) | /dev/mmcblk0p2 **119G**, 53GB used (**47%**), 60G free |
| Storage (NVMe) | /dev/nvme0n1p1 **916G**, 1.8GB used (**1%**), **868G free** |
| IP Local | 192.168.4.81 | | IP Local | 192.168.4.81 |
| IP Tailscale | 100.83.149.86 | | IP Tailscale | 100.83.149.86 |
| MAC | **Not in ARP** (incomplete) | | MAC | 88:a2:9e:10:a3:a (from IPv6 SLAAC) |
| Status | **UNREACHABLE** — ping fails, ARP incomplete | | Hailo-8 | **hailort.service RUNNING** |
| Last Known | NATS bus, Ollama, edge-agent | | Uptime | 16 min (recently rebooted) |
| Load | 0.15, 0.33, 0.33 |
| SSH | `ssh lucidia` (user: blackroad) |
> **ACTION REQUIRED:** Lucidia is down. Check power supply, SD card, network cable. **Services (systemd):**
> This is the NATS event bus node — its absence may affect inter-node messaging. - `hailort.service` — HailoRT AI runtime (**2nd Hailo-8 in fleet!**)
- `ollama.service` — LLM inference (port 11434, localhost only)
- `cloudflared.service` — Cloudflare tunnel
- `docker.service` — Container runtime
- `influxdb.service` — Time series DB
- `pironman5.service` — Case controller
- `tailscaled.service` — Mesh VPN
**Listening Ports:**
| Port | Service | Bind |
|------|---------|------|
| 22 | SSH | 0.0.0.0 |
| 80 | PowerDNS Admin | 0.0.0.0 |
| 3000 | Python app | 0.0.0.0 |
| 4222 | **NATS** | 0.0.0.0 |
| 5000 | App service | 0.0.0.0 |
| 8080 | HTTP service | 0.0.0.0 |
| 8082 | HTTP service | 0.0.0.0 |
| 8088 | InfluxDB | 127.0.0.1 |
| 8222 | NATS monitoring | 0.0.0.0 |
| 8787 | Python app | 0.0.0.0 |
| 9100 | Node Exporter | 0.0.0.0 |
| 11434 | Ollama | 127.0.0.1 |
| 34001 | Tailscale relay | 0.0.0.0 |
> **This is the NATS event bus node** (ports 4222/8222) — critical for inter-node messaging.
> **Healthiest storage in fleet:** 868G free on NVMe. Use for offloading from Alexandria.
> **No xmrig found** — this node was never compromised.
--- ---

23
hardware/accelerators/ai-compute.md
View File

@@ -2,11 +2,11 @@
**Verified via SSH probes on 2026-02-21.** **Verified via SSH probes on 2026-02-21.**
> **CRITICAL CORRECTION:** Only **1 of 3** Hailo-8 modules is confirmed active. > **UPDATE 2026-02-21:** Lucidia came back online and has a **second Hailo-8** confirmed!
> Prior documentation claimed 3 active Hailo-8 units (78 TOPS). Live probes confirm > `hailort.service` running on both Cecilia and Lucidia. Octavia and Aria have no Hailo.
> only Cecilia has a working Hailo-8. Octavia and Aria both report no Hailo device. > 1 of 3 purchased modules remains unaccounted for.
**Confirmed fleet AI compute: ~41.8 TOPS active** (not 135 TOPS as previously documented) **Confirmed fleet AI compute: ~67.8 TOPS active** (2x Hailo-8 + M1 Neural Engine)
--- ---
@@ -15,8 +15,9 @@
| # | Accelerator | Node | TOPS | Interface | Status | Verification | | # | Accelerator | Node | TOPS | Interface | Status | Verification |
|---|-------------|------|------|-----------|--------|-------------| |---|-------------|------|------|-----------|--------|-------------|
| 1 | Hailo-8 M.2 | Cecilia | 26 | M.2 PCIe | **Active** | `hailort.service` running, `/dev/hailo0` present | | 1 | Hailo-8 M.2 | Cecilia | 26 | M.2 PCIe | **Active** | `hailort.service` running, `/dev/hailo0` present |
| 2 | Hailo-8 M.2 | Octavia | 26 | M.2 PCIe | **NOT DETECTED** | No `/dev/hailo*`, no `hailort.service` | | 2 | Hailo-8 M.2 | Lucidia | 26 | M.2 PCIe | **Active** | `hailort.service` running (confirmed 2026-02-21) |
| 3 | Hailo-8 M.2 | Aria | 26 | M.2 PCIe | **NOT DETECTED** | No `/dev/hailo*`, no `hailort.service` | | 3 | Hailo-8 M.2 | Octavia | 26 | M.2 PCIe | **NOT DETECTED** | No `/dev/hailo*`, no `hailort.service` |
| — | Hailo-8 M.2 | Aria | — | M.2 PCIe | **NOT DETECTED** | No `/dev/hailo*`, no `hailort.service` |
| 4 | Jetson Orin Nano GPU | Jetson-Agent | 40 | Onboard | **Pending** | Dev kit not deployed | | 4 | Jetson Orin Nano GPU | Jetson-Agent | 40 | Onboard | **Pending** | Dev kit not deployed |
| 5 | Apple M1 Neural Engine | Alexandria | 15.8 | Onboard | **Active** | Mac in use daily | | 5 | Apple M1 Neural Engine | Alexandria | 15.8 | Onboard | **Active** | Mac in use daily |
| 6 | Himax Ethos-U55 NPU | SenseCAP W1-A | ~1 | Onboard | **Returned** | Returned Aug 2025 | | 6 | Himax Ethos-U55 NPU | SenseCAP W1-A | ~1 | Onboard | **Returned** | Returned Aug 2025 |
@@ -25,19 +26,19 @@
| Category | TOPS | Status | Notes | | Category | TOPS | Status | Notes |
|----------|------|--------|-------| |----------|------|--------|-------|
| Hailo-8 (1x confirmed) | 26 | **Active** | Cecilia only | | Hailo-8 (2x confirmed) | 52 | **Active** | Cecilia + Lucidia |
| Hailo-8 (2x unverified) | 52 | **Unknown** | Purchased but not detected on Octavia/Aria | | Hailo-8 (1x unverified) | 26 | **Unknown** | 3rd module purchased, not detected on Octavia or Aria |
| NVIDIA Jetson Orin Nano | 40 | **Pending** | Dev kit not deployed | | NVIDIA Jetson Orin Nano | 40 | **Pending** | Dev kit not deployed |
| Apple M1 Neural Engine | 15.8 | **Active** | Alexandria Mac | | Apple M1 Neural Engine | 15.8 | **Active** | Alexandria Mac |
| Arm Ethos-U55 | ~1 | **Returned** | SenseCAP Watcher | | Arm Ethos-U55 | ~1 | **Returned** | SenseCAP Watcher |
| **Confirmed Active** | **41.8** | | Hailo-8 (Cecilia) + M1 | | **Confirmed Active** | **67.8** | | 2x Hailo-8 + M1 |
| **Potential (if all working)** | **~135** | | Requires physical verification | | **Potential (if all working)** | **~135** | | +1 Hailo-8 + Jetson |
--- ---
## Missing Hailo-8 Investigation ## Missing Hailo-8 Investigation
3 Hailo-8 M.2 modules were purchased (serial numbers documented: HLLWM2B233704667, HLLWM2B233704606, third unknown). Only 1 is confirmed active on Cecilia. 3 Hailo-8 M.2 modules were purchased (serial numbers documented: HLLWM2B233704667, HLLWM2B233704606, third unknown). 2 are confirmed active: Cecilia and Lucidia. 1 remains unaccounted for.
### Possible Explanations ### Possible Explanations

9
hardware/devices/raspberry-pi.md
View File

@@ -10,9 +10,9 @@
| Node | Board | RAM | Storage | Case | Accelerator | IP (Local) | IP (Tailscale) | Status | | Node | Board | RAM | Storage | Case | Accelerator | IP (Local) | IP (Tailscale) | Status |
|------|-------|-----|---------|------|-------------|------------|----------------|--------| |------|-------|-----|---------|------|-------------|------------|----------------|--------|
| Cecilia | Pi 5 | 8GB | 500GB NVMe (49% used) | Standard | **Hailo-8 26T** | 192.168.4.89 | 100.72.180.98 | **Active** | | Cecilia | Pi 5 | 8GB | 500GB NVMe (15% used) | Standard | **Hailo-8 26T** | 192.168.4.89 | 100.72.180.98 | **Active** |
| Octavia | Pi 5 | 8GB | 29GB SD (60% used) | Pironman | None confirmed | 192.168.4.38 | 100.66.235.47 | **Active (OVERLOADED)** | | Lucidia | Pi 5 | 8GB | 119G SD + **1TB NVMe** (1% used) | Pironman | **Hailo-8 26T** | 192.168.4.81 | 100.83.149.86 | **Active** |
| Lucidia | Pi 5 | 8GB | Unknown | ElectroCookie | Unknown | 192.168.4.81 | 100.83.149.86 | **DOWN** | | Octavia | Pi 5 | 8GB | 238GB SD (34% used) | Pironman | None confirmed | 192.168.4.38 | 100.66.235.47 | **Active** |
| Aria | Pi 5 | 8GB | 29GB SD (74% used) | Pironman | None confirmed | 192.168.4.82 | 100.109.14.17 | **Active** | | Aria | Pi 5 | 8GB | 29GB SD (74% used) | Pironman | None confirmed | 192.168.4.82 | 100.109.14.17 | **Active** |
| Anastasia | Pi 5 | 8GB | Unknown | Pironman | Unknown | 192.168.4.33 | — | **SSH Closed** | | Anastasia | Pi 5 | 8GB | Unknown | Pironman | Unknown | 192.168.4.33 | — | **SSH Closed** |
| Cordelia | Pi 5 | 8GB | Unknown | Standard | Unknown | 192.168.4.27 | — | **SSH Closed** | | Cordelia | Pi 5 | 8GB | Unknown | Standard | Unknown | 192.168.4.27 | — | **SSH Closed** |
@@ -25,6 +25,9 @@
|------|----------------------|---------------| |------|----------------------|---------------|
| Octavia Tailscale IP | 100.83.149.86 | **100.66.235.47** | | Octavia Tailscale IP | 100.83.149.86 | **100.66.235.47** |
| Lucidia Tailscale IP | 100.66.235.47 | **100.83.149.86** | | Lucidia Tailscale IP | 100.66.235.47 | **100.83.149.86** |
| Lucidia status | DOWN | **BACK ONLINE** (has 1TB NVMe + Hailo-8 + NATS) |
| Lucidia /etc/hostname | "lucidia" | **"octavia"** (wrong, needs fixing) |
| Hailo-8 count | 1 (Cecilia only) | **2 confirmed** (Cecilia + Lucidia) |
| Hailo-8 on Octavia | Active (26 TOPS) | **Not detected** (no `/dev/hailo*`, no `hailort.service`) | | Hailo-8 on Octavia | Active (26 TOPS) | **Not detected** (no `/dev/hailo*`, no `hailort.service`) |
| Hailo-8 on Aria | Active (26 TOPS) | **Not detected** (no `/dev/hailo*`, no `hailort.service`) | | Hailo-8 on Aria | Active (26 TOPS) | **Not detected** (no `/dev/hailo*`, no `hailort.service`) |
| SSH user | `alexandria` | **`blackroad`** | | SSH user | `alexandria` | **`blackroad`** |