Six plug-in drives, one crash, one green light that lied — and the 5 checks that showed the damage it was hiding
Six plug-in drives, one crash, one green light that lied — and the 5 checks that showed the damage it was hidingSix small USB drives were doing the job of one big disk. That bundle is called a pool, and mine is bcbc-storage.
One drive dropped out while I was working. I plugged it back, pressed clear, and the system answered ONLINE — its greenest possible word.
Its own report still listed blocks it could never repair.
ONLINE is a heartbeat, not a bill of health.
The whole day in one picture
6 USB drives ──► ONE storage "bcbc-storage"
│
❌ one drive leaves ──┤ → FROZEN. It stops writing instead of writing into a hole.
│
✅ same drive back ───┤ → new name, alarm cleared → 🟢 ONLINE
│
❌ bad blocks still ──┘ → listed as permanent. The green light never mentions them.
What the green light actually covered
- all six drives answering
- storage no longer frozen
- state reads
ONLINE - the damaged blocks repaired — no, still listed
- my files copied and checked — not verified
- why the drive left — unknown
Three green ticks. Three empty boxes. That is the whole post.
The 5 checks I now run on any drive that scares me
Five read-only commands on ZFS — the system guarding this storage. It keeps a checksum of every block, which is exactly how it could still say damaged while the light was green.
lsblk -o NAME,SIZE,MODEL,SERIAL,TRAN,FSTYPE,MOUNTPOINTS # 1 · which disks are really there
lsusb -t # 2 · how they are wired, slow hub included
zpool status -v bcbc-storage # 3 · state, error counters, bad blocks
zpool status -LP bcbc-storage # 4 · the view that survives a reboot
journalctl -k -b --no-pager # 5 · what the system's own diary said
None of them changes anything. Keep the output with a timestamp, and blur serial numbers and private paths before posting it anywhere.
Your turn
If you have chased the same thing — a USB drive bundle that drops out and comes back looking fine — what told you it was the cable, the hub, or the drive itself?
My takeaway: a green light is permission to keep looking. It is never permission to stop.
🔍 The three lights, the wire smell, and the drive that came back with a new name
I spent a day on this. Six SSDs at about 238 GB each, and one of them simply left.
The pool went suspended — frozen, not writing — and one member was missing. It came back as /dev/sdn. I did not trust the letter: the serial number ending 1274 is what told me it was the same drive.
A disk returning as a different
/dev/sdXis not a different disk. Letters are seat numbers the system hands out at every boot. On USB, it reshuffles the whole bus.
The wire had a smell too: one hub at 480M, the other at 5000M.
| Link | What it is | What I got |
|---|---|---|
| 480 Mbps | a skinny pipe | ~40 MB/s |
| 5000 Mbps | the fat pipe I paid for | the fast lane |
Worth chasing. Not proof that the slow link caused the dropout, and not proof a drive is fake or dying.
zpool clear clears device errors, and it can bring a frozen pool back once its devices answer — that is what the manual says, and it says nothing about the files inside being healthy.
Three lights, and almost everyone only reads the first:
| Light | It tells you | It hides |
|---|---|---|
| Pool state | the drives answer, the pool is not frozen | whether the blocks on them are intact |
| Error counters | how many failures each drive logged | that the failures are over |
| Permanent errors | blocks known bad and not repairable | that they left when the light went green |
Clearing the alarm is not the same as putting out the fire.
💾 The copy, the ghost that knocked every six seconds, and what I still owe
The copy first. Personal files off the pool, proposed as:
/srv/BCBC-STORAGE/WJF-Personal-File-Store
→ /srv/BCBC-FAST/personal-rescue-<timestamp>/
Proposed with rsync -aHAXS and a copy.log. I have no verified completion and no read-back for that copy, so the rescue is not done — and the flags are worth knowing: -H hard links, -A permissions, -X extra file attributes, -S sparse files.
A folder existing is not a successful copy. A successful copy is not a verified backup.
The ghost. On 23 October I took the filesystem offline on purpose to test link speeds — a different event from the dropout, and I will not blur the two. The pool was no longer in the imported list; only the fast drive and the two internal disks were. So: not imported, not destroyed, not repaired.
Meanwhile the machine kept dialling a network share that no longer existed — an automount firing about every six seconds in the system’s own diary. Over roughly an hour the machine got slower until it was barely usable. Num Lock still toggled, just slowly. Getting out meant holding the power button.
Taking storage offline needs a client-side plan too. Mounts keep reaching for what is no longer there.
Not proven: which process kept triggering it, whether the file server was reachable at all, or whether those failed mounts caused the slowdown. A plausible contributor, not a demonstrated cause. Disabling that mount for good and watching is still pending.
What I still can’t prove — and the order I’ll fix it in.
The data — the copy’s completion and a read-back check.
The pool — a fresh status and the real device layout.
The hardware — the disconnect and reset lines, why one hub sat at 480M, and whether it was power, a cable, a hub, a bridge, one SSD, or a mix.
The drives — a clean full scan, genuine capacity, a sustained transfer rate.
Protect the data first. Then test the link again under controlled conditions. Then: 1 establish what the pool really is → 2 protect and verify the data → 3 pick the replacement layout → 4 migrate → 5 verify before retiring the old drives.
Replacement hardware is not chosen yet; the spare 1TB NVMe being checked first has not even been detected.
Scope. This USB pool is separate from the NVMe drive used for models — nothing from that work belongs in this report, and I am not mixing the two results.
Two gaps I’m still recovering. The exact offline command and its full sequence from the test transcript; and what kept triggering that mount.
!