If software depends on specific infrastructure to work, then it should have it as a hard requirement. Haven’t tried Ceph myself, but this sounds like there’s something very wrong with it conceptually or the way it’s set up.
It’s more that Proxmox has a weird niche laying between both enterprise/small-medium business and also homelab infrastructure.
Ceph is designed towards the former where 10Gbe fibre and hosts with a boatload of RAM is fairly commonplace.
My problems were a result of not paying close enough attention to those requirements and just hoping it would work on small micro PCs with 12GB of ram.
Yeah, but even if that’s the case, I would expect it 1) to refuse to run at all if something’s not fit or 2) to just be bottlenecked by the network instead of making data disappear altogether.
A Proxmox cluster with Ceph pretty much won’t work without 10Gbps.
It’ll slowly fail and all your data will get lost. Ask me how I know.
(Also, you need at least 4GB memory dedicated to Ceph on each node… Another compounding lesson learned)
If software depends on specific infrastructure to work, then it should have it as a hard requirement. Haven’t tried Ceph myself, but this sounds like there’s something very wrong with it conceptually or the way it’s set up.
It’s more that Proxmox has a weird niche laying between both enterprise/small-medium business and also homelab infrastructure.
Ceph is designed towards the former where 10Gbe fibre and hosts with a boatload of RAM is fairly commonplace.
My problems were a result of not paying close enough attention to those requirements and just hoping it would work on small micro PCs with 12GB of ram.
Yeah, but even if that’s the case, I would expect it 1) to refuse to run at all if something’s not fit or 2) to just be bottlenecked by the network instead of making data disappear altogether.
Oh, to be fair the data disappeared because I was stupid for a different reason and proxmox upgrades were done out of order.
I’m basically 100% to blame for the data loss, but it was impacting performance from an IO and memory perspective before I fucked it all up.