Soleil Levant
DECOMISSIONED DUE TO RAID ARRAY FAILURE
Soleil Levant is a server hosted in France for Project Segfault.
It is currently the most powerful server, running a large majority of our services.
As of now, it runs Proxmox VE as its hypervisor.
Decomissioning
In mid-August, a disk on the server failed. This however was fixed after a 24 hour downtime (for getting the disk through someone he knew) and a backup restore for just our matrix instance which experienced corruption.
After this, soleil ran relatively fine for a while, till 12/09/2023, when MrLeRien noticed one of the drives were clicking again, and after investigation realized it went into a degraded state.
However, the spare drives we bought after the first disk failure didn't work since the LSI Hardware RAID was serialized to work with only hitachi drives (the drives we got had same part number and such but was from HP).
Later, when we got another hitachi drive from the same person, RAID failed to re build since 2 more drives failed. At this point, we thought we would just delete everything and restore from backup on a new ZFS, but then the LSI HBA we had didn't support IT firmware (so we basically cant disable raid and use the drives separately). This was the final nail in the coffin, and soleil levant was officially shutdown on 16/09/2023 due to these issues.
We however, didn't have any good server in the EU, so we had to move everything to our India server.
Along with that, we had to discontinue matrix which was too big.
change of plans, i'll just fully retire Soleil Levant. it fucks me up mentally and financially and i don't feel like spending another sleepless night trying to get the server up when the most likely situation is that the entire RAID array is fucked and i'd need to replace every single disk + get an HBA for ZFS since i cant flash the raid card.
i know it's not the outcome you guys wanted but i need to focus on my health, sorry again
we'll make a blog post when everybody will be online and everything we want to back up is backed up but yeah.
Message from MrLeRien on our status page on 16/09/2023, when the server was discontinued
Hardware
The server is a Hitachi Compute Rack 210H, with dual socket Intel Xeon E5-2690s.
It has 320 GiB of memory, and a raid 5 array with 4 TB of usable space.
VMs
As with using Proxmox, the virtualization backend is QEMU/KVM.
The server is almost entirely virtualized, avoiding running anything on bare metal.
You can see the different VMs outlined in restructure.