21 may 2013: This issue has been fixed

Too long; didn’t read: Lost 3 hard drives today, but are working on it. Most search engines may be down to Tuesday the 21.

It take some resources to run all the 9 search engines for this site. Each search servers run as a separate virtual machines, hosted on one of two IBM X3550 (4 Core Xeon cpu, 12 gb ram ) servers. Each hosts has 2 hard disks configured in raid 1 (both hard disks has an exact copy of all data).

Yesterday one of the hard disks failed. That is not a problem since there is a copy of the data on the other hard disk, and I had two spare disk in case something like this should happen. The hard disk is hot swappable so I replaced it with a new one and the raid started to rebuild automatically.

Then the new one failed 🙁 It got replaced again, but apparently the stress of rebuilding the raid cause both the new and the old hard disk with the data to fail!

Now I can’t rebuild the raid, and can’t copy back from backup because there is no spare disks left to put into the server.

As this wasn’t enough tomorrow and Monday are national holiday, so it is proving hard to get spare parts.

Everything is backed up, but need to get the server back up before I can restore anything. As it looks now, this will not be fixed before Tuesday the 21.