Restoring a raid5 with two failed drives

Just so I wouldn't have to search for this again. It's more or less described in The Software-RAID HOWTO but when shit happens, step by step instruction are what I want.

First off, it would only work if the drives were marked failed manually or because of a recoverable error. Glitches happen, once I saw four drives fail within two seconds for no reason we could identify. After the server was switched off and powered up again, they all came back to life and worked flawlessly for the next two years. So, once again, this is for the case when the drives are marked as failed but are perfectly all right otherwise. If they emit smoke or funny clicky sounds, just kiss your array goodbye and recreate it from scratch. You did do backups, didn't you? Huh?

All right, here we go.

1. Panic. I mean it. It'll take a lot of time to get your array working anyway, so a few minutes won't really matter. What matters is that you do the following steps right. So if you feel like getting nervous, do it now, as vigorously as possible. When through with that,

2. Investigate. You have to know exactly in what order the drives in question were marked failed.

3. Edit /etc/raidtab. Change the status of the drive that was the first to fail to failed-disk.

4. Once again check if you have understood and done everything right so far. Next step is the point of no return.

5. mkraid —dangerous-no-resync —force $YourArray

6. raidstart $YourArray # if mkraid hasn't done this for you yet.

7. raidhotadd $FirstFailedDrive $YourArray

8. Wait for the array to rebuild. I repeat, wait for the array to rebuild. watch cat /proc/mdstat and meditate. Or get a shower. Or both.

With luck, after that you'll have a working array once again.