Adding _live_ spare to raid1+0. Howto?

Posted by MarchH4re@reddit | linuxadmin | View on Reddit | 25 comments

I've got a set of 4 jumbo HDDs on order. When they arrive, I want to replace the 4x 4TB drives in my Raid 1+0 array.

However, I do not wish to sacrifice the safety I get by putting one in, adding it as a hot spare, failing over from one of the old ones to the spare, and having that 10hr time window where the power could go out and a second drive drop out of the array and fubar my stuff. Times 4.

If my understanding of mdadm -D is correct, the two Set A drives are mirrors of each other, and Set B are mirrors of each other.

Here's my current setup:

Number Major Minor RaidDevice State
7 8 33 0 active sync set-A /dev/sdc1
5 8 49 1 active sync set-B /dev/sdd1
4 8 65 2 active sync set-A /dev/sde1
8 8 81 3 active sync set-B /dev/sdf

Ideally, I'd like to add a live spare to set A first, remove one of the old set A drives, then do the same to set B, repeat until all four new drives are installed.

I've seen a few different things, like breaking the mirrors, etc. These were the AI answers from google, so I don't particularly trust those. If failing over to a hot spare is the only way to do it, then so be it, but I'd prefer to integrate the new one before failing out the old one.

Any help?

[-]

michaelpaoli@reddit

I'd suggest testing it out on some loop devices or the like first, e.g. smaller scaled down version - but put some actual (but unimportant) data on there so you can check it remains throughout, and you don't drop below the redundancy you want. It'd be much simpler if it were just md raid1 - in that case you can add spare(s), and also change the nominal number of drives in the md device, e.g. raid1 would nominally have 2, but if you change it to 3 you've got double redundancy once it's synced, then once that's synced on 3 drives, tell md that the one you want to remove is "failed" and also change the nominal # of drives back to 2. And repeat that as needed until all replaced. And with larger, once they're all larger in the raid1, you can then grow it to the size of the smallest - but not before that.

With RAID-1+0 you might not be able to only add one drive, sync, remove drive, etc. - notably with only one spare available bay, in a manner where you never drop below being fully RAID-1 protected ... but I'm not 100% sure - not fully sure exactly what your setup is, so perhaps it's possible?

So, let's see if I test a bit:

md200
        Array Size : 126976 (124.00 MiB 130.02 MB)
    Number   Major   Minor   RaidDevice State
       0       7        1        0      active sync set-A   /dev/loop1
       1       7        2        1      active sync set-B   /dev/loop2
       2       7        3        2      active sync set-A   /dev/loop3
       3       7        4        3      active sync set-B   /dev/loop4
Each device is 64MiB
The devices I add will be 96MiB - with intent to at least eventually grow
by ~50%.
# dd if=/dev/random of=/dev/md200 bs=1024 count=126976 status=none
# sha512sum /dev/md200
8e9d55346f3379a39082849f6a3c800f7e9b81239080ca181d60fdd76b889976d281b2eb5beaab6e013a4d74882e7bece00bd092ee104a6db1fb750a3dd8441e  /dev/md200
# 
So, if I add drives it's spare, if I then use
--grow --raid-disks 5
I then end up with something very different:
        Array Size : 158720 (155.00 MiB 162.53 MB)
    Number   Major   Minor   RaidDevice State
       0       7        1        0      active sync   /dev/loop1
       1       7        2        1      active sync   /dev/loop2
       2       7        3        2      active sync   /dev/loop3
       3       7        4        3      active sync   /dev/loop4
       4       7        5        4      active sync   /dev/loop5
It claims to be raid10, but I don't know what it's got going on there,
because if it's got Size : 158720 (155.00 MiB 162.53 MB)
that's not all RAID-1 protected on separate drives.
If I add another drive, that becomes spare, if I do
--grow --raid-disks 6
then have:
        Array Size : 190464 (186.00 MiB 195.04 MB)
       0       7        1        0      active sync set-A   /dev/loop1
       1       7        2        1      active sync set-B   /dev/loop2
       2       7        3        2      active sync set-A   /dev/loop3
       3       7        4        3      active sync set-B   /dev/loop4
       4       7        5        4      active sync set-A   /dev/loop5
       5       7        6        5      active sync set-B   /dev/loop6

So, it's not doing additional mirror(s) as it would with raid1, but rather looks like it extends as raid0, then when it get the additional drive after that, mirrors to raid10. So I don't think you can then take out the other drives, as there's only single redundancy, so couldn't just remove the other two older smaller drives.

So, I don't think there's any reasonably simple way to do what you want with md and raid10.

There may, however, be other/additional approaches that could be used. See my earlier comment on dmsetup. Still won't be able to do it all live, but at least most of it. Notably for each drive, take the array down, add drive replace the drive in md with dmsetup device that's low-level dmsetup RAID-1 mirror - of old and new drive. Once that's synced up, take the array down again, pull out the old drive, undo that dmsetup, reconfigure md to use the new drive instead of the dmsetup device, and repeat as needed for each drive to be replaced. Once they're all replaced, you should be able to use --grow --size max to initialize and bring the new space into service.

[-]

MarchH4re@reddit (OP)

I'd suggest testing it out on some loop devices or the like first, e.g. smaller scaled down version...

Like this?

I'm definitely finding your observations to be the case. Growing the array to bring the new disk live doesn't add it as an extra drive to a new set (I found Set B disk to be the mirror of the set A disk, not at all what I would consider "a set".) It reshapes the array. Once I've grown the array to grab both new devices, it won't let me remove the old ones due to the way it sets them up. I may be stuck making a backup, then failing over with the array offlined.

[-]

michaelpaoli@reddit

Or use dmsetup to do low-level device mapper (dm) RAID-1, as I suggested.

Take md array down. Replace drive with RAID-1 dm device, once that's synced, take md array down deconstruct the dm device, pull the old drive, reconfigure md array to use the replaced drive, continue as needed for each drive to be replaced. Then when all done, use --grow and --size max to grow the array out to the new available size.

And yes, sure, can test it on loop devices too. And on the "for real" you'll probably want to do all that dm RAID-1 stuff with the journal files/data or whatever they call it, to track state of the RAID-1 mirroring - most notably so it would be resumable in case, e.g. system were taken down while it was in progress. Otherwise you'd have to make presumptions as to which of the two copies should be presumed clean and used to do complete mirroring to the other.

[-]

michaelpaoli@reddit

P.S.

Oh, actually, there's probably a way to only have to do start/stop cycle of the md device twice through the entire migration.

Most notably with device mapper, should also be able to live add/drop device(s) to its RAID-1 membership. I haven't determined how to do that, but I'd guestimate it's "just" a matter of doing a live update on the table for the dm device. And yes, it's done - lots of stuff (e.g. LVM, etc.) that used dm does it quite commonly, so it's very doable. Yeah, the dm documentation is pretty good, but it could be more complete (but hey, that documentation is in the source and ... also the definitive answers are to be found in the source too).

[-]

MarchH4re@reddit (OP)

This is really awesome info! I appreciate it. You put a lot of work into explaining that clearly.

I'll have to try this stuff with some loopback images, but it's a little beyond my experience with raid at this point. At least beyond what I think I'm willing to try on live disks that have stuff I don't want to accidentally bork (even with a backup). But it's definitely something I'm going to play with on a VM, where anything I break won't matter anyway :)

And failing all else, I can just get my backup up-to-date and then fail over one-at-a-time like I did last time I upgraded it.

[-]

michaelpaoli@reddit

Well, if you do a full read of each of the old drives, right before taking it out of service, and then just have md mark it as failed, remove it, insert new drive, add that, and let it remirror that, the probability of getting unrecoverable read error that soon after is relatively low. But yeah, doing it that way, you wouldn't have the redundancy ... except, well, kind of - if it was still readable on the old drive, could pull it from there ... but if the data is in active rw use that may not be feasible, as the data may have subsequently changed.

But yeah, device mapper (dm) and dmsetup(8), pretty dang impressive in its capabilities. Would be nice if it were better documented, ... but hey, it's open source, so there is that at least, so the answers can be found - just may take some digging. And as I also mentioned in my "P.S." bit, within dm raid1, should also be possible to hot remove/add a device, thus could even further reduce downtime ... but I haven't yet looked into exactly what it takes to do that - but must be very doable (various things that use dm in fact do so to make their operations possible, e.g. LVM uses dm, and can add and drop mirrors on-the-fly).

[-]

MarchH4re@reddit (OP)

If I remount the filesystem on it as readonly, I'm wondering if that would be enough, or does the raid itself have to be taken offline? I'm not sure if the system needs to update stuff outside of me writing something to the drive or not.

The machine I'm doing this on isn't really my daily driver anymore, I'm using it more like a NAS for the most part, but I can live with the data it has being offlined for a few days (probably). If I can readonly the ext4 on the partition while the action is happening, that's even better since I can access without needing to write anything.

[-]

michaelpaoli@reddit

If I remount the filesystem on it as readonly, I'm wondering if that would be enough

Depends exactly what you're wanting to achieve with that. If the goal is to have no changes made on the filesystem, that comes exceedingly close ... however, for most filesystem types, even mounting a filesystem ro will still generally do at least some minimal checking that it's clean before mounting, and if it's not, and the fixes are trivial, it may make those changes before mounting. Also, it will still generally update some filesystem metadata, e.g. directory path where it was last mounted, and typically also a timestamp as to when. So, if you really want exactly zero writes to occur to the filesystem (also relevant for, e.g. forensics trail of evidence and legal), use blockdev --setro to set the device itself as readonly before mounting it - then no changes can be made to it. So, if you did that to the filesystem before mounting it, nothing on the filesystem or that device could change at all, however if that's md device, lower level stuff on the device can change (e.g. a disk could fail, be replaced, be remirrored, etc.). One can also start the md device --readonly (or -o), but that's even more restrictive - no resync, recovery, or reshape (see mdadm(8)).

If I can readonly the ext4 on the partition while the action is happening, that's even better since I can access without needing to write anything.

Well, probably blockdev --setro at the level of the block device for the filesystem (e.g. the md device), then mount it readonly (or may be able to remount it from rw to ro, then do the blockdev --setro), that way nothing logically on the filesytem changes (and the md device), but stuff below that layer can be changed (e.g. add/drop/replace drives, remirror them, etc. - possibly via dm to create such to first replace all the drives as single drive raid1 in degraded state, then add 2nd drive, and once synced, remove the old, and do that drive-by-drive until they're all done. But would need to know how to do that with the device mapper (I'm curious to try it out - I have a guess how it's done, but I need check some documentation and actually try it). And of course once all completed for all the drives, update the mdadm.conf file, and remove blockdev --setrw on the device itself (presuming it was earlier set ro), and remount it nominally as rw. Would probably need to do one stop and start of the array to switch it from the dm devices to the partitions on the drives, also may want to do a reboot in there to see how the drives are nominally assigned to their sd[a-...] or whatever devices, and make sure mdadm.conf is set to match (or at least correspondingly compatible), and likewise make sure initrd/initramfs is also suitably updated (notably to pick up any mdadm.conf updates). I'd think that'd well do it.

And yes, should always have backups ... but can be dang handy and faster / more efficient when one can avoid unneeded large restore operations - can also generally have much less downtime that way too.

[-]

michaelpaoli@reddit

So, let's see, 4 device mdraid10, with no losses in redundancy, and replacing each device one at a time, notably by also using RAID-1 between pairs of devices (old new replacement) via device mapper (dmsetup(8), etc.):

// md raid10 array:
# mdadm --detail /dev/md200 | sed -ne 's/^ *//;/Le/p;/y S/p;/Nu/{:l;n;s/^ *//p;bl}'
Raid Level : raid10
Array Size : 61440 (60.00 MiB 62.91 MB)
0       7        1        0      active sync set-A   /dev/loop1
1       7        2        1      active sync set-B   /dev/loop2
2       7        3        2      active sync set-A   /dev/loop3
3       7        4        3      active sync set-B   /dev/loop4
# dd if=/dev/random of=/dev/md200 bs=1024 count=61440 status=none && sha512sum /dev/md200
569d3672090b4508e23712ccd2969baa1da7c236c71c1d7d2c3d904794fb66fcc3cdceef88b7dd9b26ec661930980764837afceca36463ab1d8495c740111aec  /dev/md200
# (cd /sys/block && grep . loop[1-8]/size)
loop1/size:65536
loop2/size:65536
loop3/size:65536
loop4/size:65536
loop5/size:131072
loop6/size:131072
loop7/size:131072
loop8/size:131072
# 
// 32MiB for the old devices loop[1-4],
// 64MiB for the new devices loop[5-8]
# mdadm --stop /dev/md200
(
  o=1
  while [ "$o" -le 4 ]; do
    n=$((o+4))
    dmsetup create r1 --table "0 65536 raid raid1 5 0 region_size 8 rebuild 1 2
- /dev/loop$o - /dev/loop$n"
    devs=
    for d in 1 2 3 4; do
      if [ "$d" -gt "$o" ]; then
        devs="${devs:+$devs }/dev/loop$o"
      elif [ "$d" -eq "$o" ]; then
        devs="${devs:+$devs }/dev/mapper/r1"
      else
        devs="${devs:+$devs }/dev/loop$n"
      fi
    done
    mdadm -A /dev/md200 $devs
    while sleep 1; do
      set -- $(dmsetup status r1)
      case "$5" in AA) break;; esac
    done
    mdadm --stop /dev/md200
    dmsetup remove r1
    o=$((o+1))
  done
)
# mdadm -A /dev/md200 /dev/loop[5-8
# mdadm --grow /dev/md200 --size max
mdadm: component size of /dev/md200 has been set to 63488K
# mdadm --detail /dev/md200 | sed -ne 's/^ *//;/Le/p;/y S/p;/Nu/{:l;n;s/^ *//p;bl}'
Raid Level : raid10
Array Size : 126976 (124.00 MiB 130.02 MB)
0       7        5        0      active sync set-A   /dev/loop5
1       7        6        1      active sync set-B   /dev/loop6
2       7        7        2      active sync set-A   /dev/loop7
3       7        8        3      active sync set-B   /dev/loop8
# set -- $(dd if=/dev/md200 bs=1024 count=61440 status=none | sha512sum); [ "$1" = 569d3672090b4508e23712ccd2969baa1da7c236c71c1d7d2c3d904794fb66fcc3cdceef88b7dd9b26ec661930980764837afceca36463ab1d8495c740111aec ] && echo MATCHED; set --
MATCHED
#

So, do have to stop the md array when replacing devices (old --> dm RAID-1 --> new), but other than that, md device can be running the entire time, and never have less than the md raid10 full redundancy at any given point in time (in fact have additional (being) remirrored old --> new) drive at each step along the way.

For actual data I'd strongly recommend also adding the metadevices in the dm RAID-1, that way if, e.g. there's crash while that drive pair is syncing, it's fully recoverable (otherwise one has to presume the old drive is clean and current, but if any pending writes didn't make it to that drive that may not be 100% the case). See also the kernel dmsetup documentation (it might not specify how to size the metadevices, but one can probably figure that out from the sources or some testing).

[-]

deeseearr@reddit

It sounds like you have two RAID 1 arrays and you want to add and then remove drives from each of them. You can just expand each of your two disk RAID 1 arrays to three. That way you will temporarily have a third mirror for each set and can keep using them without worrying about a single disk failure ruining your day.

# mdadm /dev/md*XX* --add /dev/sd*Y*

# mdadm /dev/md*XX* --fail /dev/sd*Z*

# mdadm /dev/md*XX* --remove /dev/sd*Z*

If you feel bold you can chain those all into a single command, but if you would prefer to leave a few hours of time between adding the new drive and failing the old one, that's entirely fine.

If you're not entirely clear on what mdadm -D is telling you /proc/mdstat should give you a more readable version of the same thing.

If you can afford some down time, a much simpler way would be to shut down each of the arrays (umount them and then "mdadm -S" to stop them), replace one drive from each mirrored set and then just rebuild both mirrors. Even if you have a drive failure or meteor strike and lose everything the drives you pulled out will still be good.

[-]

MarchH4re@reddit (OP)

Does this add the new drive as a live active on one of the mirrored sets? I kind of got the impression it would do something like mess up the mirroring or striping or something. I assume I can "--grow" the disks once I've given the new ones time to make sure they don't fail (and as I get closer to full capacity), retaining the old drives as backup cold spares.

If it takes one drive offline while the new one builds, that's what I want to avoid. We get too many power bumps.

[-]

deeseearr@reddit

That should add a new drive as a third mirror and immediately start rebuilding it. If you were to add it to a RAID0 set instead of RAID1 then it could mess with striping, but that doesn't sound like what you want. I strongly recommend that you start by taking a backup of everything and then verify that every command does what you think it should. (Don't trust random people on Reddit if you can avoid it. I could be a complete idiot, I could be working from an outdated set of documentation, and I could also have completely misunderstood how your devices are set up.)

If you are concerned about power bumps and don't have a reliable UPS then I would also recommend taking the entire array offline while rebuilding it. The rebuild process should be able to resume after a power failure but it's best not to tempt fate.

If the new drives are larger than the originals then yes, you will want to use mdadm --grow once you're done. That should raise the size of each md device to the maximum available, and then you can grow the filesystems on top of them so that all that space becomes usable.

[-]

MarchH4re@reddit (OP)

No, I definitely don't want to mess with the striping X) It sounds like you're describing what I want to do. Taking the raid offline seems like a good idea though.

Looks like the best way to do that is to reboot into single user mode, make sure home is unmounted, mdadm --stop /dev/md0, and -add the new drive? The machine isn't my daily driver anymore, so I can stand to have it offline for awhile while it rebuils, I use it more like a NAS these days. A really overpowered NAS.

I haven't gotten the HDD set yet, so I'm still looking at doing this right. I may do a few experiments with small disk images on the machine I DO use as a daily driver, just to play with it.

[-]

deeseearr@reddit

If you can stop any processes accessing the filesystem and then unmount it, then you should be able to mdadm --stop without having to reboot, but otherwise that sounds right.

[-]

MarchH4re@reddit (OP)

could also have completely misunderstood how your devices are set up.

It's a bog standard raid10. Stripe of mirrors. That was part of the reason I was asking. I wanted to know if there were any gotchas if/when adding a live spare to one of these. I know with a raid10 you generally need an even number of drives, but not if you can have an extra mirror in one set and just the two in the other, for instance.

[-]

deeseearr@reddit

It's a bog standard raid10

Ah. Standard RAID levels only go up to six, and Linux MD uses its own non-standard implementation of a whole bunch of different nested RAID levels and calls it "raid10". It sounds like that's the standard that you're using.

What I was describing was doing operations on one of the two RAID1 devices which could be used to make up a RAID 1+0 device. For example, if /dev/md1 and /dev/md2 are each RAID1 devices then /dev/md3 could be a RAID0 device with md1 and md2 as members. If you're looking at a single device with four raw disks as members then you probably have gone straight to md's "--level=10" RAID10. Left to its own devices it will look like a RAID 1+0 device, with all the stripes and mirrors in similar places, but it can also support odd numbers of drives and a variety of mirror/stripe layouts that don't look anything at all like RAID0 or RAID1.

With sufficient additional disk space you may be able to reshape your array into something larger with five disks and three replicas of the data but I'm afraid you're on your own with that.

[-]

MarchH4re@reddit (OP)

A quick test of your procedure on disk images seems to indicate that using --add only adds the 5th disk as a spare. If I grow the array to 5 devices, it looks like it maybe completely fubars the sets? (I didn't try it with any test data, yet).

I want something like this after I've done added the first disk:

7 8 33 0 active sync set-A /dev/sdc1
5 8 49 1 active sync set-B /dev/sdd1
4 8 65 2 active sync set-A /dev/sde1
8 8 81 3 active sync set-B /dev/sdf1
9 8 92 4 active sync set-A /dev/sdg1 <- New device!

I tried this with some loopbacks pointed at blank image files, and I wound up with this when I grew it:

0       7        0        0      active sync   /dev/loop0
1       7        1        1      active sync   /dev/loop1
2       7        2        2      active sync   /dev/loop2
3       7        3        3      active sync   /dev/loop3
5       7        5        4      active sync   /dev/loop5
4       7        4        -      spare   /dev/loop4

Not sure if this is just a display thing, or if it screws up the layout. The fifth drive DOES become active, just doesn't look like there are sets anymore?

[-]

archontwo@reddit

Yeah. At this stage you ought to be considering BTRFS or ZFS.

Would make life so much easier.

[-]

MarchH4re@reddit (OP)

I grew up using ext2. Guess I'm an old fogey stuck in my ways. I see a lot of people extolling how great BTRFS and ZFS are, but I'm not sure quite how advanced my needs are that these would improve things for me. Maybe they would? I dunno, once we got ext4, the constant fscks on hard power losses stopped being an issue anymore.

Still, I'm game. Wanna sell me on these?

[-]

archontwo@reddit

Well volume management and expansion or reduction are built in. This greatly simplifies adding and removing storage as needs arise.

BTRFS will allow more exotic setup by virtue of metadata and data being logically seperate. This allows thing like having data in a RAID 5 or 6 configuration while your metadata can be RAID 1, 10 or 1c3

ZFS has a robust caching and metadata system, which allows features like dataset encryption and compression or deduplication. It will use more resources to do this sort of thing online but if you plan ahead it is quite sensible to put priority on storage rather than compute.

Honestly, I have been using Linux for ahem years and was using MD raid for ages. Which is why I started getting tired or rebuilding arrays and planning weeks in advance for a migration.

With modern filesystems, that all goes away and with ZFS and BTRFS being able to export their data over the network to another storage box, well you can see how I doubt I will ever go back to simple MD raid or even lvm except for very specific purposes.

Good luck.

[-]

FireWyvern_@reddit

What's the size of new HDD?

[-]

MarchH4re@reddit (OP)

Lol, each is 22TB. Yet another reason I wanna be cautious. Originals are 4TB each. I'll grow into them after I've given them a few months to break in. I'm at about 60% capacity right now, I'm just trying to beat Tariff Mussolini.

[-]

FireWyvern_@reddit

So i take it you use software raid?

[-]

evild4ve@reddit

assuming 3-2-1 backup has been done

old RAID array > offline backup

offline backup > new RAID array

but if there is not yet offline and offsite backup, use the new HDDs for that instead

[-]

MarchH4re@reddit (OP)

I'm waiting on the 5TB hdd for that 3-2-1 backup. Like most terrible sysadmins, I have a bad habit of slacking on the backups.

I agree that running the intermediary backup is probably safest, but in my case, I still want to do the one-at-a-time upgrade anyway, just to catch any potentially failing new drives.