Growing a RAID5 array – MDADM

With the release of kernel 2.6.17, there’s new functionality to add a device (partition) to a RAID 5 array and make this new device part of the actual array rather than a spare.

I cam across this post a while ago on the LK list.

LKML: Sander: RAID5 grow success (was: Re: 2.6.16-rc6-mm2)

So I gave it a go. My HOME directory is mounted on a 3x70gb SCSI RAID5 array. so I tried adding a further drive.

Although with the release of mdadm > 2.4 the only real critical part of the process is safer (it backs up some live data that is being copied), I didn;t fancy risking growing a mounted array. So I did plenty of backups, then switched to single user run level.

Basically the step includes adding a disc to the array as a spare, then growing the array onto this device.

mdadm --add /dev/md1 /dev/sdf1
mdadm --grow /dev/md1 --raid-devices=4

This then took about 3 hours to reshape the array.

The filesystem the needs to be expanded to fill up the new space.

fsck.ext3 /dev/md1
resize2fs /dev/md1

I then remounted the drive and wahey. Lots extra space….! Cool or what

EDIT  It’s over 18 months since I wrote this post, and Linux kernel RAID and mdadm have continued to move on. All the info here is still current, but as extra info check out the Linux RAID Wiki.

EDIT 2  The Linux Raid Wiki has moved

This entry was posted in Computergeekery and tagged , . Bookmark the permalink.

127 Responses to Growing a RAID5 array – MDADM

  1. Frode says:

    Excellent stuff. It inspired me to grow my 3x400GB by one drive. Here’s the result:


    [root@coruscant ~]# mdadm --add /dev/md1 /dev/sde1
    [root@coruscant ~]# mdadm --grow /dev/md1 --raid-devices=4
    [root@coruscant ~]# cat /proc/mdstat
    [>....................] reshape = 0.0% (166656/390708736) finish=1347.2min speed=4830K/sec
    :
    [root@coruscant ~]# cat /proc/mdstat
    Personalities : [raid1] [raid5] [raid4]
    md2 : active raid5 sde1[3] sdd1[1] sdf1[2] sdc1[0]
    1172126208 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
    :
    [root@coruscant ~]# pvdisplay
    :
    PV Size 745.21 GB / not usable 0
    :
    [root@coruscant ~]# pvresize /dev/md2
    Physical volume "/dev/md2" changed
    1 physical volume(s) resized / 0 physical volume(s) not resized
    [root@coruscant ~]# pvdisplay
    :
    PV Size 1.09 TB / not usable 0
    :
    [root@coruscant ~]# df -BMB /dev/mapper/VG1-LV0
    Filesystem 1MB-blocks Used Available Use% Mounted on
    /dev/mapper/VG1-LV0 105690MB 7894MB 92428MB 8% /mnt/backup
    [root@coruscant ~]# lvresize -L +10G /dev/VG1/LV0
    Extending logical volume LV0 to 110.00 GB
    Logical volume LV0 successfully resized
    [root@coruscant ~]# ext2online /dev/VG1/LV0
    ext2online v1.1.18 - 2001/03/18 for EXT2FS 0.5b
    [root@coruscant ~]# df -BMB /dev/mapper/VG1-LV0
    Filesystem 1MB-blocks Used Available Use% Mounted on
    /dev/mapper/VG1-LV0 116259MB 7894MB 102460MB 8% /mnt/backup

    Reshaping the raid took almost 24 hours! But in the end everything just worked – online.

  2. Nicholas says:

    I added another 300GB drive to my 4x300GB RAID5. It worked great!

    Keep in mind you have to enable the experimental kernel option before mdadm will let you grow your RAID5.

    The reason it took almost 24 hours for Frode is probably because of the minimum reconstruction speed configured. Mine was looking like it would take 30+ until I bumped it up by doing “echo 25000 > /proc/sys/dev/raid/speed_limit_min”. After that it ran for about 9 hours.

    Then I used xfs_growfs, worked like a charm!

    I’m really happy Linux is getting mature with great options like this.

  3. Nicholas says:

    Oh, I should mention that I also had to do “echo 1024 > /sys/block/md0/md/stripe_cache_size” before mdadm would let me grow the array. The mdadm maintainer said a future version would do this automatically. If you’re having an error trying to grow it, check dmegs, if it says the stripe cache is too low you need to change the number (mine said it needed at least 1024) and run that command. You should also replace md0 with your device name.

  4. Ferg says:

    > I’m really happy Linux is getting mature with great options like this.

    I’m pretty pleased too. The Linux kernel is pretty dammed good at the moment. For another project I’m setting myself a MythTV HTPC box. All the hardware had decent drivers in the kernel.. Well apart from a crappy Linksys WMP54GS card I should never have bought!!

  5. Jack chen says:

    Dears:
    I couldn’t use pvresize while the RAID is reshaping. Is there any method to enhance this limitation ?

    Thanks
    Jack

  6. Pingback: bfish.xaedalus.net » Software RAID 5 in Ubuntu with mdadm

  7. Nicholas says:

    Well, I copied all my DVDs to the RAID and now it’s full again. So I’m adding another disk, up to 6 now. I’m getting 14MB/s reconstruction and it’s telling me it will take around 6 hours.

    Jack, I’m pretty sure the new space doesn’t become available until the reshape is finished. It would be theoretically possible to provide it, but accessing it would be very slow, and it’s extra complexity the RAID system doesn’t need. Use the method I mention above to speed it up, and just wait until it’s finished.

  8. Muganor says:

    Just in case there is Ubuntu readers, they need to know that RAID5 array reshaping is disabled in the stock kernels, giving an “invalid argument” error when trying to grow the array. So they need to compile a custom kernel and enable the option CONFIG_MD_RAID5_RESHAPE=y in the .config file. Note that kernel 2.6.17 or above is required.

  9. Ferg says:

    hi Muganor,

    thanks for commenting. Is that just a Ubuntu thing, or a Debian thing in general?

  10. krampo says:

    Hello, thanks for the post, I’m realy excited and going to try to add 5th 250GB disk to RAID5.

    So theoretically everything can be done without losing data on the raid, yes? I was thinking, can I do the reshape working in normal mode, just umount’ing /dev/md0 ?

    Can I add 2 devices at a time?
    mdadm –add /dev/md0 /dev/hdc
    mdadm –add /dev/md0 /dev/hdd
    mdadm –grow –raid-devices=6 (assuming, that before there were 4 of them)
    or should I add one device at a time?

    Are there any requirements for the resize2fs version? I have 1.38 on my system. Looks that there’s newer version available.

    Thanks!

  11. Ferg says:

    I just unmounted the RAID device. Apparently this can be done with the device still mounted. But it sounds a bit risky to me!!

    Personally I would add a single device at a time. But the functionality has been improving, and it might be fine to add two at a time. You might want to ask on the linux-Raid list. Or do more searching around.

  12. Eric says:

    I’m in the process of adding a 5th drive to my 4 drive RAID5 array. My root partition is on the array, so everything’s still mounted. I haven’t seen any problems yet. Disk access is a lot slower than normal, but that makes sense because it’s rewriting the whole drive. It’s only 10% done, so we’ll see if it gets through the whole reshape without problems.

  13. Eric says:

    It just finished adding the 5th drive. Everything still seems to be running okay.

  14. Pingback: attic » Blog Archive » RAID5 grow

  15. krampo says:

    Does it makes any difference to do fsck BEFORE growing to check whether everything is ok with fs?

  16. krampo says:

    Does it make any difference to do fsck BEFORE growing to check whether everything is ok with fs?

  17. Ferg says:

    Not sure. But it cannot hurt to ensure that the filesystem is good.

  18. krampo says:

    Just started to reshape 🙂

    [>………………..] reshape = 1.5% (3676288/244195840) finish=459.6min speed=8719K/sec

    Will let you know tomorrow how it ends. I’m going to add 2 disks to existing 4 drive raid5, but I’m little superstitious and will do it one at a time. Thanks a lot for the post!

  19. Ferg says:

    Good luck!!!!

  20. krampo says:

    First grow went like a charm, now second is in progress, looks ok.

  21. krampo says:

    The second one didn’t go as smooth as first one, but everything turned out ok. After reshape had finished, I did fsck /dev/md0 – and it said that file system is clean, so I executed resize2fs -p /dev/md0 which said that filesystem actually isn’t clean and I should rund fsck -f /dev/md0. I ran with force flag and then ran the resize2 and that’s it – everything looks good to me. So I growed RAID5 from 4 to 6 disks in 2 steps (took about 8-9hours for each 250G 7200 disk to reshape, ~30min fsck, ~30min resize2s). NICE!

  22. Tommi says:

    Is it possible to speed up the build process. I have just created RAID5 4x250G.

  23. Ferg says:

    You can change the value that is in the proc filesystem for that raid device:

    e.g.

    echo 400000 >/proc/sys/dev/raid/speed_limit_max

    That will increase the maximum speed at which it will sync. Obviously if you are already running at the maximum, it will not help. Good luck.

  24. Tommi says:

    What kind on speed should I get with dual P3 733. At the time it is going 890K/Sec

  25. Pie Pants says:

    Thanks heaps for this guide, you’re a lifesaver! Got my array reshaping right now, adding a 5th disk.

    md1 : active raid5 hda1[4] sdb1[3] hdg1[2] hde1[1] hdc1[0]
    234420096 blocks super 0.91 level 5, 64k chunk, algorithm 2 [5/5] [UUUUU]
    [>………………..] reshape = 3.6% (2837120/78140032) finish=161.9min speed=7748K/sec
    bitmap: 0/150 pages [0KB], 256KB chunk

  26. Pingback: itspudding.com » Linux as a fileserver….

  27. Richard says:

    Would it be a similar step for adding drives to a linear RAID you think?

  28. Ferg says:

    I Don’t know I’m afraid. Probably not. I saw your posts and replies on the LinuxRaid mailing list. A better source of knowledge than me!!!

    Cheers
    Ferg

  29. SpeedyGonsales says:

    To 3 x 500GB I added one disk (second is waiting in line :-), reshaping took 9h 47min
    Speed was between 12000 and 14000KB/s
    It all depends on size of disks and whole array,
    as bigger the size, as longer it take.
    Thanks for the info how to do it, man pages on mdadm are…. manpages 🙂

  30. Pingback: Imagination « News from Cfleesia

  31. Edward Murphy says:

    looks like Ubuntu Feisty’s server kernel has the correct options to start re-shaping without re-compiling the kernel.

    root@tank:~# cat /boot/config-2.6.20-15-server |grep CONFIG_MD_RAID5_RESHAPE
    CONFIG_MD_RAID5_RESHAPE=y
    root@tank:~#

  32. Donald Teed says:

    Thanks for the useful resource/howto.

    I have grown my existing raid 5 array at home from 3 to 4 disks, having several md partitions on each disk for /var, /usr, etc. I’m running Debian 4 with an array that was created several years ago.

    In my experience, running grow on a live system can lead to the file system being partially unavailable. For /home I could stop services using it, umount, fsck, resize2fs and move along. Doing this operation on /usr and /var partitions required going to init 1. Doing it on / caused me to have a system that had no access to commands init, reboot, halt, nor response to Ctrl+Alt+delete. I was at init 1 anyway so I used the power button and it rebooted fine.

    For production system, I would not use mdadm –grow unless it was a partition not a part of the OS and I could turn off the services on it while these steps are taken. I’m sure in years to come this will improve.

    The Software Raid Howto page at the Linux Documentation Project is now very out of date so I expect this page will have more visitors.

  33. Ferg says:

    Hi Donald,

    thanks for the comments. I only tried this on an umounted partition. Which was fairly easy due to it being my HOME partition.

    There’s a few other useful RAID resources. None more useful than the linux-Raid mailing list.

    http://vger.kernel.org/vger-lists.html#linux-raid

    It’s fairly low traffic. I would guess around about 40 posts a week (that is a very, very rough guess). There may be a digest version.

  34. Tommi says:

    Hi,

    Something wrong with my debian based Storage server. I have compiles new kernel with raid5 add disk “setting”

    Drive failed after my kernel upgrade. kernel version 2.6.21.1 Debian 4,0 Etch.

    Still I get this error:
    i# mdadm –add /dev/md0 /dev/hdk1
    mdadm: add new device failed for /dev/hdk1 as 3: Invalid argument

    I have 3 disk array witch off one drive has failed.

  35. Ferg says:

    I can only guess that either your MD device or the SCSI device is wrong. If not try removing the failed device and re-adding the new device to the array.

    If upgrading the kernel has caused you grief, I would also suggest you go back to the last workign kernel, until all is well again.

    Good luck!

  36. Tommi says:

    I added the failed device back to array, worked just fine. All HD’s are connected to IDE controllers, Promise chipset.

  37. Tommi says:

    Could someone check these SMART outputs. I used webmin and smartmontool.

    Drive I can’t add to array
    http://pastebin.ca/507188

    Error:
    sudo mdadm –manage /dev/md0 –add /dev/hdk1
    mdadm: add new device failed for /dev/hdk1 as 3: Invalid argument

    Drive that drops from array at boot. http://pastebin.ca/507185

  38. Bubba says:

    Heya, thanks a bunch for this howto. I’m running a 6x 500g array (coming from 3×500 in one go!). The only thing I did differently was that I put LVM2 in between the array and the filesystem. This way I can steal some extra space from a (hardware) raid 1 set that’s also in the system (has root, swap, etc. on it but still 250G free space… would’t want to waste them).

    Very neat that raid5 resizing now works with sw raids as well! – saved me a bundle on an hw controller there.

  39. Ferg says:

    No problem. Glad it was of some use! Have a look at the Linux-Raid mailing list to see just how well MDADM and Linux kernel software raid is coming on!

  40. Mike says:

    I’d been waiting for this functionality for years, and was very psyched when 2.6.17 came out. I just had occasion to make use of it for the first time, went from 4x320GB to 5. Left everything mounted and network services running, all worked okay even with continuous accesses to the array. RIP raidreconf.

  41. Ferg says:

    MDADM truly rocks for anything Linux software related. Neil Brown deserves a medal for writing this.

  42. Chris says:

    I also have this strange error from mdadm … add new device failed. I copied the partition table with dd. and added the hdc2 to md2 and now I can’t add hdc3 to md3 ?!?!?! Very strange.. Anyone an idea ?

  43. Jeremy Leigh says:

    Guys, thought this might help.
    You need to create a blank partition first, and then set the ‘raid’ flag on the drive. You can do this through gparted.
    Then add it with the mdadm –add

  44. Jeremy Leigh says:

    Oooh guys. Careful when adding drives like this. I just lost 3 hours trying to find what was going on.
    When you resize your raid, make sure you edit your /etc/mdadm.conf file to update the number of devices in your raid configuration!
    Otherwise you reboot and poof, ur raid is gone 🙁
    At least this happened to me on Fedora Core 7.
    Stupid thing fails on boot and goes to this “Repair filesystem” thingy because it can’t mount /dev/”insert your lvm or fs here”.
    The anoying thing is, the Repair Filesystem mount’s your root readonly!
    To get your / writeable:
    mount -w -o remount /

    Then edit your /etc/mdadm.conf to the new number of drives in your configuration.
    Or alternatively, edit your /etc/fstab and take out the line that mounts your raid (then you can fix it later).

  45. Ferg says:

    As long as each member of the array has the partition type set to FD – Raid autodetect, then you should not have to update mdadm.conf. The MD driver should autodetect, assemble and start each array.

    You mention that the ROOT was mounted read only. This cannot have been a RAID5 device, as to the best of my knowledge you can only boot from a RAID1 array. IS that true?

    Cheers

  46. Jeremy Leigh says:

    I am not booting from the raid. I have another drive in my pc seperate from raid that is running the linux.
    Can you explain a little more about this FD partition type.
    I have my raid drives formatted as a blank partition, and then i have set the flags ‘boot’, and ‘raid’, on each partition. This is also in fact how DiskDruid creates ‘software raid’ partitions.
    And yes, mdadm auto-detects the drives, but only up until the number of devices that is specified in /etc/mdadm.conf

    DEVICE partitions
    MAILADDR root
    ARRAY /dev/md0 level=raid5 num-devices=6 uuid=46e9ab9d:c365cd2c:e08c1bc3:73c070ca

    maybe, if the ‘num-devices=6’ was removed entirely, then it would detect the drives fine. I will try that.
    It is also possible to specify which drives you want in your array specifically i.e.

    ARRAY /dev/md0 level=raid5 num-devices=6 uuid=46e9ab9d:c365cd2c:e08c1bc3:73c070ca devices=/dev/sdb1,/dev/sdc1,/dev/sdd1,/dev/sde1,/dev/sdf1,/dev/sdg1

    if you specify like this, then the config will ignore any drives not in the devices list.

  47. Daniel Albuschat says:

    I’ve just successfully completed the RAID growth of my 3x500GB RAID5. It’s now 4x500GB, and it took something about a full day, including filesystem-check and resize.

  48. Som says:

    I’m actually doing the same thing now with Ubuntu 7.04 — expanding a 3x500GB RAID5 to 4x500GB.

    The grow took about 12 hours. I ran fsck.ext3 with no issue, but when I went to run resize2fs it said, “Please run ‘e2fsck -f /dev/md0’ first”. Did you have to do this, Daniel?

    That’s where I am now… the first time I ran it my ssh connection was dropped. Wasn’t listed as a process when I relogged in, so I ran it again. We’ll see how it goes…

    – Som

  49. Som says:

    Well, I wasn’t able to run e2fsck from over ssh — the connection kept dropping.

    I went home and ran it directly on the box. Everything worked great.

    So, as a recap:

    Ubuntu 7.04 (Feisty Fawn)

    3x500GB grown to 4x500GB

    mdadm –add /dev/md0 /dev/sde1
    mdadm –grow /dev/md0 –raid-devices=4
    (~12 hours)
    e2fsck -f /dev/md0
    (~15 minutes)
    resize2fs /dev/md0
    (~10 minutes)

    – Som

Comments are closed.