Migrating My Root Drive Raid1 Array to a Pair of NVME Drives.

I’ve always tried to keep upgrading my own Linux boxes. I enjoy it and as I found out a few years ago if I do not regularly keep updating hardware then I completely lose knowledge of doing so.
My latest project is to move all the services from my workstation to a separate box. I’ve got a few RPis around the house running a few services, but I’ve always used my workstation as a games machine/server/development/everything else. In fact for a number of years I stopped using Linux as a workstation at all and this machine was a headless server.
Because of this cycle of continuous upgrades this computer has existed for probably twenty years. Always running some form of Linux (mainly Gentoo).
Currently it’s using some space heater of a server motherboard with a pair of E5 2967v2 on a Supermicro X9dri-LN4 motherboard with 128Gb of ECC DDR3 RAM (very cheap!). That’s fine over winter (my office has no heating) but I do need to fully transfer all the services to the low power Debian box under my desk instead!
Anyway this computer boots from pair of SATA SSD drives in a RAID1 array, with a six disk RAID10 array for data. That array needs to be replaced by a single large drive when I’ve finished moving services to a new machine….!
The motherboard is too old to EFI boot from NVME drives. However, whilst browsing Reddit I came across some people talking about using an adaptor card to add in four NVME drives and using bifurcation toggle each drive proper access to the four PCIe channels that NVME devices need. So x4,x4,x4,x4 instead of x16.
This was not supported on this board, but turns out Supermicro did release a new BIOS that does support bifurcation.
So I bought the card they suggest and a pair of 1TB NVMe drives. The drives are only PCIev3 as that’s all the motherboard supports. PCIe is backwards/forwards compatible. but PCIev4 drives are considerably more expensive than PCIeV3 ones. I may as well get a pair of these, then when I upgrade to a PCIeV4 motherboard the available drives will likely be larger and cheaper!
– Asus M.2. X16 Gen 4
– 2 x WD Blue SN570 NVMe SSD 1TB
The adaptor and cards came. The adapter has got a lovely heatsink that sandwiches the drives in with a small low noise fan.
The adaptor took ten minutes to install. When booted up the BIOS setup disk was a little tricky to enable bifurcation as the slots are numbered from the bottom. This one was CPU1/slot 1.
I had to recompile the kernel to add NVME support, but once booted the pair of drives were there.
After many, many years of using /dev/sdX to refer to storage devices (I was using SCSI hardware before SATA), it does seem a little strange to be running parted on /dev/nvme1n1 then getting partition devices like /dev/nvme1n1p2
I know I should likely move to ZFS, but I’m knowledgeable enough about mdadm not to completely mess up things! ..and replacing a pair of RAID1 devices is just so easy with mdadm.

Workflow is:
– partition drive.
– add to raid array as spare
– fail drive to be removed and then remove it.
– Wait until raid1 array is synced again.
– repeat with second drive
– resize array, then resize the filesystem

procedure

fdisk /dev/nvme0n1

We can use fdisk again as fdisk is GPT aware. Previously we’d always used parted. But I prefer fdisk as I know it!
– partition
– Label the drive as GPT
– Make 256mb partition and mark as EFI boot.
– Make 2nd partition for the rest of the drive and mark that as type Linux raid.

Now add that drive to our raid1 array. For some reason it was not added to a spare, but was instead immediately synced to make a 3 drive raid1 array. I think this is because I previously created this array as a three drive array (for reasons I forget). I guess that’s stored in the metadata of the array.

mdadm /dev/md127 --add /dev/nvme0n1p2

We can watch the progress:

watch cat /proc/mdstat

Once completed we can fail and then remove the drive

mdadm --manage /dev/md127 --fail /dev/sdh3
mdadm /dev/md127 --remove /dev/sdh3

Then let’s update our mdadm.conf file

mdadm --detail --scan >> /etc/mdadm/mdadm.conf

Then remove the oldlines:

vi /etc/mdadm/mdadm.conf

Finally let’s wipe that old drive of RAID data so the array does not try to reassemble it to the array.

wipefs -a /dev/sdh3

A reboot is a good idea now to ensure the array is correctly assembled (and new PAT reread).

Now let#s copy the partition table (PAT) to the second new drive.

sgdisk /dev/nvme0n1 -R /dev/nvme1n1

Then randomise the UIDs

sgdisk -G /dev/nvme1n1

Check all is OK

Now repeat adding the second new drive:

mdadm /dev/md127 --add /dev/nvme1n1p2
mdadm --manage /dev/md127 --fail /dev/sdg3
mdadm /dev/md127 --remove /dev/sdg3
mdadm --detail --scan >> /etc/mdadm/mdadm.conf
wipefs -a /dev/sdg3

resize after adding new devices

mdadm –grow –size max /dev/md127
df -h
Then resize the filesystem
resize2fs -p /dev/md127

benchmarks

dd if=/dev/zero of=/home/chris/TESTSDD bs=1G count=2 oflag=dsync 

2+0 records in
2+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 4.7162 s, 455 MB/s

dd if=/dev/zero of=/mnt/storage/TESTSDD bs=1G count=2 oflag=dsync

2+0 records in
2+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 10.0978 s, 213 MB/s

dd if=/home/chris/TESTSDD of=/dev/null bs=8k                                                              !10032

262144+0 records in
262144+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 0.666261 s, 3.2 GB/s

dd if=/mnt/storage/TESTSDD of=/dev/null bs=8k                                                             !10034

262144+0 records in
262144+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 0.573111 s, 3.7 GB/s

I did think that the write speed would be faster. But I guess dd is not the most accurate of benchmark tools.

Upgrading Linux boxes

After returning to upgrading my main Linux box (due to playing with Docker and using some stuff (genome annotation pipelines) that needs more than the system max of 16GB RAM) I came across this blog post about a similar situation (albeit more time). The beautiful machine
I’ve generally always upgraded my own computers. My main Linux box has been upgraded in bits and pieces for some time. I think the oldest current part is the case that is at least 15 years old. It’s been heavily updated with sound insulation from the car audio scene, although TBH it is much quieter now than when the motherboard was an Asus PC-DL running a pair of power hungry overclocked Xeons.
However, for one reason or another (lack of time, stable hardware, iPadOS etc) I’ve not done this for some time. My box was last opened two years ago when the TV tuner card (PCIe TBS6980) died and I replaced it, with an almost exact model. The previous real upgrade was seven or more years ago.
So I got myself a 10 yr old server board with dual Xeons and Max 32gb ram. Intel S5500BC board with a pair of Xeon E6240s. The setup only cost £70 but each CPU is far faster than the previous single Xeon X3470. Plus max RAM is double.
However, when I could previously swap over a motherboard in less than an hour, now we got multiple beginner errors.
– First error was refusing to boot due to a grounding error. I’d assumed there would be the usual 9 standoffs in the ATX format. Nope. There is no motherboard hole for a middle bottom standoff. What compounded the gorging error was that the lower left standoff was too short. Whoops!
– Then I’d not inserted RAM correctly. Turns out a proper server board does not fail to POST, but just omits the DIMM slots and allows the rest to work. Luckily a red LED indicates the incorrect DIMM slots.
– Then all 32GB of RAM (8 x 4GB) was recognised, but once booted into Linux only 24 GB was seen. Turns out another beginner error and the DIMM was inserted enough to be recognised, but not enough to work properly.
As well as the beginner errors the board is a server one and CPU fans ran so fast that it was difficult to think with the noise. Turns out most Intel server boards are intended to be paired with an Intel chassis. If the board does not detect the chassis it just switches on the fans full speed instead of managing then due to CPU heat! Noisy!
I found a few blog posts on reflashing the BIOS to a more recent one AND also something called a Baseboard Management Controller. When did they come along? This adds a non-Intel chassis profile for fan speed and allows it to be managed in line with CPU heat.
Even though Intel have EOLed these boards, I still found the latest BIOS on their site. The current BIOS was so old though I needed to update to an intermediate build, BIOS 66. Then flash to BIOS 69 which is the latest. Flashing the BIOS on servers boards is easy! The board used EFI can can be booted to a console, which allows you to flash the BIOS from a USB stick. Even easier there’s a BAT script to do this from the USB. Funky!
BUT the BMC firmware was very difficult to find. I eventually found it on a niche You Tube video.
Anyway the lesson I should learn is hardware upgrades can only be easy if you spend a lot of money. If you want to save money then you need to do them regularly to keep your skills up!