I have for a while had a RAID-1 disk array on my Linux based (Centos 5) home server. As I was beginning to run out of disk space, I wanted to add an extra disk and change it to a RAID-5 disk array thereby doubling the amount of disk space available.
My Dell PowerEdge 930 has a drive bay with room for 4 hard drives. The server originally came with a Western Digital 80 GB SATA drive. This contains everything, except for /home. 2 Maxtor DiamondMax 10 300 GB drives were originally setup in a software RAID-1 array. So to get that extra disk space, I bought a new hard disk that I wanted to add to the server and then turn the RAID-1 array into a RAID-5 array,
I was going to do it the hard way, take a backup, wipe away the old array, create a new one and restore the data. But I found a blog entry here that explains how to change it online. So this blog entry is first of all to tell anyone interested that it actually works, but also to document those steps that weren’t entirely clear (to me at least). It was quite some time ago that I created the original software RAID array on the server, so there were a few additional steps necessary that are not documented in the above. You need to follow the instructions in Scotts document, and you can then refer to my document in case you run into some of the same issues I did.
My home server hosts a couple of web sites, a mail server + several home services such as being a file server for, downtime had to be minimised which is why the in-place conversion appealed so much to me. I was even braver than Scott and except for a few actions where I stopped some services and then started them again shortly after when everything seemed all right, all services were active throughout the conversion
Stopping the RAID array
My server has no keyboard or monitor attached, so I was going to do this via ssh. That was only possible because I wasn’t touching the boot drive. If you want to do this with the boot drive, you need a real screen and keyboard attached and do what you are doing there. The mdadm –stop command would not stop the array and it took me a while to realise I had forgotten to umount the array. I couldn’t do that either, I kept being told that the device was in use or busy. The lsof command
lsof |grep /home
finally showed me the issue. I had logged in as a normal user and then issued the su command, so /home/users was in use. I needed to login directly as root.
New partition on new disk
The disks I originally purchased are no longer in production, and the new disk of course doesn’t have the same size as the original disks. Some Linux RAID documentation seems to imply that it doesn’t matter if the partitions are of different size, whilst other documents said that you would lose data if the partitions were not of the same size. I wasn’t going to experiment, so I made the partition on the new disk the exact same size as the partitions on the old disks. The simplest way to get a partition on the new disk to be similar to the partitions on the old disk is to use the sfdisk command which on my server meant I did
sfdisk -d /dev/sdb | sfdisk /dev/sdd
Re-shape process
Doing “cat /proc/mdstat” gives you statistics about the data transfer speed and the number of minutes it will take. The value for the data transfer speed was fairly consistently around 4000 KB/sec and it claimed it would ~1100 minutes. I tried speeding it up by manipulating the values for:
/proc/sys/dev/raid/speed_limit_min
/proc/sys/dev/raid/speed_limit_max
It didn’t seem to make any difference so I finally left it and went to bed. Next morning when I woke up it was finished reshaping, and I did not sleep for that long. As you can see from the graph, at some stage, it picked up speed and raced through the rest of the reshaping.
Resizing the array
Reading various RAID documentation makes it clear that there is an optimum “stride” value which is equal to the chunk size divided by the block size. In my case the chunk size is 64 KB and the block size is 4 KB, so the stride value should be 16. You can find the block size with
dumpe2fs –h /dev/md0
and the chunk size with
cat /proc/mdstat
The man pages for resize2fs “will heuristically determine the RAID stride that was specified when the file system was created”. I have no idea whether that means that it will pick the optimum stride value, so I chose to specify the value manually, so on my system I did
resize2fs –p –S 16 /dev/md0
By doing a du with one minute in between, I could see that the resize was happening at about 16 GB/min.
Summrary: It works, I had minimal downtime, and I essentially had two sets of data to operate on. The data on the array being converted and the data on the backup.