Software RAID on Linux

Software RAID is performed via the Linux kernel. No RAID controller is required on the computer. If the computer has a Linux compatible RAID controller then hardware RAID can be used. Hardware RAID is not discussed here.

mdadm is a Linux utility used to manage and monitor software RAID devices. It is used in modern GNU/Linux distributions in place of older software RAID utilities such as raidtools2 or raidtools.

Software RAID on 3.x Linux

the /boot filesystem

In the past the /boot filesystem must be used either without md support, or else with RAID1. With modern kernels the it is possible to load the MD support as a kernel module through the initramfs mechanism. This approach allows the /boot filesystem to be inside any RAID system without the need of a complex manual configuration.

• In the past you couldn't put /boot inside the RAID array, now you can.

So you have a couple options in order to have a setup where each drive can be bootable. One is to have a /boot partition on each drive, or the other involves having /boot within a RAID array.

Mdadm

The mdadm utility can be used to create and manage storage arrays.

Example: 3-way RAID-1 array

The /boot partition is stored at /dev/md0. This installs GRUB to each disk, so that if one disk fails, you can boot off one of the other disks.

# grub
grub> find /grub/stage1
 (hd0,0)
 (hd1,0)
 (hd2,0)
grub> device (hd0) /dev/sda
grub> root (hd0,0)
grub> setup (hd0)
grub> device (hd0) /dev/sdb
grub> root (hd0,0)
grub> setup (hd0)
grub> device (hd0) /dev/sdc
grub> root (hd0,0)
grub> setup (hd0)
grub> quit

Legacy Linux Software RAID

Linux Software RAID - An unorganized and random comments page which may or may not be helpful to you. *Created: Mon Mar 29 18:28:01 CST 2004

For the most part software RAID functions under linux now as it did ten years ago. However, since the information in this section was collected over ten years ago it will be considered outdated.

Software RAID 1 Example

For this example an older 32-bit Pentium computer is used as a file server with two large capacity drives configured to RAID Level 1: Drive Mirroring.

Trisquel Linux is a lightweight Debian based distribution. Booting from the Live CD load the Trisuel environment. Installation can be performed here only after the drives are properly configured for RAID. Partition the drives and create the RAID array before installing the Linux distribution.

Each drive is the same size. Each drive needs to have identical partitions created. Using fdisk create a small partition of less than 1GB and use the remaining space for the second partition. Due this for each drive. The starting block will likely be at 2048, which is fine. Ensure that you use the same starting block and ending block on both.

When specifying the ending block in fdisk new partition creation, you can specify the starting block as 2048 and then for the ending block specify a size in megabytes. Use the format beginning with the + sign. For 840MB enter "+840" when it asks for the ending block.

Step Summary:

1. Create primary partition at the beginning of each of the two hard drives and of a small size such as 1GB or less.
2. Create another primary partition using the remaining disk space on each of the drives.
3. Use fdisk to convert partitions to a type needed for building the RAID array
4. Install RAID manager: mdadm apt-get install mdadm
5. The first partition will be used to mount /boot
6. The second partition will be used for SWAP space and / root.
7. Install Linux on the RAID partitions

When specifying the ending block in fdisk new partition creation, you can specify the starting block as 2048 and then for the ending block specify a size in megabytes. Use the format beginning with the + sign. For 840MB enter "+840" when it asks for the ending block.

If you have 2 drives, /dev/sda and /dev/sdb

fdisk /dev/sda

Create Partitions with fdisk

• d - delete any existing partitions
• n - create a new partition

fdisk /dev/sdb

Create Partitions with fdisk

• d - delete any existing partitions
• n - create a new partition

Once again using fdisk go back into each drive and change the "type" of the partitions to

Now use fdisk to convert partitions to a type needed for building the RAID array

fdisk /dev/sda

• t - change the partition type to "fd". You can get a list of available types

Software RAID 5 Example

/boot on the boot partition cannot be RAID 5. This is a limitation in Linux software raid. For this example I have 3 IDE harddrives set up for RAID level five and a seperate raid partition for the following:

/boot assigned to md0 with RAID level 1 
/tmp  assigned to md2 with RAID level 5 
/     assigned to md3 with RAID level 5 
/home assigned to md4 with RAID level 5 

and the linux swap partition on md1 RAID level 5.

Three harddrives are hda, hdb, and hdc.

md0 is a raid partition, consisting of hda1, hdb1, and hdc1. A complete table of raid partitions with drive assignments is as follows:

hda1 + hdb1 + hdc1 assigned to md0 (/boot)
hda2 + hdb2 + hdc2 assigned to md4 (/home)
hda3 + hdb3 + hdc3 assigned to md3 (/)
hda5 + hdb5 + hdc5 assigned to md2 (/tmp)
hda6 + hdb6 + hdc6 assigned to md1 (swap)

(Q). Did everything load ok with RAID when I booted?

Take a look at the boot messages and see for yourself. Type the following command at the console:

'dmesg|less'

(Q). Hows RAID doing now? or Has a drive died while its running?

Take a look at the raid status by typing the following command at the console:

'cat /proc/mdstat'

(Q). What does it mean when I see "kicking hda1" or "kicking" whatever in the dmesg and RAID / the system will not start?

Looks like you have a bad drive or the drive is fine but RAID seems to think there is a problem. See which drive was kicked and try using the following command to add it back into the array: 'raidhotadd /dev/md0 /dev/hda1' replacing md0 with the array that the drive was kicked from and hda1 with the actual drive that was kicked from that array. Be careful not to accidently use raidhotadd to add a drive partition into the wrong array. Linux/RAID will let you do this but you will regret it.

raidhotadd and raidhotremove will only work on a running array. If only 1 out of the 3 drive partitions are functioning within an array, then raidhotadd and raidhotremove are useless.

You can usually tell what drive belongs to what array by looking at your RAID configuration file, which is /etc/raidtab.

(*). Useful command summary:

vi /etc/raidtab
dmesg
cat /proc/mdstat
raidhotadd /dev/md? /dev/hd??
raidhotadd
raidhotremove
lsraid -a /dev/md0
lsraid -A -d /dev/hdc2

(man). lsraid usage examples

• lsraid -A -a /dev/md0

Display a short listing of the md0 device.

• lsraid -A -d /dev/sda1

Display a short listing of the array that sda1 belongs to.

• lsraid -A -f -a /dev/md0

Display the failed devices belonging to the md0 device.

• lsraid -D -l -a /dev/md0

Display a long dump of the on-disk md superblock of every disk in

md0.

• lsraid -D -a /dev/md0 -d /dev/sda1

Display a short discription of the disks in md0 as well as a short

description of the disk sda1. sda1 will only be described once if

it belongs to md0.

• lsraid -R -a /dev/md0 -a /dev/md1 -a /dev/md2

Display a description of the arrays in an output format suitable

for using in raidtab(5) files. Note that if md0 and md1 are raid0

arrays and md2 is a raid1 created from md0 and md1, this command

will output the information in the correct order.

• lsraid -R -p

Scan all block devices in /proc/partitions and display all discov-

ered md devices in a format suitable for using in raidtab(5) files.

Command Reference by Example

vi /etc/raidtab                 

dmesg                           

cat /proc/mdstat                

raidhotadd /dev/mdN /dev/hdXN   

raidhotremove                   

lsraid -a /dev/mdN              

lsraid -A -d /dev/hdcN          

raidstop                        

raidsetfaulty                   

mkraid -f (dangerous!)          

badblocks                       

mount /proc /proc -t proc