This document illustrates swapping a drive in intel ISW fakeraid - the BIOS
utility is aware of the RAID sets, and Linux interfaces with them via the older
dmraid utility. In this example the machine has a RAID 10 setup.

Note that RAID 10 is implemented as a stripe of two mirrors, with the stripe
visible as the "superset" device and the mirrors visible as separate
sub-devices, each of which has a pair of actual physical drives.

Note also that the /dev/sda, /dev/sdb, etc. names correspond to actual
physical drives (not RAID volumes), and that they are assigned to the drives
in the order that they initialize, so they don't necessarily correspond to the
slot numbers.

useful commands for interrogating the arrays:

To check what the drive letters correspond to:

		ls -l /dev/disk/by-path

(see example output in transcript below -- scsi-0:0:0:0 indicates slot 0,
scsi-1:0:0:0 is slot 1, etc.)

see what drives are part of fakeraid sets, and subsets, with lots of
verbosity:

		dmraid -s -s -vv

brief listing of what drives are part of what fakeraid superset:

		dmraid -r

monitor rebuild status:

		dmsetup status

show which sets are mirrors, stripes, etc.:

		dmsetup status

List raid volumes & partitions handled by device mapper / dmraid:

		dmsetup ls

or:

		ls -l /dev/mapper/

List which drives are part of each dmraid subset / superset (the dm-? numbers
correspond to the second numbers in the output from "dmsetup ls"):

		ls /sys/block/dm-?/slaves

Tell fakeraid to start rebuilding the specified array using the specified new
disk (make sure you check which device is degraded first [will be the one with
fewer drives than it's supposed to have, in "dmraid -s -s -vv"], and which
drive is definitely the new one):

		dmraid -R isw_dxifadxaii_Volume0-0 /dev/sdd


================================================================================

Below is a transcript of adding a new drive to a degraded RAID 10 array on
the server 'fishie'. The new drive was zeroed first, to get rid of any
stray RAID signatures that may have been on it.

In the transcript, the system had been booted with one array component
degraded (the drive that was in slot 0 had failed), so sda, sdb, sdc were
slots 1, 2, and 3 respectively, and slot 0 was not responding, so it did not
get a device name at boot time. A new drive was then added in slot 0, which
was the fourth SATA drive the OS saw initialize, so slot 0 became /dev/sdd.
This was then re-added to the failed array to start a rebuild.

================================================================================


verify which drive is dead:

	dmesg | less

look for drive initialization, e.g. look shortly before the first time sda is mentioned
reveals this:

	ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
	ata1.00: both IDENTIFYs aborted, assuming NODEV

so the bad drive is on ata1; see what the good drives got:

	sd 1:0:0:0: Attached scsi disk sda
	sd 2:0:0:0: Attached scsi disk sdb
	sd 3:0:0:0: Attached scsi disk sdc

the first disk usually gets "sd 0:0:0:0", so this implies it's the first one.
confirm by blinking all the drive lights:


	[root@fishie tmp]# dd if=/dev/zero of=testfile bs=1M conv=fdatasync


...yep, first one is dark, all three others are lit. Pull the first drive and
insert the replacement.

Now look at dmesg to verify that the OS can see the new disk:

	[root@fishie tmp]# dmesg | tail
	ata1: hard resetting link
	ata1: SATA link down (SStatus 0 SControl 300)
	ata1: EH complete
	ata1: exception Emask 0x10 SAct 0x0 SErr 0x4050000 action 0xe frozen
	ata1: irq_stat 0x00400040, connection status changed
	ata1: SError: { PHYRdyChg CommWake DevExch }
	ata1: hard resetting link
	ata1: link is slow to respond, please be patient (ready=0)
	ata1: softreset failed (device not ready)
	ata1: hard resetting link

	[root@fishie tmp]# dmesg | tail
	sdd: Write Protect is off
	sdd: Mode Sense: 00 3a 00 00
	SCSI device sdd: drive cache: write back
	SCSI device sdd: 976773168 512-byte hdwr sectors (500108 MB)
	sdd: Write Protect is off
	sdd: Mode Sense: 00 3a 00 00
	SCSI device sdd: drive cache: write back
	 sdd: unknown partition table
	sd 0:0:0:0: Attached scsi disk sdd
	sd 0:0:0:0: Attached scsi generic sg4 type 0


OK, the new disk is sdd. We can see that it it's on "sd 0:0:0:0" as expected;
we can also see this from /dev/disk/by-path:

	[root@fishie tmp]# ls -l /dev/disk/by-path
	lrwxrwxrwx 1 root root 9 Feb 17 15:36 pci-0000:00:1f.2-scsi-0:0:0:0 -> ../../sdd
	lrwxrwxrwx 1 root root 9 Feb 15 16:32 pci-0000:00:1f.2-scsi-1:0:0:0 -> ../../sda
	lrwxrwxrwx 1 root root 9 Feb 15 16:32 pci-0000:00:1f.2-scsi-2:0:0:0 -> ../../sdb
	lrwxrwxrwx 1 root root 9 Feb 15 16:32 pci-0000:00:1f.2-scsi-3:0:0:0 -> ../../sdc
	lrwxrwxrwx 1 root root 9 Feb 15 16:32 pci-0000:00:1f.2-scsi-5:0:0:0 -> ../../sr0

Now look at dmraid status:

	[root@fishie tmp]# dmraid -s -s -vv
	NOTICE: /dev/sda: asr     discovering
	NOTICE: /dev/sda: ddf1    discovering
	NOTICE: /dev/sda: hpt37x  discovering
	NOTICE: /dev/sda: hpt45x  discovering
	NOTICE: /dev/sda: isw     discovering
	NOTICE: /dev/sda: isw metadata discovered
	NOTICE: /dev/sda: jmicron discovering
	NOTICE: /dev/sda: lsi     discovering
	NOTICE: /dev/sda: nvidia  discovering
	NOTICE: /dev/sda: pdc     discovering
	NOTICE: /dev/sda: sil     discovering
	NOTICE: /dev/sda: via     discovering
	NOTICE: /dev/sdb: asr     discovering
	NOTICE: /dev/sdb: ddf1    discovering
	NOTICE: /dev/sdb: hpt37x  discovering
	NOTICE: /dev/sdb: hpt45x  discovering
	NOTICE: /dev/sdb: isw     discovering
	NOTICE: /dev/sdb: isw metadata discovered
	NOTICE: /dev/sdb: jmicron discovering
	NOTICE: /dev/sdb: lsi     discovering
	NOTICE: /dev/sdb: nvidia  discovering
	NOTICE: /dev/sdb: pdc     discovering
	NOTICE: /dev/sdb: sil     discovering
	NOTICE: /dev/sdb: via     discovering
	NOTICE: /dev/sdc: asr     discovering
	NOTICE: /dev/sdc: ddf1    discovering
	NOTICE: /dev/sdc: hpt37x  discovering
	NOTICE: /dev/sdc: hpt45x  discovering
	NOTICE: /dev/sdc: isw     discovering
	NOTICE: /dev/sdc: isw metadata discovered
	NOTICE: /dev/sdc: jmicron discovering
	NOTICE: /dev/sdc: lsi     discovering
	NOTICE: /dev/sdc: nvidia  discovering
	NOTICE: /dev/sdc: pdc     discovering
	NOTICE: /dev/sdc: sil     discovering
	NOTICE: /dev/sdc: via     discovering
	NOTICE: /dev/sdd: asr     discovering
	NOTICE: /dev/sdd: ddf1    discovering
	NOTICE: /dev/sdd: hpt37x  discovering
	NOTICE: /dev/sdd: hpt45x  discovering
	NOTICE: /dev/sdd: isw     discovering
	NOTICE: /dev/sdd: jmicron discovering
	NOTICE: /dev/sdd: lsi     discovering
	NOTICE: /dev/sdd: nvidia  discovering
	NOTICE: /dev/sdd: pdc     discovering
	NOTICE: /dev/sdd: sil     discovering
	NOTICE: /dev/sdd: via     discovering
	NOTICE: added /dev/sda to RAID set "isw_dxifadxaii"
	NOTICE: added /dev/sdb to RAID set "isw_dxifadxaii"
	NOTICE: added /dev/sdc to RAID set "isw_dxifadxaii"
	ERROR: isw: wrong number of devices in RAID set "isw_dxifadxaii_Volume0-0" [1/2] on /dev/sda
	*** Group superset isw_dxifadxaii
	--> Active Superset
	name   : isw_dxifadxaii_Volume0
	size   : 1953535744
	stride : 128
	type   : raid01
	status : ok
	subsets: 2
	devs   : 3
	spares : 0
	--> *Inconsistent* Active Subset
	name   : isw_dxifadxaii_Volume0-0
	size   : 976767872
	stride : 128
	type   : mirror
	status : inconsistent
	subsets: 0
	devs   : 1
	spares : 0
	--> *Inconsistent* Active Subset
	name   : isw_dxifadxaii_Volume0-1
	size   : 976767872
	stride : 128
	type   : mirror
	status : inconsistent
	subsets: 0
	devs   : 2
	spares : 0

OK, dmraid sees /dev/sdd, and it's not part of any arrays, and it looks as if
"isw_dxifadxaii_Volume0-0" is the sub-array with the missing disk (it shows
"devs: 1"), i.e. where we want to add it. Another way of looking at the array:

	[root@fishie tmp]# dmsetup status
	isw_dxifadxaii_Volume0: 0 1953535744 striped 2 253:0 253:1 1 AA
	isw_dxifadxaii_Volume0-1: 0 976767995 mirror 2 8:32 8:16 7453/7453 1 AA 1 core
	isw_dxifadxaii_Volume0p3: 0 12578895 linear
	isw_dxifadxaii_Volume0-0: 0 976767995 linear
	isw_dxifadxaii_Volume0p2: 0 12578895 linear
	isw_dxifadxaii_Volume0p1: 0 256977 linear
	isw_dxifadxaii_Volume0p5: 0 1928105172 linear

0-1 says "mirror", but 0-0 says "linear". Let's add the new drive to 0-0 with
the rebuild command:

	[root@fishie tmp]# dmraid -R isw_dxifadxaii_Volume0-0 /dev/sdd
	ERROR: isw: wrong number of devices in RAID set "isw_dxifadxaii_Volume0-0" [1/2] on /dev/sda
	isw: drive to rebuild: /dev/sdd

	RAID set "isw_dxifadxaii_Volume0" already active
	device "isw_dxifadxaii_Volume0-0" is now registered with dmeventd for monitoring
	device "isw_dxifadxaii_Volume0-1" is now registered with dmeventd for monitoring
	device "isw_dxifadxaii_Volume0" is now registered with dmeventd for monitoring
	Error: Unable to write to descriptor!
	Error: Unable to execute set command!
	Error: Unable to write to descriptor!
	Error: Unable to execute set command!

Somewhat alarming error messages, but it did work:

	[root@fishie tmp]# dmraid -s -s -vv
	NOTICE: /dev/sda: asr     discovering
	NOTICE: /dev/sda: ddf1    discovering
	NOTICE: /dev/sda: hpt37x  discovering
	NOTICE: /dev/sda: hpt45x  discovering
	NOTICE: /dev/sda: isw     discovering
	NOTICE: /dev/sda: isw metadata discovered
	NOTICE: /dev/sda: jmicron discovering
	NOTICE: /dev/sda: lsi     discovering
	NOTICE: /dev/sda: nvidia  discovering
	NOTICE: /dev/sda: pdc     discovering
	NOTICE: /dev/sda: sil     discovering
	NOTICE: /dev/sda: via     discovering
	NOTICE: /dev/sdb: asr     discovering
	NOTICE: /dev/sdb: ddf1    discovering
	NOTICE: /dev/sdb: hpt37x  discovering
	NOTICE: /dev/sdb: hpt45x  discovering
	NOTICE: /dev/sdb: isw     discovering
	NOTICE: /dev/sdb: isw metadata discovered
	NOTICE: /dev/sdb: jmicron discovering
	NOTICE: /dev/sdb: lsi     discovering
	NOTICE: /dev/sdb: nvidia  discovering
	NOTICE: /dev/sdb: pdc     discovering
	NOTICE: /dev/sdb: sil     discovering
	NOTICE: /dev/sdb: via     discovering
	NOTICE: /dev/sdc: asr     discovering
	NOTICE: /dev/sdc: ddf1    discovering
	NOTICE: /dev/sdc: hpt37x  discovering
	NOTICE: /dev/sdc: hpt45x  discovering
	NOTICE: /dev/sdc: isw     discovering
	NOTICE: /dev/sdc: isw metadata discovered
	NOTICE: /dev/sdc: jmicron discovering
	NOTICE: /dev/sdc: lsi     discovering
	NOTICE: /dev/sdc: nvidia  discovering
	NOTICE: /dev/sdc: pdc     discovering
	NOTICE: /dev/sdc: sil     discovering
	NOTICE: /dev/sdc: via     discovering
	NOTICE: /dev/sdd: asr     discovering
	NOTICE: /dev/sdd: ddf1    discovering
	NOTICE: /dev/sdd: hpt37x  discovering
	NOTICE: /dev/sdd: hpt45x  discovering
	NOTICE: /dev/sdd: isw     discovering
	NOTICE: /dev/sdd: isw metadata discovered
	NOTICE: /dev/sdd: jmicron discovering
	NOTICE: /dev/sdd: lsi     discovering
	NOTICE: /dev/sdd: nvidia  discovering
	NOTICE: /dev/sdd: pdc     discovering
	NOTICE: /dev/sdd: sil     discovering
	NOTICE: /dev/sdd: via     discovering
	NOTICE: added /dev/sda to RAID set "isw_dxifadxaii"
	NOTICE: added /dev/sdb to RAID set "isw_dxifadxaii"
	NOTICE: added /dev/sdc to RAID set "isw_dxifadxaii"
	NOTICE: added /dev/sdd to RAID set "isw_dxifadxaii"
	*** Group superset isw_dxifadxaii
	--> Active Superset
	name   : isw_dxifadxaii_Volume0
	size   : 1953535744
	stride : 128
	type   : raid01
	status : ok
	subsets: 2
	devs   : 4
	spares : 0
	--> Active Subset
	name   : isw_dxifadxaii_Volume0-0
	size   : 976767872
	stride : 128
	type   : mirror
	status : nosync
	subsets: 0
	devs   : 2
	spares : 0
	--> Active Subset
	name   : isw_dxifadxaii_Volume0-1
	size   : 976767872
	stride : 128
	type   : mirror
	status : nosync
	subsets: 0
	devs   : 2
	spares : 0

Looks like it was added successfully nonetheless. Both 0-0 and 0-1 show 2 devs
and "mirror" now, and "nosync" implies that it hasn't synched up yet. We can
monitor rebuild progress with:

	[root@fishie tmp]# dmsetup status
	isw_dxifadxaii_Volume0: 0 1953535744 striped 2 253:0 253:1 1 AA
	isw_dxifadxaii_Volume0-1: 0 976767995 mirror 2 8:32 8:16 1005/7453 1 AA 1 core
	isw_dxifadxaii_Volume0p3: 0 12578895 linear
	isw_dxifadxaii_Volume0-0: 0 976767995 mirror 2 8:0 8:48 1065/7453 1 AA 1 core
	isw_dxifadxaii_Volume0p2: 0 12578895 linear
	isw_dxifadxaii_Volume0p1: 0 256977 linear
	isw_dxifadxaii_Volume0p5: 0 1928105172 linear

the following demonstrate other ways to see some of the same information in
various forms:

	[root@fishie tmp]# dmraid -r
	/dev/sda: isw, "isw_dxifadxaii", GROUP, ok, 976773165 sectors, data@ 0
	/dev/sdb: isw, "isw_dxifadxaii", GROUP, ok, 976773165 sectors, data@ 0
	/dev/sdc: isw, "isw_dxifadxaii", GROUP, ok, 976773165 sectors, data@ 0
	/dev/sdd: isw, "isw_dxifadxaii", GROUP, ok, 976773165 sectors, data@ 0

	[root@fishie tmp]# dmsetup ls
	isw_dxifadxaii_Volume0  (253, 2)
	isw_dxifadxaii_Volume0-1        (253, 1)
	isw_dxifadxaii_Volume0p3        (253, 5)
	isw_dxifadxaii_Volume0-0        (253, 0)
	isw_dxifadxaii_Volume0p2        (253, 4)
	isw_dxifadxaii_Volume0p1        (253, 3)
	isw_dxifadxaii_Volume0p5        (253, 6)

	[root@fishie tmp]# ls -l /dev/mapper/
	total 0
	crw------- 1 root root  10, 63 Feb 15 16:31 control
	brw-rw---- 1 root disk 253,  2 Feb 15 16:31 isw_dxifadxaii_Volume0
	brw-rw---- 1 root disk 253,  0 Feb 15 16:31 isw_dxifadxaii_Volume0-0
	brw-rw---- 1 root disk 253,  1 Feb 15 16:31 isw_dxifadxaii_Volume0-1
	brw-rw---- 1 root disk 253,  3 Feb 15 16:53 isw_dxifadxaii_Volume0p1
	brw-rw---- 1 root disk 253,  4 Feb 15 16:32 isw_dxifadxaii_Volume0p2
	brw-rw---- 1 root disk 253,  5 Feb 15 16:31 isw_dxifadxaii_Volume0p3
	brw-rw---- 1 root disk 253,  6 Feb 15 16:53 isw_dxifadxaii_Volume0p5

The device minor numbers (second column, after "253,") correspond to:

	[root@fishie tmp]# ls /sys/block/dm-?/slaves

	/sys/block/dm-0/slaves:
	sda  sdd

	/sys/block/dm-1/slaves:
	sdb  sdc

	/sys/block/dm-2/slaves:
	dm-0  dm-1

	/sys/block/dm-3/slaves:
	dm-2

	/sys/block/dm-4/slaves:
	dm-2

	/sys/block/dm-5/slaves:
	dm-2

	/sys/block/dm-6/slaves:
	dm-2


...which shows quite nicely how the disks are grouped.


finally, after the rebuild is done, everything is well again:

	[root@fishie tmp]# dmraid -s -s -vv
	NOTICE: /dev/sda: asr     discovering
	NOTICE: /dev/sda: ddf1    discovering
	NOTICE: /dev/sda: hpt37x  discovering
	NOTICE: /dev/sda: hpt45x  discovering
	NOTICE: /dev/sda: isw     discovering
	NOTICE: /dev/sda: isw metadata discovered
	NOTICE: /dev/sda: jmicron discovering
	NOTICE: /dev/sda: lsi     discovering
	NOTICE: /dev/sda: nvidia  discovering
	NOTICE: /dev/sda: pdc     discovering
	NOTICE: /dev/sda: sil     discovering
	NOTICE: /dev/sda: via     discovering
	NOTICE: /dev/sdb: asr     discovering
	NOTICE: /dev/sdb: ddf1    discovering
	NOTICE: /dev/sdb: hpt37x  discovering
	NOTICE: /dev/sdb: hpt45x  discovering
	NOTICE: /dev/sdb: isw     discovering
	NOTICE: /dev/sdb: isw metadata discovered
	NOTICE: /dev/sdb: jmicron discovering
	NOTICE: /dev/sdb: lsi     discovering
	NOTICE: /dev/sdb: nvidia  discovering
	NOTICE: /dev/sdb: pdc     discovering
	NOTICE: /dev/sdb: sil     discovering
	NOTICE: /dev/sdb: via     discovering
	NOTICE: /dev/sdc: asr     discovering
	NOTICE: /dev/sdc: ddf1    discovering
	NOTICE: /dev/sdc: hpt37x  discovering
	NOTICE: /dev/sdc: hpt45x  discovering
	NOTICE: /dev/sdc: isw     discovering
	NOTICE: /dev/sdc: isw metadata discovered
	NOTICE: /dev/sdc: jmicron discovering
	NOTICE: /dev/sdc: lsi     discovering
	NOTICE: /dev/sdc: nvidia  discovering
	NOTICE: /dev/sdc: pdc     discovering
	NOTICE: /dev/sdc: sil     discovering
	NOTICE: /dev/sdc: via     discovering
	NOTICE: /dev/sdd: asr     discovering
	NOTICE: /dev/sdd: ddf1    discovering
	NOTICE: /dev/sdd: hpt37x  discovering
	NOTICE: /dev/sdd: hpt45x  discovering
	NOTICE: /dev/sdd: isw     discovering
	NOTICE: /dev/sdd: isw metadata discovered
	NOTICE: /dev/sdd: jmicron discovering
	NOTICE: /dev/sdd: lsi     discovering
	NOTICE: /dev/sdd: nvidia  discovering
	NOTICE: /dev/sdd: pdc     discovering
	NOTICE: /dev/sdd: sil     discovering
	NOTICE: /dev/sdd: via     discovering
	NOTICE: added /dev/sda to RAID set "isw_dxifadxaii"
	NOTICE: added /dev/sdb to RAID set "isw_dxifadxaii"
	NOTICE: added /dev/sdc to RAID set "isw_dxifadxaii"
	NOTICE: added /dev/sdd to RAID set "isw_dxifadxaii"
	*** Group superset isw_dxifadxaii
	--> Active Superset
	name   : isw_dxifadxaii_Volume0
	size   : 1953535744
	stride : 128
	type   : raid01
	status : ok
	subsets: 2
	devs   : 4
	spares : 0
	--> Active Subset
	name   : isw_dxifadxaii_Volume0-0
	size   : 976767872
	stride : 128
	type   : mirror
	status : ok
	subsets: 0
	devs   : 2
	spares : 0
	--> Active Subset
	name   : isw_dxifadxaii_Volume0-1
	size   : 976767872
	stride : 128
	type   : mirror
	status : ok
	subsets: 0
	devs   : 2
	spares : 0

Note that after next boot, the device letters will be different, and slot 0
will be /dev/sda again. The RAID BIOS and dmraid driver should have no problem
with this, hopefully.

================================================================================