Here are some attempts at making sense of what the sector numbers in mdadm
error messages refer to. It's not entirely successful; let me know if you know
of any real documentation of this.


first, we have the disk layer error messages:

[ 9942.044556] end_request: I/O error, dev sda, sector 2484336
[ 9967.532508] end_request: I/O error, dev sda, sector 2484336
[ 9985.730611] end_request: I/O error, dev sda, sector 2484354

Notice that there are two distinct sector addresses that are failing.

These sector numbers line up with the actual LBA block addresses, i.e. sectors
relative to start of whole physical disk. you can verify this by looking at the
SMART log:

smartctl -a /dev/sda
[...]
SMART Error Log Version: 1
ATA Error Count: 67 (device log contains only the most recent five errors)
[...]
Error 67 occurred at disk power-on lifetime: 33060 hours (1377 days + 12 hours)
(bunch more error detail header text...)
  40 51 00 82 e8 25 00  Error: UNC at LBA = 0x0025e882 = 2484354  <- matches dmesg
[...]


Then we have the errors from the md raid layer:

[ 9942.044597] md/raid1:md124: sda3: rescheduling sector 2338720
[ 9942.044634] md/raid1:md124: sda3: rescheduling sector 2338968
[ 9967.591358] md/raid1:md124: read error corrected (8 sectors at 2355312 on sda3)
[ 9985.767163] md/raid1:md124: read error corrected (8 sectors at 2355328 on sda3)
[ 9986.292742] md/raid1:md124: redirecting sector 2338720 to other mirror: sda3
[ 9988.719406] md/raid1:md124: redirecting sector 2338968 to other mirror: sda3

Notice that there are four distinct numbers here. The ones in parens, "(8
sectors at ... on sda3)" refer to the offset relative to the start of the
partition sda3. The other ones, "rescheduling sector" and "redirecting sector"
refer to the offset relative to ...what?

You can check this by checking that the dump of that sector address on sda3
matches the dump of the LBA sector on sda:

(using the first address, LBA 2484336)

hexdump -C -n512 -s2484336b /dev/sda    <- 2484336 blocks from start of disk
hexdump -C -n512 -s2355312b /dev/sda3   <- equals 2355312 blocks from start of partition

This one doesn't work:
hexdump -C -n512 -s2338720b /dev/md124  <- equals 2338720 blocks from start of md124
but: 2338720 - 16384 from start of sda3 == 2338720 from start of md124

Notes:
- the -n512 and 'b' suffix to the address means this is a 512-byte sector
  drive. if you have a 4K sector drive, the addresses in the dmesg log may
  refer to 4K sector offsets and you might need to specify -n4096 and specify
  the argument to -s as something like, e.g. $((2484336*4096)) [the "$(())"
  syntax denotes a shell arithmetic expression].
- if these sectors are actually bad, you'll get read errors in the first two commands.
- there's a good chance they will be readable, because mdadm tries to re-write
  the data by copying it from another mirror... I think?