RAID5のストレージを交換。

# cat /var/log/messages | grep sdc
Sep 4 00:02:38 serv3 smartd[3868]: Device: /dev/sdc [SAT], 4 Currently unreadable (pending) sectors
Sep 4 00:02:38 serv3 smartd[3868]: Device: /dev/sdc [SAT], 2 Offline uncorrectable sectors
Sep 4 00:32:38 serv3 smartd[3868]: Device: /dev/sdc [SAT], 4 Currently unreadable (pending) sectors
Sep 4 00:32:38 serv3 smartd[3868]: Device: /dev/sdc [SAT], 2 Offline uncorrectable sectors
Sep 4 01:02:38 serv3 smartd[3868]: Device: /dev/sdc [SAT], 4 Currently unreadable (pending) sectors
Sep 4 01:02:38 serv3 smartd[3868]: Device: /dev/sdc [SAT], 2 Offline uncorrectable sectors
Sep 4 01:32:39 serv3 smartd[3868]: Device: /dev/sdc [SAT], 4 Currently unreadable (pending) sectors
Sep 4 01:32:39 serv3 smartd[3868]: Device: /dev/sdc [SAT], 2 Offline uncorrectable sectors
Sep 4 02:02:39 serv3 smartd[3868]: Device: /dev/sdc [SAT], 4 Currently unreadable (pending) sectors
Sep 4 02:02:39 serv3 smartd[3868]: Device: /dev/sdc [SAT], 2 Offline uncorrectable sectors
Sep 4 02:32:39 serv3 smartd[3868]: Device: /dev/sdc [SAT], 4 Currently unreadable (pending) sectors
Sep 4 02:32:39 serv3 smartd[3868]: Device: /dev/sdc [SAT], 2 Offline uncorrectable sectors
Sep 4 03:02:39 serv3 smartd[3868]: Device: /dev/sdc [SAT], 4 Currently unreadable (pending) sectors
Sep 4 03:02:39 serv3 smartd[3868]: Device: /dev/sdc [SAT], 2 Offline uncorrectable sectors
Sep 4 03:32:38 serv3 smartd[3868]: Device: /dev/sdc [SAT], 4 Currently unreadable (pending) sectors
Sep 4 03:32:38 serv3 smartd[3868]: Device: /dev/sdc [SAT], 2 Offline uncorrectable sectors
Sep 4 04:02:38 serv3 smartd[3868]: Device: /dev/sdc [SAT], 4 Currently unreadable (pending) sectors
Sep 4 04:02:38 serv3 smartd[3868]: Device: /dev/sdc [SAT], 2 Offline uncorrectable sectors
Sep 4 04:32:38 serv3 smartd[3868]: Device: /dev/sdc [SAT], 4 Currently unreadable (pending) sectors
Sep 4 04:32:38 serv3 smartd[3868]: Device: /dev/sdc [SAT], 2 Offline uncorrectable sectors
Sep 4 05:02:38 serv3 smartd[3868]: Device: /dev/sdc [SAT], 4 Currently unreadable (pending) sectors
Sep 4 05:02:38 serv3 smartd[3868]: Device: /dev/sdc [SAT], 2 Offline uncorrectable sectors
Sep 4 05:32:38 serv3 smartd[3868]: Device: /dev/sdc [SAT], 4 Currently unreadable (pending) sectors
Sep 4 05:32:38 serv3 smartd[3868]: Device: /dev/sdc [SAT], 2 Offline uncorrectable sectors
Sep 4 06:02:38 serv3 smartd[3868]: Device: /dev/sdc [SAT], 4 Currently unreadable (pending) sectors
Sep 4 06:02:38 serv3 smartd[3868]: Device: /dev/sdc [SAT], 2 Offline uncorrectable sectors
Sep 4 06:32:38 serv3 smartd[3868]: Device: /dev/sdc [SAT], 4 Currently unreadable (pending) sectors
Sep 4 06:32:38 serv3 smartd[3868]: Device: /dev/sdc [SAT], 2 Offline uncorrectable sectors
Sep 4 07:02:38 serv3 smartd[3868]: Device: /dev/sdc [SAT], 4 Currently unreadable (pending) sectors
Sep 4 07:02:38 serv3 smartd[3868]: Device: /dev/sdc [SAT], 2 Offline uncorrectable sectors
Sep 4 07:32:38 serv3 smartd[3868]: Device: /dev/sdc [SAT], 4 Currently unreadable (pending) sectors
Sep 4 07:32:38 serv3 smartd[3868]: Device: /dev/sdc [SAT], 2 Offline uncorrectable sectors
Sep 4 08:02:38 serv3 smartd[3868]: Device: /dev/sdc [SAT], 4 Currently unreadable (pending) sectors
Sep 4 08:02:38 serv3 smartd[3868]: Device: /dev/sdc [SAT], 2 Offline uncorrectable sectors
Sep 4 08:32:38 serv3 smartd[3868]: Device: /dev/sdc [SAT], 4 Currently unreadable (pending) sectors
Sep 4 08:32:38 serv3 smartd[3868]: Device: /dev/sdc [SAT], 2 Offline uncorrectable sectors
Sep 4 09:02:38 serv3 smartd[3868]: Device: /dev/sdc [SAT], 4 Currently unreadable (pending) sectors
Sep 4 09:02:38 serv3 smartd[3868]: Device: /dev/sdc [SAT], 2 Offline uncorrectable sectors
Sep 4 09:32:38 serv3 smartd[3868]: Device: /dev/sdc [SAT], 4 Currently unreadable (pending) sectors
Sep 4 09:32:38 serv3 smartd[3868]: Device: /dev/sdc [SAT], 2 Offline uncorrectable sectors
Sep 4 10:02:38 serv3 smartd[3868]: Device: /dev/sdc [SAT], 4 Currently unreadable (pending) sectors
Sep 4 10:02:38 serv3 smartd[3868]: Device: /dev/sdc [SAT], 2 Offline uncorrectable sectors
Sep 4 10:32:38 serv3 smartd[3868]: Device: /dev/sdc [SAT], 4 Currently unreadable (pending) sectors
Sep 4 10:32:38 serv3 smartd[3868]: Device: /dev/sdc [SAT], 2 Offline uncorrectable sectors
Sep 4 11:02:38 serv3 smartd[3868]: Device: /dev/sdc [SAT], 4 Currently unreadable (pending) sectors
Sep 4 11:02:38 serv3 smartd[3868]: Device: /dev/sdc [SAT], 2 Offline uncorrectable sectors
Sep 4 11:32:39 serv3 smartd[3868]: Device: /dev/sdc [SAT], 4 Currently unreadable (pending) sectors
Sep 4 11:32:39 serv3 smartd[3868]: Device: /dev/sdc [SAT], 2 Offline uncorrectable sectors
Sep 4 12:02:38 serv3 smartd[3868]: Device: /dev/sdc [SAT], 4 Currently unreadable (pending) sectors
Sep 4 12:02:38 serv3 smartd[3868]: Device: /dev/sdc [SAT], 2 Offline uncorrectable sectors
Sep 4 12:32:38 serv3 smartd[3868]: Device: /dev/sdc [SAT], 4 Currently unreadable (pending) sectors
Sep 4 12:32:38 serv3 smartd[3868]: Device: /dev/sdc [SAT], 2 Offline uncorrectable sectors
Sep 4 13:02:38 serv3 smartd[3868]: Device: /dev/sdc [SAT], 4 Currently unreadable (pending) sectors
Sep 4 13:02:38 serv3 smartd[3868]: Device: /dev/sdc [SAT], 2 Offline uncorrectable sectors
Sep 4 13:32:38 serv3 smartd[3868]: Device: /dev/sdc [SAT], 4 Currently unreadable (pending) sectors
Sep 4 13:32:38 serv3 smartd[3868]: Device: /dev/sdc [SAT], 2 Offline uncorrectable sectors
Sep 4 14:02:38 serv3 smartd[3868]: Device: /dev/sdc [SAT], 4 Currently unreadable (pending) sectors
Sep 4 14:02:38 serv3 smartd[3868]: Device: /dev/sdc [SAT], 2 Offline uncorrectable sectors
Sep 4 14:32:38 serv3 smartd[3868]: Device: /dev/sdc [SAT], 4 Currently unreadable (pending) sectors
Sep 4 14:32:38 serv3 smartd[3868]: Device: /dev/sdc [SAT], 2 Offline uncorrectable sectors
Sep 4 15:02:38 serv3 smartd[3868]: Device: /dev/sdc [SAT], 4 Currently unreadable (pending) sectors
Sep 4 15:02:38 serv3 smartd[3868]: Device: /dev/sdc [SAT], 2 Offline uncorrectable sectors
Sep 4 15:32:38 serv3 smartd[3868]: Device: /dev/sdc [SAT], 4 Currently unreadable (pending) sectors
Sep 4 15:32:38 serv3 smartd[3868]: Device: /dev/sdc [SAT], 2 Offline uncorrectable sectors
Sep 4 16:02:38 serv3 smartd[3868]: Device: /dev/sdc [SAT], 4 Currently unreadable (pending) sectors
Sep 4 16:02:38 serv3 smartd[3868]: Device: /dev/sdc [SAT], 2 Offline uncorrectable sectors
Sep 4 16:32:38 serv3 smartd[3868]: Device: /dev/sdc [SAT], 4 Currently unreadable (pending) sectors
Sep 4 16:32:38 serv3 smartd[3868]: Device: /dev/sdc [SAT], 2 Offline uncorrectable sectors
Sep 4 17:02:38 serv3 smartd[3868]: Device: /dev/sdc [SAT], 4 Currently unreadable (pending) sectors
Sep 4 17:32:38 serv3 smartd[3868]: Device: /dev/sdc [SAT], 4 Currently unreadable (pending) sectors
Sep 4 17:32:38 serv3 smartd[3868]: Device: /dev/sdc [SAT], 2 Offline uncorrectable sectors
Sep 4 18:02:38 serv3 smartd[3868]: Device: /dev/sdc [SAT], 4 Currently unreadable (pending) sectors
Sep 4 18:02:38 serv3 smartd[3868]: Device: /dev/sdc [SAT], 2 Offline uncorrectable sectors
Sep 4 18:32:38 serv3 smartd[3868]: Device: /dev/sdc [SAT], 4 Currently unreadable (pending) sectors
Sep 4 18:32:38 serv3 smartd[3868]: Device: /dev/sdc [SAT], 2 Offline uncorrectable sectors
Sep 4 19:02:38 serv3 smartd[3868]: Device: /dev/sdc [SAT], 4 Currently unreadable (pending) sectors
Sep 4 19:02:38 serv3 smartd[3868]: Device: /dev/sdc [SAT], 2 Offline uncorrectable sectors
Sep 4 19:32:38 serv3 smartd[3868]: Device: /dev/sdc [SAT], 4 Currently unreadable (pending) sectors
Sep 4 19:32:38 serv3 smartd[3868]: Device: /dev/sdc [SAT], 2 Offline uncorrectable sectors
Sep 4 20:02:38 serv3 smartd[3868]: Device: /dev/sdc [SAT], 4 Currently unreadable (pending) sectors
Sep 4 20:02:38 serv3 smartd[3868]: Device: /dev/sdc [SAT], 2 Offline uncorrectable sectors
Sep 4 20:32:38 serv3 smartd[3868]: Device: /dev/sdc [SAT], 4 Currently unreadable (pending) sectors
Sep 4 20:32:38 serv3 smartd[3868]: Device: /dev/sdc [SAT], 2 Offline uncorrectable sectors
Sep 4 21:02:38 serv3 smartd[3868]: Device: /dev/sdc [SAT], 4 Currently unreadable (pending) sectors
Sep 4 21:02:38 serv3 smartd[3868]: Device: /dev/sdc [SAT], 2 Offline uncorrectable sectors

# umount /mnt/raid5
# cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md126 : active raid5 sdd1[3] sde1[4] sdc1[1]
3906764800 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/3] [UUU]
bitmap: 0/15 pages [0KB], 65536KB chunk

md127 : active raid1 sdb1[2] sda1[3]
2930134016 blocks super 1.2 [2/2] [UU]
bitmap: 1/22 pages [4KB], 65536KB chunk

unused devices: 

# mdadm --manage /dev/md126 --fail /dev/sdc1
mdadm: set /dev/sdc1 faulty in /dev/md126

# cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md126 : active raid5 sdd1[3] sde1[4] sdc1[1](F)
3906764800 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/2] [U_U]
bitmap: 0/15 pages [0KB], 65536KB chunk

md127 : active raid1 sdb1[2] sda1[3]
2930134016 blocks super 1.2 [2/2] [UU]
bitmap: 2/22 pages [8KB], 65536KB chunk

unused devices: 

# mdadm --manage /dev/md126 --remove /dev/sdc1
mdadm: hot removed /dev/sdc1 from /dev/md126

# cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md126 : active raid5 sdd1[3] sde1[4]
3906764800 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/2] [U_U]
bitmap: 0/15 pages [0KB], 65536KB chunk

md127 : active raid1 sdb1[2] sda1[3]
2930134016 blocks super 1.2 [2/2] [UU]
bitmap: 2/22 pages [8KB], 65536KB chunk

unused devices: 

# parted /dev/sdc print
モデル: ATA WDC WD20EZRX-00D (scsi)
ディスク /dev/sdc: 2000GB
セクタサイズ (論理/物理): 512B/4096B
パーティションテーブル: gpt
ディスクフラグ:

番号 開始 終了 サイズ ファイルシステム 名前 フラグ
1 1049kB 2000GB 2000GB ext4 raid

 

# shutdown -h
Terminated

# parted /dev/sdc print
エラー: /dev/sdc: ディスクラベルが認識できません。
モデル: ATA WDC WD30EZRZ-00G (scsi)
ディスク /dev/sdc: 3001GB
セクタサイズ (論理/物理): 512B/4096B
パーティションテーブル: unknown
ディスクフラグ:

# cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md126 : active raid5 sdd1[3] sde1[4]
3906764800 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/2] [U_U]
bitmap: 0/15 pages [0KB], 65536KB chunk

md127 : active raid1 sda1[3] sdb1[2]
2930134016 blocks super 1.2 [2/2] [UU]
bitmap: 0/22 pages [0KB], 65536KB chunk

unused devices: 

# smartctl -a /dev/sdc
=== START OF INFORMATION SECTION ===
Model Family: Western Digital Blue
Device Model: WDC WD30EZRZ-00GXCB0
Serial Number: WD-WCC7********
LU WWN Device Id: 5 0014ee 20f7ecfbc
Firmware Version: 80.00A80
User Capacity: 3,000,592,982,016 bytes [3.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5400 rpm
Form Factor: 3.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-3 T13/2161-D revision 5
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Wed Sep 4 21:46:59 2019 JST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

# parted /dev/sdc
GNU Parted 3.1
/dev/sdc を使用
GNU Parted へようこそ! コマンド一覧を見るには 'help' と入力してください。
(parted) mklabel gpt
(parted) mkpart primary ext4 0% 100%
(parted) set 1 raid on
(parted) print
モデル: ATA WDC WD30EZRZ-00G (scsi)
ディスク /dev/sdc: 3001GB
セクタサイズ (論理/物理): 512B/4096B
パーティションテーブル: gpt
ディスクフラグ:

番号 開始 終了 サイズ ファイルシステム 名前 フラグ
1 1049kB 3001GB 3001GB primary raid

(parted) quit
通知: 必要であれば /etc/fstab を更新するのを忘れないようにしてください。

# mdadm --add /dev/md126 /dev/sdc1
mdadm: added /dev/sdc1
# cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md126 : active raid5 sdc1[5] sdd1[3] sde1[4]
3906764800 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/2] [U_U]
[>....................] recovery = 0.0% (1656532/1953382400) finish=294.5min speed=110435K/sec
bitmap: 0/15 pages [0KB], 65536KB chunk

md127 : active raid1 sda1[3] sdb1[2]
2930134016 blocks super 1.2 [2/2] [UU]
bitmap: 0/22 pages [0KB], 65536KB chunk

unused devices: