PROBLEM:

You have just started to build an array but realized you didn't set the –bitmap=internal on it.  So at this point I wanted to stop the array and recreate it with the –bitmap=internal option instead of separately, shortly after starting to build one but got.

# mdadm –grow /dev/md0 –bitmap=internal
mdadm: Cannot add bitmap while array is resyncing or reshaping etc.
mdadm: failed to set internal bitmap.
#

NOTE: For this error, the goal was to destroy the test array and recreate it with an internal bitmap.  For a live storage array, the solution for this problem would not be suitable however components of this solution would be appropriate for stopping and reassembling one back up.  This also simulates a mistake a would be operator could make.  See the AVAILABILITY TEST # 7 & 8 above as well.

SOLUTION 1:

(Solution 1 steps below assume we're destroying the array.  You have been warned.  Skip to SOLUTION 2 for a graceful way.)

So I tried to stop the array and start fresh.  But:

# mdadm –stop /dev/md0
mdadm: Cannot get exclusive access to /dev/md0:Perhaps a running process, mounted filesystem or active volume group?
#

(WARNING: This will destroy data.  This step could also cause a problem ) Then I tried to fail the array:

mdadm /dev/md0 –fail /dev/sda –remove /dev/sda
mdadm /dev/md0 –fail /dev/sdb –remove /dev/sdb
mdadm /dev/md0 –fail /dev/sdc –remove /dev/sdc
mdadm /dev/md0 –fail /dev/sdd –remove /dev/sdd
mdadm /dev/md0 –fail /dev/sdf –remove /dev/sdf
mdadm /dev/md0 –fail /dev/sdg –remove /dev/sdg

But that resulted in the same stop message.  So I tried:

echo 1 > /sys/block/sd{a,b,c,d,f,g}/device/delete

to spin them down but still:

# mdadm –stop /dev/md0
mdadm: Cannot get exclusive access to /dev/md0:Perhaps a running process, mounted filesystem or active volume group?
#

Then I realize that the LVM is still running with the LV's and VG's still on /dev/md0:

# lsof|grep -i md0
md0_raid6  1095      root  cwd       DIR              253,0      4096          2 /
md0_raid6  1095      root  rtd       DIR              253,0      4096          2 /
md0_raid6  1095      root  txt   unknown                                         /proc/1095/exe
# ps -ef|grep -i 1095
root      1095     2  2 Apr11 ?        02:02:17 [md0_raid6]
root     23611 14803  0 17:15 pts/0    00:00:00 grep -i 1095
#

My storage VG is still online:

# lvm vgs
  VG          #PV #LV #SN Attr   VSize   VFree 
  MBPCStorage   1   1   0 wz–n-   3.64t 931.70g
#

So then I try to remove the LV first, (which is not a good thing btw as we'll see once all the disks have been failed in the RAID array):

# lvm lvremove /dev/MBPCStorage/MBPCBackup
  /dev/md0: read failed after 0 of 4096 at 0: Input/output error
  /dev/MBPCStorage/MBPCBackup: read failed after 0 of 4096 at 2000003465216: Input/output error
  /dev/MBPCStorage/MBPCBackup: read failed after 0 of 4096 at 2000003522560: Input/output error
  /dev/MBPCStorage/MBPCBackup: read failed after 0 of 4096 at 0: Input/output error
  /dev/MBPCStorage/MBPCBackup: read failed after 0 of 4096 at 4096: Input/output error
  /dev/md0: read failed after 0 of 4096 at 4000814661632: Input/output error
  /dev/md0: read failed after 0 of 4096 at 4000814718976: Input/output error
  /dev/md0: read failed after 0 of 4096 at 4096: Input/output error
  Volume group "MBPCStorage" not found
  Skipping volume group MBPCStorage
#

But no luck.  Turns out I need to use dmsetup which is a low level logical volume manipulation tool (And the reason for the Input/Output errors above are because I failed ALL the RAID6 array disks above before effectively stopping the MBPCBackup VG).  A typical time when one would use this is when the RAID6 experiences 3+ disk failure and is effectively destroyed before the VG can be unmounted:

# dmsetup info -c /dev/MBPCStorage/MBPCBackup
Name                   Maj Min Stat Open Targ Event  UUID                                                               
MBPCStorage-MBPCBackup 253   6 L–w    0    1      0 LVM-FqixjaGjMc8xYcudSRD43Y6wMpLXXR92JBHkxjnwZ31axUUoIpkWWr6qh9Boal58
# dmsetup remove MBPCStorage-MBPCBackup
# dmsetup info -c /dev/MBPCStorage/MBPCBackup
Device /dev/MBPCStorage/MBPCBackup not found
Command failed
#

And this time I can stop the array:

# mdadm –stop /dev/md0
mdadm: stopped /dev/md0
#

SOLUTION 2 (Stops the array gracefully):

See the availability test # 7 and 8 above for the graceful procedure steps.