Header Shadow Image


HTPC / Backup Home Server Solution using Linux

Pages: 1 2 3 4 5

 

CONCLUSION

The conclusion from the ZFS test is quite clear.  No matter what we tried, read and write speeds were only a very small fraction what could be achieved from XFS.  The main culprit in this from what I see is the zfs-daemon process, which is single threaded and is expected to handle all workload to/from the FS.   This is evident in the %CPU reported to be used by the jobs and is clearly the limiting factor in ZFS Fuse unfortunately.  So at this time, I cannot recommend ZFS on Linux for this sort of setup.  Perhaps other numbers would work but we haven't been able to come up with a better combination.
 

PROBLEMS and RESOLUTIONS

I suppose one could say that the ideal setup is without it's hickups.  In practice setting something like this up tends to be more involving.  Then again, perhaps it's better to encounter issues early to weed out problems and properly test drive a setup before the same problems happen in production.  Here I've documented the errors and issues encountered while creating the array.  These likely won't represent all the errors one would encounter however. 

#1

PROBLEM:

You have just started to build an array but realized you didn't set the –bitmap=internal on it.  So at this point I wanted to stop the array and recreate it with the –bitmap=internal option instead of separately, shortly after starting to build one but got.

# mdadm –grow /dev/md0 –bitmap=internal
mdadm: Cannot add bitmap while array is resyncing or reshaping etc.
mdadm: failed to set internal bitmap.

#

NOTE: For this error, the goal was to destroy the test array and recreate it with an internal bitmap.  For a live storage array, the solution for this problem would not be suitable however components of this solution would be appropriate for stopping and reassembling one back up.  This also simulates a mistake a would be operator could make.  See the AVAILABILITY TEST # 7 & 8 above as well.

 

SOLUTION 1:

(Solution 1 steps below assume we're destroying the array.  You have been warned.  Skip to SOLUTION 2 for a graceful way.)

So I tried to stop the array and start fresh.  But:

# mdadm –stop /dev/md0
mdadm: Cannot get exclusive access to /dev/md0:Perhaps a running process, mounted filesystem or active volume group?
#

(WARNING: This will destroy data.  This step could also cause a problem ) Then I tried to fail the array:

mdadm /dev/md0 –fail /dev/sda –remove /dev/sda
mdadm /dev/md0 –fail /dev/sdb –remove /dev/sdb
mdadm /dev/md0 –fail /dev/sdc –remove /dev/sdc
mdadm /dev/md0 –fail /dev/sdd –remove /dev/sdd
mdadm /dev/md0 –fail /dev/sdf –remove /dev/sdf
mdadm /dev/md0 –fail /dev/sdg –remove /dev/sdg

But that resulted in the same stop message.  So I tried:

echo 1 > /sys/block/sd{a,b,c,d,f,g}/device/delete

to spin them down but still:

# mdadm –stop /dev/md0
mdadm: Cannot get exclusive access to /dev/md0:Perhaps a running process, mounted filesystem or active volume group?
#

Then I realize that the LVM is still running with the LV's and VG's still on /dev/md0:

# lsof|grep -i md0
md0_raid6  1095      root  cwd       DIR              253,0      4096          2 /
md0_raid6  1095      root  rtd       DIR              253,0      4096          2 /
md0_raid6  1095      root  txt   unknown                                         /proc/1095/exe
# ps -ef|grep -i 1095
root      1095     2  2 Apr11 ?        02:02:17 [md0_raid6]
root     23611 14803  0 17:15 pts/0    00:00:00 grep -i 1095
#

My storage VG is still online:

# lvm vgs
  VG          #PV #LV #SN Attr   VSize   VFree 
  MBPCStorage   1   1   0 wz–n-   3.64t 931.70g
#

So then I try to remove the LV first, (which is not a good thing btw as we'll see once all the disks have been failed in the RAID array):

# lvm lvremove /dev/MBPCStorage/MBPCBackup
  /dev/md0: read failed after 0 of 4096 at 0: Input/output error
  /dev/MBPCStorage/MBPCBackup: read failed after 0 of 4096 at 2000003465216: Input/output error
  /dev/MBPCStorage/MBPCBackup: read failed after 0 of 4096 at 2000003522560: Input/output error
  /dev/MBPCStorage/MBPCBackup: read failed after 0 of 4096 at 0: Input/output error
  /dev/MBPCStorage/MBPCBackup: read failed after 0 of 4096 at 4096: Input/output error
  /dev/md0: read failed after 0 of 4096 at 4000814661632: Input/output error
  /dev/md0: read failed after 0 of 4096 at 4000814718976: Input/output error
  /dev/md0: read failed after 0 of 4096 at 4096: Input/output error
  Volume group "MBPCStorage" not found
  Skipping volume group MBPCStorage
#

But no luck.  Turns out I need to use dmsetup which is a low level logical volume manipulation tool (And the reason for the Input/Output errors above are because I failed ALL the RAID6 array disks above before effectively stopping the MBPCBackup VG).  A typical time when one would use this is when the RAID6 experiences 3+ disk failure and is effectively destroyed before the VG can be unmounted:

# dmsetup info -c /dev/MBPCStorage/MBPCBackup
Name                   Maj Min Stat Open Targ Event  UUID                                                               
MBPCStorage-MBPCBackup 253   6 L–w    0    1      0 LVM-FqixjaGjMc8xYcudSRD43Y6wMpLXXR92JBHkxjnwZ31axUUoIpkWWr6qh9Boal58
# dmsetup remove MBPCStorage-MBPCBackup
# dmsetup info -c /dev/MBPCStorage/MBPCBackup

Device /dev/MBPCStorage/MBPCBackup not found
Command failed
#

And this time I can stop the array:

# mdadm –stop /dev/md0
mdadm: stopped /dev/md0
#

 

SOLUTION 2 (Stops the array gracefully):

See the availability test # 7 and 8 above for the graceful procedure steps.

Hope you found this useful.  Feel free to leave a reply and let us know your favorite storage solution DIY or not.

Cheers!
TK

Pages: 1 2 3 4 5


     
  Copyright © 2003 - 2013 Tom Kacperski (microdevsys.com). All rights reserved.

Creative Commons License
This work is licensed under a Creative Commons Attribution 3.0 Unported License