Header Shadow Image


HTPC / Backup Home Server Solution using Linux

Pages: 1 2 3 4 5

Overview:

HTPC / Home Backup: MDADM, RAID6, LVM, XFS, CIFS and NFS

In this post we'll take a look at setting up a 4TB storage, HTPC, and active backup using basic components that can be bought from the local PC store and freely available software.   Here I set out to try and consolidate my storage to something central running RHEL Linux (Or a clone, such as the Scientific Linux that's freely available) that could also serve as a makeshift entertainment center in replacement of my current one.  In other words, upgrade my HTPC and turn it into an HTPC+B, if you will, with the following statistics:

  • 4TB usable storage (6TB HDD – 2 HDD for RAID6).
  • XFS Filesystem
  • 317MB/s WRITES.
  • 300 MB/s READS.
  • A Wireless N card + 1GIG ETH in a bonding configuration. (Redundancy in case of failure)
  • Scientific Linux 6.1 (RHEL 6.1).

When it's all said and done then shared with the rest of the network , including a new Nexus 7 running Android through AndSMB, and tweaked for quick data transfers it should make a nice subtle addition to the decor as well.  ( And yes, it is also very very quiet ):

HTPC+B Entertainment System

 

So here is what a goal would be to achieve with this central storage server:

  1. HTPC: Have it function as a media center.
  2. Backup. (Single drives backed up to a RAID 6 setup within the same box)
  3. File Server (NFS, CIFS, Mountable from my other windows systems etc)
  4. Flash storage center for the most critical files.
  5. (Optional) Authentication server using something like kerberos.

There are a few good posts out there that would suggest or argue against using RAID, but I still need a backup.  The other thing that I wanted to do is to find out how reliable this thing could be.  (As we'll see, it turned out to be quite an eye opener.)  A good read as well is the Triple Parity RAID and Beyond article.

So why not use a store baught backup solution?

That seamed like a good idea sometime back if it were not an episode a colleague of mine had with one of those.  Based on the noises and symptoms he describes as one having while he couldn't access his data was enough for me.  The other reason is recoverability and supportability options one has if trouble does occur.  So this brings us to the most real question here:

What is a backup in the true sense given today's technology?

This is a loaded question and there's no right or wrong answer.  With today's estimates that neither RAID5 nor RAID6 would be enough in the upcoming years (see the link above), the word backup would appear to start to take on the meaning:  Storage media that would be available when needed to restore a copy from rather then storage that would hold a copy of my data. At first glance there doesn't appear to be a difference yet the first statement doesn't suggest a promise that the backup would be there at all.  YMMV or YOMV.

So what does that mean for RAID 6?  Let's illustrate with an example given the following to determine the likelihood of an unrecoverable data failure:

  • A home laptop (contains one HDD)
  • The HTPC+B setup (contains 6 HDD in a RAID 6 configuration)

If my laptop is being backed up to my HTPC+B server, the point at which one would loose data is when both the laptop HDD and 3 of the RAID 6 disks failed at the same time.  So in essence, a 4 disk failure at the same time which is highly unlikely.  So what this means is that, which would be more likely, when my laptop disk fails, my HTPC+B would likely still be around for me to restore from.  Likewise, if my backup fails, my PC would still be running along fine long enough to fix up my backup outage.  In either case, I would still have my data.

The flipside of the argument would suggest that the more hardware in a configuration, the more maintenance it would require over the same time frame.

Setup Summary:

A quick note.  The hardware used here is a combination of old and new and where price was right vs other options.  Other hardware should work just as well but your mileage may vary and my pocket book isn't that deep at all:

  • LS ULTRA 600W ATX Power Supply (Reused from another PC)
  • GA-890XA-UD3 Motherboard ( 333 – SATA 3, USB Power 3.0, USB 3.0).  6x SATA 3 for our software RAID 6.
  • Seagate 1TB Green Barracuda  HDD x 6  ( ST31000520AS )
  • ( Optional – Failed drive replacement ) Barracuda® Green SATA 6Gb/s 1.5TB Hard Drive ( ST1500DL003 )
  • AMD RADEON HD 6450 1GB
  • TP-LINK TL-WN821N (V2) Wireless USB stick.
  • AMD Athlon(tm) 5200 Dual-Core Processor
  • A Western Digital Caviar Green drive ( WD15EADS ) for the root FS.

For this setup, the motherboard is the most powerful item I got.  Lot's of room for upgrades here as a result.  I wasn't looking to break the bank and at about $50 per HDD in Aug of 2011, it made perfect sense, before a certain flood in Thailand.  I hope cheap doesn't transfer to the hardware but only time will tell.  All hooked up, it looked like this:

 

NOTE: The makeshift fan under the video card PCI card.  Turned out the heatsink of the AMD RADEON was getting too hot even though it is sold as a fanless GPU.  )

Once the hardware is hooked up, it's time for the software.  The steps below are the base commands I needed after a rather lengthy, trial and error setup of the HTPC+B:

 

ITEM #: DESCRIPTION COMMAND(S)
1 Enable HOT-SWAP in BIOS.

The below will enable HOT PLUG in BIOS.  Enter BIOS and enable the following:

OnChip SATA Type = AHCI
OnChip SATA Port 4/5 Type = As SATA Type

Once this is set, removing and readding SDD devices has the following effect:

DISCONNECT MESSAGES:
Oct 23 10:48:01 mbpc kernel: ata6: exception Emask 0×10 SAct 0×0 SErr 0×90202 action 0xe frozen
Oct 23 10:48:01 mbpc kernel: ata6: irq_stat 0×00400000, PHY RDY changed
Oct 23 10:48:01 mbpc kernel: ata6: SError: { RecovComm Persist PHYRdyChg 10B8B }
Oct 23 10:48:01 mbpc kernel: ata6: hard resetting link
Oct 23 10:48:02 mbpc kernel: ata6: SATA link down (SStatus 0 SControl 300)
Oct 23 10:48:07 mbpc kernel: ata6: hard resetting link
Oct 23 10:48:07 mbpc kernel: ata6: SATA link down (SStatus 0 SControl 300)
Oct 23 10:48:07 mbpc kernel: ata6: limiting SATA link speed to 1.5 Gbps
Oct 23 10:48:12 mbpc kernel: ata6: hard resetting link
Oct 23 10:48:13 mbpc kernel: ata6: SATA link down (SStatus 0 SControl 310)
Oct 23 10:48:13 mbpc kernel: ata6.00: disabled
Oct 23 10:48:13 mbpc kernel: ata6: EH complete
Oct 23 10:48:13 mbpc kernel: ata6.00: detaching (SCSI 5:0:0:0)
Oct 23 10:48:13 mbpc kernel: sd 5:0:0:0: [sda] Synchronizing SCSI cache
Oct 23 10:48:13 mbpc kernel: sd 5:0:0:0: [sda] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
Oct 23 10:48:13 mbpc kernel: sd 5:0:0:0: [sda] Stopping disk
Oct 23 10:48:13 mbpc kernel: sd 5:0:0:0: [sda] START_STOP FAILED
Oct 23 10:48:13 mbpc kernel: sd 5:0:0:0: [sda] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK

CONNECT:
Oct 23 10:49:10 mbpc kernel: ata6: exception Emask 0×10 SAct 0×0 SErr 0x40d0002 action 0xe frozen
Oct 23 10:49:10 mbpc kernel: ata6: irq_stat 0×00000040, connection status changed
Oct 23 10:49:10 mbpc kernel: ata6: SError: { RecovComm PHYRdyChg CommWake 10B8B DevExch }
Oct 23 10:49:10 mbpc kernel: ata6: hard resetting link
Oct 23 10:49:10 mbpc kernel: ata6: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
Oct 23 10:49:10 mbpc kernel: ata6.00: ATA-8: ST31000520AS, CC32, max UDMA/133
Oct 23 10:49:10 mbpc kernel: ata6.00: 1953525168 sectors, multi 16: LBA48 NCQ (depth 31/32)
Oct 23 10:49:10 mbpc kernel: ata6.00: configured for UDMA/133
Oct 23 10:49:10 mbpc kernel: ata6: EH complete
Oct 23 10:49:10 mbpc kernel: scsi 5:0:0:0: Direct-Access     ATA      ST31000520AS     CC32 PQ: 0 ANSI: 5
Oct 23 10:49:10 mbpc kernel: sd 5:0:0:0: Attached scsi generic sg0 type 0
Oct 23 10:49:10 mbpc kernel: sd 5:0:0:0: [sda] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB)
Oct 23 10:49:10 mbpc kernel: sd 5:0:0:0: [sda] Write Protect is off
Oct 23 10:49:10 mbpc kernel: sd 5:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Oct 23 10:49:10 mbpc kernel: sda: unknown partition table
Oct 23 10:49:10 mbpc kernel: sd 5:0:0:0: [sda] Attached SCSI disk

 

2 (Optional) Setup UDEV rules for the newly installed SDD devices. Follow the link LINUX: Persistent naming of SATA devices through UDEV rules to potentially setup UDEV naming rules for the RAID 6 devices.  This will help with the rest of the steps and uniquely name the RAID6 devices we'll be working with and using in various commands.
3 Start another session to monitor /var/log/messages before we create the array using mdadm.

Before we create the array, we want to be monitoring the log files.  The result of the creation in the next step is:

# tail -f /var/log/messages
Jan 30 00:22:17 mbpc kernel: md: bind<sdf>
Jan 30 00:22:17 mbpc kernel: md: bind<sde>
Jan 30 00:22:17 mbpc kernel: md: bind<sdd>
Jan 30 00:22:17 mbpc kernel: md: bind<sdc>
Jan 30 00:22:17 mbpc kernel: md: bind<sdb>
Jan 30 00:22:17 mbpc kernel: md: bind<sda>
Jan 30 00:22:17 mbpc kernel: async_tx: api initialized (async)
Jan 30 00:22:17 mbpc kernel: xor: automatically using best checksumming function: generic_sse
Jan 30 00:22:17 mbpc kernel:   generic_sse:  3260.000 MB/sec
Jan 30 00:22:17 mbpc kernel: xor: using function: generic_sse (3260.000 MB/sec)
Jan 30 00:22:17 mbpc kernel: raid6: int64x1    687 MB/s
Jan 30 00:22:17 mbpc kernel: raid6: int64x2    960 MB/s
Jan 30 00:22:17 mbpc kernel: raid6: int64x4    718 MB/s
Jan 30 00:22:17 mbpc kernel: raid6: int64x8    625 MB/s
Jan 30 00:22:17 mbpc kernel: raid6: sse2x1    1210 MB/s
Jan 30 00:22:17 mbpc kernel: raid6: sse2x2    2035 MB/s
Jan 30 00:22:17 mbpc kernel: raid6: sse2x4    2355 MB/s
Jan 30 00:22:17 mbpc kernel: raid6: using algorithm sse2x4 (2355 MB/s)
Jan 30 00:22:17 mbpc kernel: md: raid6 personality registered for level 6
Jan 30 00:22:17 mbpc kernel: md: raid5 personality registered for level 5
Jan 30 00:22:17 mbpc kernel: md: raid4 personality registered for level 4
Jan 30 00:22:17 mbpc kernel: bio: create slab <bio-1> at 1
Jan 30 00:22:17 mbpc kernel: md/raid:md0: not clean — starting background reconstruction
Jan 30 00:22:17 mbpc kernel: md/raid:md0: device sda operational as raid disk 5
Jan 30 00:22:17 mbpc kernel: md/raid:md0: device sdb operational as raid disk 4
Jan 30 00:22:17 mbpc kernel: md/raid:md0: device sdc operational as raid disk 3
Jan 30 00:22:17 mbpc kernel: md/raid:md0: device sdd operational as raid disk 2
Jan 30 00:22:17 mbpc kernel: md/raid:md0: device sde operational as raid disk 1
Jan 30 00:22:17 mbpc kernel: md/raid:md0: device sdf operational as raid disk 0
Jan 30 00:22:17 mbpc kernel: md/raid:md0: allocated 6386kB
Jan 30 00:22:17 mbpc kernel: md/raid:md0: raid level 6 active with 6 out of 6 devices, algorithm 2
Jan 30 00:22:17 mbpc kernel: md0: detected capacity change from 0 to 4000814792704
Jan 30 00:22:17 mbpc kernel: md: resync of RAID array md0
Jan 30 00:22:17 mbpc kernel: md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
Jan 30 00:22:17 mbpc kernel: md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for resync.
Jan 30 00:22:17 mbpc kernel: md: using 128k window, over a total of 976761424 blocks.
Jan 30 00:22:17 mbpc kernel: md0: unknown partition table

4 Create the array:

The steps in order are as follows.  Of course for your RAID6 type, # of disks and disk identifiers, you'll need to tweak these lines to suit your setup:

# mdadm –create –verbose  /dev/md0 –level=raid6 –chunk=64K –auto=p –raid-devices=6  –spare-devices=0  –bitmap=internal /dev/rsd{a,b,c,d,e,f}

We'll line up the chunk sizes to the XFS sizes below: 64K for improved performance.  The –bitmap=internal will allow for very very fast resync's when new disks are added to replace failed ones.  Instead of 5 – 8 hours, the resyncs literally took only a few minutes during testing. Check the array using:

# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid6 sda[5] sdb[4] sdd[2] sdf[3] sdc[0] sde[1]
      3907045632 blocks super 1.2 level 6, 64k chunk, algorithm 2 [6/6] [UUUUUU]
      bitmap: 0/8 pages [0KB], 65536KB chunk

unused devices: <none>
#

In our case, the array should be building now:

# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid6 sda[5] sdb[4] sdc[3] sdd[2] sde[1] sdf[0]
      3907045696 blocks super 1.2 level 6, 16k chunk, algorithm 2 [6/6] [UUUUUU]
      [>....................]  resync =  1.5% (15464592/976761424) finish=183.1min speed=87454K/sec
     
unused devices: <none>

#

(Optional) # mdadm –grow /dev/md0 –bitmap=internal                # To add an internal bitmap later.

NOTES: To stop or start the array, issue one of the following commands:
STOP:  mdadm –stop /dev/md0
START: mdadm –assemble –scan

IMPORTANT: To spin down the SDD platters and power off the individual disks, to simulate disks being taken out or prior to taking out the disks, issue the following:

echo 1 > /sys/block/sda/device/delete
echo 1 > /sys/block/sdb/device/delete
echo 1 > /sys/block/sdc/device/delete
echo 1 > /sys/block/sdd/device/delete
echo 1 > /sys/block/sdf/device/delete
echo 1 > /sys/block/sdg/device/delete

To rescan the system SATA connectors to detect and readd the removed disks (That were removed by the above method for example):

echo "0 0 0" >/sys/class/scsi_host/host0/scan
echo "0 0 0" >/sys/class/scsi_host/host1/scan
echo "0 0 0" >/sys/class/scsi_host/host2/scan
echo "0 0 0" >/sys/class/scsi_host/host3/scan
echo "0 0 0" >/sys/class/scsi_host/host4/scan
echo "0 0 0" >/sys/class/scsi_host/host5/scan

We will use the above methods during testing of the array prior to usage.  Once built fully, our array should be marked as active:

# mdadm –detail /dev/raidmd0
/dev/raidmd0:
        Version : 1.2
  Creation Time : Mon Mar 26 00:06:24 2012
     Raid Level : raid6
     Array Size : 3907045632 (3726.05 GiB 4000.81 GB)
  Used Dev Size : 976761408 (931.51 GiB 1000.20 GB)
   Raid Devices : 6
  Total Devices : 6
    Persistence : Superblock is persistent

  Intent Bitmap : Internal

    Update Time : Mon Apr  2 00:59:01 2012
          State : active
 Active Devices : 6
Working Devices : 6
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 64K

           Name : mbpc:0  (local to host mbpc)
           UUID : 2f36ac48:5e3e4c54:72177c53:bea3e41e
         Events : 3213

    Number   Major   Minor   RaidDevice State
       0       8       32        0      active sync   /dev/sdc
       1       8       64        1      active sync   /dev/sde
       2       8       48        2      active sync   /dev/sdd
       3       8       80        3      active sync   /dev/sdf
       4       8       16        4      active sync   /dev/sdb
       5       8        0        5      active sync   /dev/sda
#

5 Create the PV, VG and LV's.

In our case, we'll only create a 2TB LV out of the 4TB we have.  We will extend this in the future as need arises (These and following steps can be done while the array is building above).

# lvm pvcreate /dev/raidmd0
# lvm vgcreate MBPCStorage /dev/raidmd0
# lvm lvcreate -L3906254360S -n MBPCBackup MBPCStorage           
# 2TB.  Leaving the last 2TB for possible expansion and LVM / XFS resizing.

6 Create the XFS filesystem and mount the partition.

Create the XFS filesystem in this manner (The selection of size=64 and agcount=64 are the result of testing using various values that we did.  For this configuration, 64 for both options for these 1TB X6 SDD's was optimal):

# mkfs.xfs -l size=64m -d agcount=64 -i attr=2,maxpct=5 -L MBPCBackupx /dev/MBPCStorage/MBPCBackup -f


# mount -t xfs -o logbufs=8,noatime,nodiratime,allocsize=512m /dev/MBPCStorage/MBPCBackup MBPCBackupx/

At this point the RAID6 array space is accessible and writes to it are possible however more tweaking is in order as per further steps below. 

7

Edit the /etc/fstab file to add the mount point and details so mounting can occur at boot time.

 

The mount points and mount parameters can be specified in this manner in the /etc/fstab file:

# /etc/fstab
.
.
.
/dev/MBPCStorage/MBPCBackup            /mnt/MBPCBackupx/    xfs    logbufs=8,noatime,nodiratime,allocsize=512m 0 0
#

8 For XFS filesystems and RAID6, it's rather dangerous, according to readings, to have the write_cache enabled.  There is the potential of metadata corruption on power failures.  Disable write_cache on the disks participating in the RAID6 XFS array.

Add the following entries to the /etc/rc.local file to take effect on each reboot:

# /etc/rc.local file
hdparm -W 0 /dev/rsda;
hdparm -W 0 /dev/rsdb;
hdparm -W 0 /dev/rsdc;
hdparm -W 0 /dev/rsdd;
hdparm -W 0 /dev/rsde;
hdparm -W 0 /dev/rsdf;

An alternative is to add these lines to the hdparm.conf file, though it did not work in this scenario and Scientific Linux 6.1. 

# cat /etc/hdparm.conf
/dev/rsda {
    write_cache = off
}
/dev/rsdb {
    write_cache = off
}
/dev/rsdc {
    write_cache = off
}
/dev/rsdd {
    write_cache = off
}
/dev/rsde {
    write_cache = off
}
/dev/rsdf {
    write_cache = off
}
#

Confirm that the write cache is disabled:

# for rdn in $(ls /dev/rsd*); do hdparm -I $rdn|grep -i "write cache"; done
            Write cache
            Write cache
            Write cache
            Write cache
            Write cache
            Write cache
#

NOTE: A * would be seen before each entry if the write_cache were enabled.

9 Setup the MDADM config file.

Setup the MDADM config file by issuing the following:

# mdadm –detail –scan >> /etc/mdadm.conf

Verify the config file:

# cat /etc/mdadm.conf
ARRAY /dev/md/0 metadata=1.2 name=mbpc:0 UUID=2f36ac48:5e3e4c54:72177c53:bea3e41e
#

10 GET the block device default optimization parameters for the RAID6 array (md0), individual disks and filesystem.

Get the default parameters for the devices.  In this case, for the hardware selected, the default detected are to the right behind the # (If your RAID6 array is NOT md0, substitute accordingly):

# READ
cat /sys/block/md0/md/stripe_cache_size                                        # 256
blockdev –getra $(echo $(ls -al /dev/rsd*|awk '{ print "/dev/"$NF }'))                        # 256
blockdev –getra /dev/md0                                            # 1024
blockdev –getra /dev/MBPCStorage/MBPCBackup                                    # 1024
for mskb in $(ls -al /dev/rsd*|awk '{ print $NF }'); do cat /sys/block/$mskb/queue/max_sectors_kb; done        # 512
for qdepth in $(ls -al /dev/rsd*|awk '{ print $NF }'); do cat /sys/block/$qdepth/device/queue_depth; done    # 31

 

Record the parameters in case you'll want to put them back.

11 SET the block device default optimization parameters for the RAID6 array (md0), individual disks and filesystem.

Again, as with the write_cache parameters, add these to the /etc/rc.local file on your system so they take effect on each reboot, restart and shutdown:

# SET (rc.local file)
echo 16384 > /sys/block/md0/md/stripe_cache_size
blockdev –setra 32768 $(echo $(ls -al /dev/rsd*|awk '{ print "/dev/"$NF }'))
blockdev –setra 32768 /dev/md0
blockdev –setra 32768 /dev/MBPCStorage/MBPCBackup
for mskb in $(ls -al /dev/rsd*|awk '{ print $NF }'); do echo 8192 > /sys/block/$mskb/queue/max_sectors_kb; done
for qdepth in $(ls -al /dev/rsd*|awk '{ print $NF }'); do echo 31 > /sys/block/$qdepth/device/queue_depth; done

11 Test using bonnie++ and copying files from another drive to the targer XFS RAID 6 array you've just built.

Test the array using bonnie++ (If not available, can download using yum install bonnie++):

# cd /mnt/MBPCBackupx
# bonnie++ -u root:root -d /mnt/MBPCBackupx/ -s 8000m -m socrates -n 16:100000:16:64

Bonnie++ test results (best of the best) and iostat -xkd 10 disk I/O metric tool (While NOT rebuilding or synchronizing the array):

Intelligent Writes (best so far):

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda               3.90 19729.80    1.10  200.60    20.00 79543.45   788.93     9.58   47.54   4.27  86.13
sdc               2.80 19719.90    0.90  208.40    14.80 78797.45   753.10    12.61   60.07   4.12  86.22
sdd               4.10 19744.90    0.50  183.10    18.40 79303.05   864.07    12.02   65.66   4.92  90.40
sdb               3.40 19714.10    0.80  213.60    16.80 79073.05   737.78    11.19   51.84   3.91  83.85
sde               4.40 19761.50    0.90  166.40    21.20 78847.05   942.84    14.34   85.33   5.47  91.51
sdf               3.40 19768.40    1.00  160.30    17.60 78366.25   971.90    17.71  108.80   5.98  96.50
sdg               0.00     9.10   54.60    1.80  1208.80    42.80    44.38     0.67   11.79   2.82  15.91
dm-0              0.00     0.00   54.60   10.70  1208.80    42.80    38.33     1.24   19.06   2.44  15.91
dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-2              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
md0               0.00     0.00    0.00 5034.70     0.00 317492.50   126.12     0.00    0.00   0.00   0.00
dm-3              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-4              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-5              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-6              0.00     0.00    0.00 5033.90     0.00 317447.70   126.12  1192.91  238.58   0.20 100.00

Intelligent ReWrites:

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda             220.70  9655.70   77.20   72.30 17930.00 40553.55   782.39    27.64  186.63   6.44  96.22
sdb             227.00  9628.50   83.90  101.10 18039.60 39713.55   624.36    11.55   63.16   4.45  82.25
sdc             212.50  9642.80   80.00   83.90 17700.80 39390.35   696.66    15.31   93.13   5.38  88.22
sde               0.00   179.60   34.70    7.70   930.40   748.80    79.21     0.86   20.34   4.08  17.30
dm-0              0.00     0.00   34.70    1.90   930.40     7.60    51.26     0.70   19.00   3.75  13.72
dm-1              0.00     0.00    0.00  185.30     0.00   741.20     8.00     6.88   37.11   0.49   9.17
dm-2              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-3              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-4              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-5              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdd             221.40  9646.40   82.20   81.40 18012.40 38993.95   696.90    13.70   83.39   5.36  87.68
sdf             218.50  9643.60   82.70   81.30 18012.40 39027.95   695.61    14.18   86.28   5.41  88.65
sdg             230.70  9650.70   81.70   79.00 17785.20 39425.95   712.02    17.83  110.54   5.63  90.53
md0               0.00     0.00 1633.80 2460.90 104126.00 155635.20   126.88     0.00    0.00   0.00   0.00
dm-6              0.00     0.00 1633.80 2460.80 104126.00 155628.80   126.88  1257.45  311.60   0.24  96.33

Intelligent Reads:

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda             493.81     0.00  247.55    0.60 47351.45     1.20   381.64    12.14   48.94   3.76  93.38
sdc             496.60     0.00  249.35    0.60 47153.25     1.20   377.31     9.56   38.09   3.10  77.44
sdb             494.91     0.00  251.45    0.60 47236.36     1.20   374.83     8.76   34.67   2.88  72.49
sdd             496.00     0.00  249.95    0.70 47184.82     1.60   376.51     8.99   35.73   2.98  74.69
sde             491.71     0.00  249.25    0.70 47312.69     1.60   378.59     9.08   36.25   3.02  75.57
sdf             488.21     0.00  252.35    0.70 47236.36     1.60   373.35     9.61   37.86   3.10  78.44
sdg              27.37    34.17   57.84    2.70  1628.37   146.65    58.64     0.70   11.54   3.48  21.09
dm-0              0.00     0.00   86.11   36.66  1628.37   146.65    28.91     1.69   13.79   1.72  21.10
dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-2              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
md0               0.00     0.00 4467.23    0.10 284796.80     0.05   127.50     0.00    0.00   0.00   0.00
dm-3              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-4              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-5              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-6              0.00     0.00 4467.23    0.10 284796.80     0.05   127.50   171.69   38.35   0.21  95.26

Bonnie++ test results (While in active non-degraded status and NOT doing a resync):
 

Version  1.96       ——Sequential Output—— –Sequential Input- –Random-
Concurrency   1     -Per Chr- –Block– -Rewrite- -Per Chr- –Block– –Seeks–
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
socrates      8000M   450  94 290382  37 127824  24  2696  99 313146  26 423.1  24
Latency             25833us     229ms     227ms    9175us     127ms   87525us
Version  1.96       ——Sequential Create—— ——–Random Create——–
socrates            -Create– –Read— -Delete– -Create– –Read— -Delete–
files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
    16:100000:16/64    71   1 +++++ +++  7879  36    71   1 +++++ +++  6917  38
Latency             28029ms    4218us     697us   40137ms     128us     194us
1.96,1.96,socrates,1,1332770295,8000M,,450,94,290382,37,127824,24,2696,99,313146,
26,423.1,24,16,100000,16,,64,71,1,+++++,+++,7879,36,71,1,+++++,+++,6917,38,25833us,
229ms,227ms,9175us,127ms,87525us,28029ms,4218us,697us,40137ms,128us,194us

 

12 Check integrity and get statistics on the configuration.

Read static configuration data and check file integrity using the below test and commands:

# xfs_bmap -v io.1.dat
# xfs_bmap -v io.2.dat
# mdadm –detail /dev/md127
# cat /proc/mdstat
# xfs_info /mnt/MBPCBackupx/

Check and test disk status of each individual RAID6 member using:

# for ddn in $(ls /dev/sd{a,b,c,d,e,f,g}); do DDN=$(smartctl -A $ddn|grep -i Current_Pending_Sector); echo $ddn": $DDN"; done
/dev/sda: 197 Current_Pending_Sector  0×0012   100   100   000    Old_age   Always       -       0
/dev/sdb: 197 Current_Pending_Sector  0×0012   100   100   000    Old_age   Always       -       0
/dev/sdc: 197 Current_Pending_Sector  0×0012   100   100   000    Old_age   Always       -       0
/dev/sdd: 197 Current_Pending_Sector  0×0012   100   100   000    Old_age   Always       -       0
/dev/sde: 197 Current_Pending_Sector  0×0012   100   100   000    Old_age   Always       -       0
/dev/sdf: 197 Current_Pending_Sector  0×0012   100   100   000    Old_age   Always       -       0
/dev/sdg: 197 Current_Pending_Sector  0×0032   200   200   000    Old_age   Always       -       0
#

Use the following to get health statistics on individual SDD's:

# smartctl -A /dev/sda
smartctl 5.39.1 2010-01-28 r3054 [x86_64-redhat-linux-gnu] (local build)
Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   114   100   006    Pre-fail  Always       -       75358912
  3 Spin_Up_Time            0×0003   094   094   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0×0032   100   100   020    Old_age   Always       -       7
  5 Reallocated_Sector_Ct   0×0033   100   100   036    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   100   253   030    Pre-fail  Always       -       198931
  9 Power_On_Hours          0×0032   100   100   000    Old_age   Always       -       44
 10 Spin_Retry_Count        0×0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0×0032   100   100   020    Old_age   Always       -       6
183 Runtime_Bad_Block       0×0032   100   100   000    Old_age   Always       -       0
184 End-to-End_Error        0×0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0×0032   100   100   000    Old_age   Always       -       0
188 Command_Timeout         0×0032   100   100   000    Old_age   Always       -       0
189 High_Fly_Writes         0x003a   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0×0022   063   058   045    Old_age   Always       -       37 (Lifetime Min/Max 25/42)
191 G-Sense_Error_Rate      0×0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0×0032   100   100   000    Old_age   Always       -       1
193 Load_Cycle_Count        0×0032   100   100   000    Old_age   Always       -       7
194 Temperature_Celsius     0×0022   037   042   000    Old_age   Always       -       37 (0 25 0 0)
195 Hardware_ECC_Recovered  0x001a   024   024   000    Old_age   Always       -       75358912
197 Current_Pending_Sector  0×0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0×0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
240 Head_Flying_Hours       0×0000   100   253   000    Old_age   Offline      -       68053756805161
241 Total_LBAs_Written      0×0000   100   253   000    Old_age   Offline      -       56966118
242 Total_LBAs_Read         0×0000   100   253   000    Old_age   Offline      -       3844511940

#

Get the static config data of the XFS FS:

# xfs_info /mnt/MBPCBackupx/
meta-data=/dev/mapper/MBPCStorage-MBPCBackup isize=256    agcount=64, agsize=7629408 blks
         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=488282112, imaxpct=5
         =                       sunit=16     swidth=64 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal               bsize=4096   blocks=16384, version=2
         =                       sectsz=512   sunit=16 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
You have new mail in /var/spool/mail/root
#

Your array should now be ready.  However, before one is to put any data on the new storage, some brutal testing is in order.  We need to know that the array can withstand some hard hits and still tick before you can trust your files to it.

Pages: 1 2 3 4 5

4 Responses to “HTPC / Backup Home Server Solution using Linux”

  1. Hi,

    Great post. You don’t need to specify the parameters when creating the XFS file system, see http://xfs.org/index.php/XFS_FAQ#Q:_I_want_to_tune_my_XFS_filesystems_for_.3Csomething.3E and http://www.spinics.net/lists/raid/msg38074.html . Of course, YMMV.

    Did you run those benchmarks while the array was resyncing?

  2. Hey Mathias,

    Thanks for posting. Just added the testing numbers so feel free to have a look and judge yourself.

    > logbsize and delaylog
    I ran another test with logbsize=128k (couldn’t find anything for delaylog in my mkfs.xfs man page so I’m not sure if that’ll do anything). Little to no difference in this case on first glance. Watch out for the results at some point for a closer look.

    One consideration here is that eventually I would grow the LVM and XFS to fill up to 4TB. I’ll be doing this soon Potentially in the future, I may try to grow this array as well to something well over 8TB (Yet to see how to do that). I’m not sure if XFS would auto-adjust in those cases for optimal values for those capacities and the link didn’t touch on that topic.

    All in all, I can still run tests on this thing recreating the FS if I need to so feel free to suggest numbers you’d be interested to see. I might leave this topic open for a week or two to see if I can think of anything else or if I’m missing anything. For my setup, having anything > 125MB/s is a bonus as the network is only 1GB/s with that theoretical max.

    Cheers!
    TK

  3. [...] could be done safely enough like this guy did and with RAID6 as well with SSD type R/W’s no less. Your size would be limited to the size of the [...]

  4. Thank you for posting this blog.  I was getting desparate.  I could not figure out why I could not stop the RAID1 device.  Even from Ubuntu Rescue Remix.  The LVM group was being assembled from the failed raid.  I removed the volume group and was finally able to gain exclusive access to the array to stop it, put in the new disk and rebuild the array.
     
    Nice job.
    Best,
    Dave.

Leave a Reply

 

     
  Copyright © 2003 - 2013 Tom Kacperski (microdevsys.com). All rights reserved.

Creative Commons License
This work is licensed under a Creative Commons Attribution 3.0 Unported License