Header Shadow Image


HTPC / Backup Home Server Solution using Linux

Pages: 1 2 3 4 5

PERFORMANCE TESTING

Now that we have our array, let's setup some test scenarios to test it's performance.  We have RAID6 then LVM on top of that then XFS on top of the LVM setup.  We already have some numbers from above on which we settled on (About 300MB/s for each R/W)  For these tests, we'll assume the following as creating a new RAID6 array each time is costly.  These items would be tested later alongside a constant XFS config:

MDADM:

  • –chunk-size=64k

BONNIE:

  • bonnie++ -u root:root -d /mnt/MBPCBackupx/ -s 8000m -m socrates -n 16:100000:16:64
  • (Optional: Quick) bonnie++ -u root:root -d /mnt/MBPCBackupx/ -s 4000m -m socrates -n 16:10000:16:64

Each alternate command represents the difference from the original setup in the summary for XFS.  For these tests, -/+ 5% would be considered statistical noise, variation or randomness and would be considered inconclusive to determine a better or worse result.  With each command we will revert further towards default settings.  Each subsequent test extend on the previous test until all commands are reverted to their defaults / basic forms:

TEST

ALTERNATIVE SETUP COMMAND + RESULTS:

bonnie++
iostat -xkd 10

1

ALTERNATIVE COMMAND:

mount -t xfs /dev/MBPCStorage/MBPCBackup MBPCBackupx/

IOSTAT:

Intelligent Writes:

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda               5.20 19297.80    1.50  185.10    26.80 77572.70   831.72     9.98   53.16   4.63  86.39
sdc               7.20 19287.30    1.50  195.50    34.80 77174.30   783.85    13.22   66.55   4.47  88.04
sdb               4.30 19271.90    0.70  210.60    20.00 77434.70   733.13    11.74   55.29   3.93  82.98
sdd               4.80 19289.00    1.20  195.60    24.00 78303.50   796.01    12.99   66.38   4.58  90.10
sde               7.10 19318.70    1.80  164.20    35.60 77358.30   932.46    15.49   93.15   5.58  92.69
sdf               5.40 19334.40    1.40  149.00    27.20 77870.70  1035.88    19.71  131.60   6.38  96.02
md0               0.00     0.00    0.00 4937.90     0.00 311456.00   126.15     0.00    0.00   0.00   0.00
dm-6              0.00     0.00    0.00 4937.90     0.00 311456.00   126.15  1196.34  243.40   0.20 100.00

Intelligent Reads:

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda             488.50     0.00  255.30    0.70 47902.00     1.45   374.25    12.09   47.38   3.64  93.08
sdc             491.00     0.00  259.10    0.70 47911.20     1.45   368.84     8.94   34.52   2.98  77.49
sdb             492.20     0.00  258.40    0.70 47950.00     1.45   370.14     8.46   32.81   2.85  73.89
sdd             490.90     0.00  259.40    0.80 47920.00     1.85   368.35     8.88   34.25   2.99  77.90
sde             485.00     0.00  258.20    0.80 47821.60     1.85   369.29     8.88   34.38   2.97  76.93
sdf             483.40     0.00  259.80    0.80 47859.60     1.85   367.32     9.22   35.49   3.06  79.71
md0               0.00     0.00 4466.60    0.10 284558.00     0.05   127.41     0.00    0.00   0.00   0.00
dm-6              0.00     0.00 4466.60    0.10 284558.00     0.05   127.41   165.24   37.12   0.21  95.38

BONNIE++ (1):

Version  1.96       ——Sequential Output—— –Sequential Input- –Random-
Concurrency   1     -Per Chr- –Block– -Rewrite- -Per Chr- –Block– –Seeks–
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
socrates      8000M   489  95 293621  38 132637  25  2290  90 323067  25 391.3  22
Latency             30605us     230ms     326ms   33651us     128ms     360ms
Version  1.96       ——Sequential Create—— ——–Random Create——–
socrates            -Create– –Read— -Delete– -Create– –Read— -Delete–
files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
    16:100000:16/64    71   0 +++++ +++  8378  38    68   0 +++++ +++  7296  40
Latency             33726ms     126us     201us   43328ms     140us     228us
1.96,1.96,socrates,1,1333399320,8000M,,489,95,293621,38,132637,25,2290,90,323067,25,391.
3,22,16,100000,16,,64,71,0,+++++,+++,8378,38,68,0,+++++,+++,7296,40,30605us,230ms,326ms,
33651us,128ms,360ms,33726ms,126us,201us,43328ms,140us,228us

BONNIE++ (2):

Version  1.96       ——Sequential Output—— –Sequential Input- –Random-
Concurrency   1     -Per Chr- –Block– -Rewrite- -Per Chr- –Block– –Seeks–
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
socrates      8000M   488  96 280658  36 123484  24  2142  85 310711  24 374.4  21
Latency             17704us     327ms     327ms   19081us     132ms     589ms
Version  1.96       ——Sequential Create—— ——–Random Create——–
socrates            -Create– –Read— -Delete– -Create– –Read— -Delete–
files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
    16:100000:16/64    70   0 +++++ +++  8194  37    67   0 +++++ +++  7165  39
Latency             38230ms     113us     397us   44328ms      92us     436us
1.96,1.96,socrates,1,1333397907,8000M,,488,96,280658,36,123484,24,2142,85,310711,
24,374.4,21,16,100000,16,,64,70,0,+++++,+++,8194,37,67,0,+++++,+++,7165,39,17704us,
327ms,327ms,19081us,132ms,589ms,38230ms,113us,397us,
44328ms,92us,436us

 

2

ALTERNATIVE COMMAND:

# mkfs.xfs -L MBPCBackupx /dev/MBPCStorage/MBPCBackup -f
meta-data=/dev/MBPCStorage/MBPCBackup isize=256    agcount=32, agsize=15258800 blks
         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=488281600, imaxpct=5
         =                       sunit=16     swidth=64 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal log           bsize=4096   blocks=238432, version=2
         =                       sectsz=512   sunit=16 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
#

 

IOSTAT:

Intelligent Writes:

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda               2.20 18679.10    1.00  180.10    12.80 75371.85   832.52    11.12   61.21   4.74  85.86
sdc               2.60 18669.30    0.70  190.30    13.20 75435.05   790.03    12.40   64.92   4.45  85.00
sdb               1.70 18656.80    0.50  201.20     8.80 75428.65   748.02    11.36   56.34   3.95  79.76
sdd               4.20 18663.70    1.20  194.60    20.90 75429.85   770.69    12.55   64.11   4.32  84.50
sde               6.20 18698.20    1.40  158.20    30.40 75422.65   945.53    14.14   88.58   5.56  88.79
sdf               2.20 18702.30    0.90  157.50    12.40 74483.20   940.60    16.09  100.19   5.75  91.15
md0               0.00     0.00    0.40 4885.30     0.90 308108.85   126.13     0.00    0.00   0.00   0.00
dm-6              0.00     0.00    0.40 4885.30     0.90 308115.25   126.13  1130.71  226.76   0.20  96.45

 

Intelligent Reads:

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda             478.70     1.50  239.00    1.20 45558.80     8.55   379.41    12.47   52.17   3.94  94.54
sdc             472.00     0.00  241.60    1.10 45485.20     2.15   374.84     9.74   40.19   3.23  78.29
sdb             471.00     0.00  240.90    2.90 45491.20     9.35   373.26     8.37   34.40   2.93  71.53
sdd             474.10     0.00  239.30    1.30 45446.40     2.95   377.80     8.70   36.26   3.07  73.95
sde             475.70     0.00  243.40    1.30 45452.80     2.95   371.52     8.55   35.08   3.04  74.46
sdf             474.70     0.00  242.10    2.70 45325.20     8.55   370.37     8.85   36.11   3.07  75.19
md0               0.00     0.00 4291.90    0.30 273612.80     7.20   127.50     0.00    0.00   0.00   0.00
dm-6              0.00     0.00 4291.90    0.20 273612.80     7.20   127.50   168.01   39.24   0.22  96.26

 

NOTABLES:

iostat await metric going ballistic:

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00   179.90    0.00   58.10     0.00  1050.40    36.16    44.12  800.78  17.21 100.00
sdc               0.00   171.20    0.00   65.20     0.00   967.60    29.68    16.83  259.36  13.41  87.44
sdb               0.00   168.50    0.00   68.70     0.00   991.20    28.86    13.25  198.22  10.97  75.38
sdd               0.00   190.90    0.00   43.50     0.00   975.20    44.84   140.54 3185.73  22.99 100.00
sde               0.00   170.70    0.00   66.80     0.00   998.00    29.88    11.88  186.03  10.91  72.88
sdf               0.00   190.20    0.00   44.10     0.00   992.80    45.02   130.85 2948.80  22.68 100.00
md0               0.00     0.00    0.00  218.10     0.00  7852.00    72.00     0.00    0.00   0.00   0.00
dm-6              0.00     0.00    0.00  218.20     0.00  7858.40    72.03  8414.55 8454053194093797.00   4.58 100.00

BONNIE++ (1):

Version  1.96       ——Sequential Output—— –Sequential Input- –Random-
Concurrency   1     -Per Chr- –Block– -Rewrite- -Per Chr- –Block– –Seeks–
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
socrates      8000M   483  95 296433  38 128335  24  2184  84 322603  25 399.4  23
Latency             20200us     227ms     326ms   16633us     129ms     276ms
Version  1.96       ——Sequential Create—— ——–Random Create——–
socrates            -Create– –Read— -Delete– -Create– –Read— -Delete–
files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
    16:100000:16/64    79   1 +++++ +++  8971  41    76   1 +++++ +++  8383  46
Latency             30627ms     533us     170us   42027ms      99us     213us
1.96,1.96,socrates,1,1333397740,8000M,,483,95,296433,38,128335,24,2184,84,322603,25,399.4,
23,16,100000,16,,64,79,1,+++++,+++,8971,41,76,1,+++++,+++,8383,46,20200us,227ms,326ms,
16633us,129ms,276ms,30627ms,533us,170us,42027ms,99us,213us

 

BONNIE++ (2):

Version  1.96       ——Sequential Output—— –Sequential Input- –Random-
Concurrency   1     -Per Chr- –Block– -Rewrite- -Per Chr- –Block– –Seeks–
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
socrates      8000M   490  95 277994  35 119279  23  2079  81 304914  25 401.3  20
Latency             33137us     233ms     228ms     121ms     122ms     401ms
Version  1.96       ——Sequential Create—— ——–Random Create——–
socrates            -Create– –Read— -Delete– -Create– –Read— -Delete–
files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
    16:100000:16/64    77   1 +++++ +++  9243  42    73   0 +++++ +++  7274  40
Latency             31526ms     102us     158us   43727ms      98us     221us
1.96,1.96,socrates,1,1333399367,8000M,,490,95,277994,35,119279,23,2079,81,304914,25,401.
3,20,16,100000,16,,64,77,1,+++++,+++,9243,42,73,0,+++++,+++,7274,40,33137us,233ms,228ms,

121ms,122ms,401ms,31526ms,102us,158us,43727ms,98us,221us

 

3

ALTERNATIVE COMMAND:

echo 256 > /sys/block/md0/md/stripe_cache_size
blockdev –setra 256 $(echo $(ls -al /dev/rsd*|awk '{ print "/dev/"$NF }'))
blockdev –setra 1024 /dev/md0
blockdev –setra 1024 /dev/MBPCStorage/MBPCBackup
for mskb in $(ls -al /dev/rsd*|awk '{ print $NF }'); do echo 512 > /sys/block/$mskb/queue/max_sectors_kb; done
for qdepth in $(ls -al /dev/rsd*|awk '{ print $NF }'); do echo 31 > /sys/block/$qdepth/device/queue_depth; done

 

IOSTAT:

Intelligent Writes:

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda              12.80  9791.50    2.00  141.20    59.20 39643.60   554.51     1.63   11.33   5.43  77.70
sdc               9.80  9772.60    1.60  143.30    40.00 39640.80   547.70     1.73   11.90   5.43  78.70
sdb              13.10  9774.10    1.80  141.20    59.60 39669.60   555.65     1.64   11.47   5.35  76.51
sdd               7.20  9773.00    1.20  142.20    33.60 39708.00   554.28     1.67   11.65   5.38  77.11
sde               6.80  9773.80    1.50  141.10    33.20 39643.60   556.48     1.68   11.79   5.41  77.09
sdf              10.20  9774.00    1.50  140.30    46.80 39640.80   559.77     1.71   12.01   5.58  79.17
md0               0.00     0.00    0.00 2515.20     0.00 158643.20   126.15     0.00    0.00   0.00   0.00
dm-6              0.00     0.00    0.00 2515.20     0.00 158643.20   126.15    60.06   23.85   0.40  99.99

Intelligent Reads:


Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda               5.60     0.00  964.30    0.70 61838.00     1.45   128.16     0.61    0.63   0.43  41.35
sdc              80.60     0.00 1118.50    0.70 61831.20     1.45   110.49     0.62    0.55   0.42  47.12
sdb              62.70     0.10 1136.90    0.80 61798.00     2.25   108.64     0.53    0.47   0.36  41.41
sdd              92.10     0.10 1108.20    0.80 61841.60     2.25   111.53     0.63    0.57   0.42  46.91
sde              19.90     0.10  949.40    0.80 61848.80     2.25   130.19     0.71    0.75   0.51  48.69
sdf              26.80     0.00  942.10    0.70 61818.40     1.45   131.14     0.97    1.03   0.63  59.24
md0               0.00     0.00 6506.00    0.10 370941.60     0.80   114.03     0.00    0.00   0.00   0.00
dm-6              0.00     0.00 6506.00    0.10 370941.60     0.80   114.03     4.50    0.69   0.15  96.76

 

BONNIE++(1):

Version  1.96       ——Sequential Output—— –Sequential Input- –Random-
Concurrency   1     -Per Chr- –Block– -Rewrite- -Per Chr- –Block– –Seeks–
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
socrates      8000M   486  95 148258  28 72908  22  2199  86 379404  51 411.3  21
Latency             17967us     228ms     185ms   35547us   51719us     385ms
Version  1.96       ——Sequential Create—— ——–Random Create——–
socrates            -Create– –Read— -Delete– -Create– –Read— -Delete–
files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
    16:100000:16/64    43   0 +++++ +++  7566  34    42   0 +++++ +++  6701  36
Latency              6650ms     272us     168us    8276ms     106us     194us
1.96,1.96,socrates,1,1333397055,8000M,,486,95,148258,28,72908,22,2199,86,379404,
51,411.3,21,16,100000,16,,64,43,0,+++++,+++,7566,34,42,0,+++++,+++,6701,36,17967us,
228ms,185ms,35547us,51719us,385ms,6650ms,272us,168us,8276ms,106us,194us

 

BONNIE++(2):

Version  1.96       ——Sequential Output—— –Sequential Input- –Random-
Concurrency   1     -Per Chr- –Block– -Rewrite- -Per Chr- –Block– –Seeks–
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
socrates      8000M   485  95 145075  27 71573  22  2239  87 369472  55 401.3  22
Latency             22517us     226ms     362ms   30212us     107ms     279ms
Version  1.96       ——Sequential Create—— ——–Random Create——–
socrates            -Create– –Read— -Delete– -Create– –Read— -Delete–
files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
    16:100000:16/64    42   0 +++++ +++  7508  34    41   0 +++++ +++  6637  36
Latency              7638ms     130us     176us    8339ms     137us     182us
1.96,1.96,socrates,1,1333427981,8000M,,485,95,145075,27,71573,22,2239,87,369472,55,401.3,
22,16,100000,16,,64,42,0,+++++,+++,7508,34,41,0,+++++,+++,6637,36,22517us,226ms,362ms,
30212us,107ms,279ms,7638ms,130us,176us,8339ms,137us,182us

 

NOTABLES: Writes were degraded to about 1/2 of their former value.
NOTABLES: Reads were up by approx 50MB/s.

3.1

Test #3 was very interesting.  So let's expand on it:

ALTERNATIVE COMMAND:

We'll put everything back with the exception of stripe_cache_size which we'll set to match for all the individual components:

echo 32768 > /sys/block/md0/md/stripe_cache_size
blockdev –setra 32768 $(echo $(ls -al /dev/rsd*|awk '{ print "/dev/"$NF }'))
blockdev –setra 32768 /dev/md0
blockdev –setra 32768 /dev/MBPCStorage/MBPCBackup
for mskb in $(ls -al /dev/rsd*|awk '{ print $NF }'); do echo 8192 > /sys/block/$mskb/queue/max_sectors_kb; done
for qdepth in $(ls -al /dev/rsd*|awk '{ print $NF }'); do echo 31 > /sys/block/$qdepth/device/queue_depth; done

IOSTAT:

Intelligent Writes:

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda             142.38 12806.91   15.83  128.16   617.23 50336.47   707.75    41.01  271.73   6.58  94.68
sdc             172.85 12729.46   16.63  162.32   759.12 49745.29   564.43    30.91  165.18   5.19  92.93
sdb             170.44 12780.56   16.33  156.31   723.05 49380.96   580.43    35.64  193.64   5.35  92.33
sdd             192.28 12776.05   18.04  134.27   808.02 50432.67   672.87    43.75  272.42   6.27  95.48
sde             181.16 12765.73   19.14  129.16   806.41 49708.02   681.26    38.57  246.31   6.49  96.29
sdf             180.26 12818.14   17.33  113.23   758.32 50658.72   787.63    43.87  323.79   7.60  99.27
md0               0.00     0.00    0.00 3434.87     0.00 204659.77   119.17     0.00    0.00   0.00   0.00
dm-6              0.00     0.00    0.00 3434.87     0.00 204659.77   119.17  1578.39  444.59   0.29 100.02

Intelligent Reads:

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda             492.30     0.00  244.90    0.70 47040.40     1.45   383.08    12.70   51.65   3.89  95.52
sdc             491.90     0.00  251.10    0.70 47091.60     1.45   374.05     9.79   38.79   3.24  81.47
sdb             494.10     0.00  249.30    0.70 47123.60     1.45   377.00     9.38   37.45   3.12  78.01
sdd             494.20     0.00  249.10    0.80 47139.20     1.85   377.28     9.81   39.20   3.25  81.29
sde             488.50     0.00  249.10    0.80 47148.80     1.85   377.36     9.81   39.22   3.23  80.78
sdf             488.70     0.00  249.00    0.80 47168.40     1.85   377.66     9.90   39.58   3.27  81.80
md0               0.00     0.00 4446.10    0.10 283443.20     0.05   127.50     0.00    0.00   0.00   0.00
dm-6              0.00     0.00 4446.10    0.10 283443.20     0.05   127.50   181.43   40.75   0.22  97.06

 

BONNIE++(1):

Version  1.96       ——Sequential Output—— –Sequential Input- –Random-
Concurrency   1     -Per Chr- –Block– -Rewrite- -Per Chr- –Block– –Seeks–
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
socrates      8000M   493  96 206133  28 111493  21  2244  87 297317  24 358.2  21
Latency             27737us     356ms     232ms   21396us     119ms     341ms
Version  1.96       ——Sequential Create—— ——–Random Create——–
socrates            -Create– –Read— -Delete– -Create– –Read— -Delete–
files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
    16:100000:16/64    75   0  4685  15 11215  53    75   0 +++++ +++  8131  45
Latency             66627ms   76193us     181us   66132ms     112us     193us
1.96,1.96,socrates,1,1333430597,8000M,,493,96,206133,28,111493,21,2244,87,297317,
24,358.2,21,16,100000,16,,64,75,0,4685,15,11215,53,75,0,+++++,+++,8131,45,27737us,
356ms,232ms,21396us,119ms,341ms,66627ms,76193us,181us,66132ms,112us,193us

 

BONNIE++(2):

Version  1.96       ——Sequential Output—— –Sequential Input- –Random-
Concurrency   1     -Per Chr- –Block– -Rewrite- -Per Chr- –Block– –Seeks–
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
socrates      8000M   488  96 174456  25 109809  21  2196  88 278683  21 374.1  22
Latency             24236us     453ms     327ms   19331us     162ms     133ms
Version  1.96       ——Sequential Create—— ——–Random Create——–
socrates            -Create– –Read— -Delete– -Create– –Read— -Delete–
files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
    16:100000:16/64    76   0 +++++ +++  8931  41    75   0 +++++ +++  7209  39
Latency             65126ms     118us     161us   63427ms     123us     180us
1.96,1.96,socrates,1,1333430451,8000M,,488,96,174456,25,109809,21,2196,88,278683,
21,374.1,22,16,100000,16,,64,76,0,+++++,+++,8931,41,75,0,+++++,+++,7209,39,24236us,
453ms,327ms,19331us,162ms,133ms,65126ms,118us,161us,63427ms,123us,180us

 

NOTABLES: Better then test 3.0 but still below our max achieved.

3.2

ALTERNATIVE COMMAND:

echo 16384 > /sys/block/md0/md/stripe_cache_size
blockdev –setra 16384 $(echo $(ls -al /dev/rsd*|awk '{ print "/dev/"$NF }'))
blockdev –setra 16384 /dev/md0
blockdev –setra 16384 /dev/MBPCStorage/MBPCBackup

for mskb in $(ls -al /dev/rsd*|awk '{ print $NF }'); do echo 8192 > /sys/block/$mskb/queue/max_sectors_kb; done
for qdepth in $(ls -al /dev/rsd*|awk '{ print $NF }'); do echo 31 > /sys/block/$qdepth/device/queue_depth; done

IOSTAT:

Intelligent Writes:

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda               7.50 18947.50    1.50  172.60    36.00 75815.50   871.36    12.00   68.38   5.00  87.01
sdc               5.20 18926.90    1.30  192.40    26.40 75673.50   781.62    13.73   70.46   4.56  88.37
sdb               2.90 18912.00    0.50  208.60    13.60 76045.10   727.49    11.01   52.47   3.93  82.17
sdd               2.60 18913.80    0.40  206.60    12.00 75798.30   732.47    13.06   63.05   4.15  85.86
sde               1.10 18968.60    0.80  149.60     7.60 74913.10   996.29    14.87   97.27   6.18  92.95
sdf               4.40 18962.30    1.00  156.20    21.60 75107.50   955.84    17.38  109.55   6.02  94.60
md0               0.00     0.00    0.00 4782.90     0.00 301676.80   126.15     0.00    0.00   0.00   0.00
dm-6              0.00     0.00    0.00 4783.00     0.00 301683.20   126.15  1222.78  253.83   0.21 100.11

 

Intelligent Reads:

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda             419.70     0.00  216.10    1.20 40678.80     2.40   374.42     7.09   32.66   4.46  96.85
sdc             419.90     0.00  225.40    1.20 40666.00     2.40   358.94     3.96   17.54   2.98  67.64
sdb             421.00     0.00  224.60    1.40 40686.80     3.20   360.09     3.59   15.95   2.82  63.84
sdd             421.50     0.00  222.10    3.00 40601.60     9.60   360.83     3.93   17.47   2.98  67.10
sde             410.10     0.00  223.50    3.00 40620.40     9.60   358.76     3.76   16.64   2.91  65.84
sdf             412.10     0.00  221.90    2.80 40662.40     8.80   362.00     4.03   17.95   3.05  68.49
md0               0.00     0.00 3828.60    0.30 243119.60     7.20   127.00     0.00    0.00   0.00   0.00
dm-6              0.00     0.00 3828.60    0.20 243119.60     7.20   127.00    77.81   20.35   0.25  97.49

 

BONNIE++(1):

Version  1.96       ——Sequential Output—— –Sequential Input- –Random-
Concurrency   1     -Per Chr- –Block– -Rewrite- -Per Chr- –Block– –Seeks–
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
socrates      8000M   487  95 283778  38 121539  25  2261  88 280154  26 402.2  24
Latency             20726us     228ms     226ms    9735us     137ms     125ms
Version  1.96       ——Sequential Create—— ——–Random Create——–
socrates            -Create– –Read— -Delete– -Create– –Read— -Delete–
files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
    16:100000:16/64    78   1 +++++ +++  9078  42    74   0 +++++ +++  7914  45
Latency             33726ms     153us     154us   45029ms    1112us     240us
1.96,1.96,socrates,1,1333434643,8000M,,487,95,283778,38,121539,25,2261,88,280154,
26,402.2,24,16,100000,16,,64,78,1,+++++,+++,9078,42,74,0,+++++,+++,7914,45,20726us,
228ms,226ms,9735us,137ms,125ms,33726ms,153us,154us,45029ms,1112us,240us

 

BONNIE++(2):

Version  1.96       ——Sequential Output—— –Sequential Input- –Random-
Concurrency   1     -Per Chr- –Block– -Rewrite- -Per Chr- –Block– –Seeks–
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
socrates      8000M   482  96 268911  34 119235  25  2289  90 256339  26 396.5  23
Latency             23678us     228ms     230ms   12734us     118ms     113ms
Version  1.96       ——Sequential Create—— ——–Random Create——–
socrates            -Create– –Read— -Delete– -Create– –Read— -Delete–
files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
    16:100000:16/64    77   1 +++++ +++  8508  39    74   0 +++++ +++ 10431  58
Latency             40828ms     149us     170us   43527ms     131us     214us
1.96,1.96,socrates,1,1333436104,8000M,,482,96,268911,34,119235,25,2289,90,256339,
26,396.5,23,16,100000,16,,64,77,1,+++++,+++,8508,39,74,0,+++++,+++,10431,58,23678us,
228ms,230ms,12734us,118ms,113ms,40828ms,149us,170us,43527ms,131us,214us

 

NOTABLES: We are slowly moving into the black once again and seeing R/W speeds closer to what we started with.

WARNING: At this point I noticed that my North Bridge controller (AMD 790X) was hot enough to cook my breakfast on and could very well be affecting my statistics.  A little makeshift fan suspended by some elastics helped but a more permanent solution is in order.  A few moments to cool off and we're ready for the next test.

3.3

ALTERNATIVE COMMAND:

Set everything with the exception of the queue_depth above to 16384:

for mskb in $(ls -al /dev/rsd*|awk '{ print $NF }'); do echo 16384 > /sys/block/$mskb/queue/max_sectors_kb; done

 

IOSTAT:

Intelligent Writes:

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda               5.60 18585.50    1.80  180.40    29.60 75505.20   829.14     9.68   53.18   4.69  85.41
sdc               3.50 18588.70    1.10  175.70    18.40 74504.00   843.01    12.52   68.54   5.04  89.19
sdb               5.20 18588.10    1.30  178.90    26.00 75592.00   839.27    12.02   66.92   4.70  84.70
sdd               4.10 18601.70    1.40  167.10    22.00 75848.40   900.54    14.06   83.98   5.45  91.80
sde               3.80 18618.90    1.10  147.30    19.60 75531.60  1018.21    13.80   92.83   6.31  93.64
sdf               1.40 18626.80    1.40  141.20    11.20 76698.00  1075.87    16.07  113.86   6.85  97.69
md0               0.00     0.00    0.00 4764.30     0.00 300468.55   126.13     0.00    0.00   0.00   0.00
dm-6              0.00     0.00    0.00 4764.30     0.00 300468.55   126.13  1195.13  248.20   0.21 100.03

 

Intelligent Reads:

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda             408.90     0.00  211.30    0.70 39609.60     1.45   373.69     7.01   33.06   4.54  96.33
sdc             413.50     0.00  216.60    0.70 39680.00     1.45   365.22     3.96   18.23   3.01  65.51
sdb             412.60     0.00  217.30    0.70 39616.00     1.45   363.46     3.63   16.66   2.90  63.27
sdd             414.70     1.50  215.10    0.80 39609.60     7.85   367.00     3.89   18.01   3.00  64.74
sde             402.30     1.50  218.00    0.80 39622.40     7.85   362.25     3.63   16.57   2.94  64.41
sdf             403.10     1.50  216.90    0.80 39603.20     7.85   363.90     3.84   17.62   3.06  66.56
md0               0.00     0.00 3753.90    0.20 238387.20     6.40   127.00     0.00    0.00   0.00   0.00
dm-6              0.00     0.00 3753.90    0.10 238387.20     6.40   127.01    76.42   20.34   0.26  96.84

 

BONNIE++(1):

Version  1.96       ——Sequential Output—— –Sequential Input- –Random-
Concurrency   1     -Per Chr- –Block– -Rewrite- -Per Chr- –Block– –Seeks–
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
socrates      8000M   488  95 279460  36 119496  24  2241  90 265914  26 397.5  21
Latency             23270us     231ms     326ms   12677us     157ms     163ms
Version  1.96       ——Sequential Create—— ——–Random Create——–
socrates            -Create– –Read— -Delete– -Create– –Read— -Delete–
files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
    16:100000:16/64    78   1 +++++ +++  7993  37    73   0 +++++ +++  7838  44
Latency             35326ms     149us     182us   45826ms     536us     201us
1.96,1.96,socrates,1,1333429965,8000M,,488,95,279460,36,119496,24,2241,90,265914,
26,397.5,21,16,100000,16,,64,78,1,+++++,+++,7993,37,73,0,+++++,+++,7838,44,23270us,
231ms,326ms,12677us,157ms,163ms,35326ms,149us,182us,45826ms,536us,201us

 

BONNIE++(2):

Version  1.96       ——Sequential Output—— –Sequential Input- –Random-
Concurrency   1     -Per Chr- –Block– -Rewrite- -Per Chr- –Block– –Seeks–
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
socrates      8000M   489  96 259889  34 107053  22  2244  87 243427  23 408.5  22
Latency             26346us     227ms     226ms   20533us     102ms     116ms
Version  1.96       ——Sequential Create—— ——–Random Create——–
socrates            -Create– –Read— -Delete– -Create– –Read— -Delete–
files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
    16:100000:16/64    75   1 +++++ +++  8628  40    73   1 +++++ +++  8316  47
Latency             39128ms     339us     164us   45727ms     231us     203us
1.96,1.96,socrates,1,1333431841,8000M,,489,96,259889,34,107053,22,2244,87,243427,23,
408.5,22,16,100000,16,,64,75,1,+++++,+++,8628,40,73,1,+++++,+++,8316,47,26346us,227ms,
226ms,20533us,102ms,116ms,39128ms,339us,164us,45727ms,231us,203us

 

NOTABLES: Performance was worse by about 10%.

3.4

ALTERNATIVE COMMAND:

Let's try a smaller value:

for mskb in $(ls -al /dev/rsd*|awk '{ print $NF }'); do echo 4096 > /sys/block/$mskb/queue/max_sectors_kb; done

 

IOSTAT:

Intelligent Writes:

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda               4.60 17788.90    1.70  176.50    25.20 72684.00   816.04     9.96   56.37   4.76  84.81
sdc               3.00 17791.00    0.90  168.60    15.60 71544.00   844.36    13.30   78.14   5.20  88.14
sdb               1.40 17770.20    0.80  191.90     8.80 71689.60   744.15    12.98   67.18   4.31  83.08
sdd               4.10 17792.30    1.10  170.00    20.80 71998.40   841.84    12.60   73.62   5.29  90.43
sde               3.90 17822.80    1.20  138.50    20.40 71908.40  1029.76    16.47  118.76   6.60  92.22
sdf               6.70 17811.60    2.00  146.80    34.80 71365.60   959.68    15.52  103.18   6.17  91.74
md0               0.00     0.00    0.00 4564.70     0.00 287891.65   126.14     0.00    0.00   0.00   0.00
dm-6              0.00     0.00    0.00 4564.70     0.00 287898.05   126.14  1199.93  262.78   0.22 100.00


Intelligent Reads:

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda             406.80     0.00  205.80    1.00 39226.40     1.90   379.38     7.13   34.43   4.75  98.29
sdc             402.30     0.00  220.30    1.00 39316.00     1.90   355.34     3.72   16.86   3.01  66.58
sdb             405.40     0.10  216.80    1.10 39265.60     2.70   360.42     3.32   15.28   2.87  62.59
sdd             405.70     0.10  217.00    1.20 39340.80     3.10   360.62     3.77   17.35   3.12  68.12
sde             391.50     0.10  220.80    1.20 39187.20     3.10   353.07     3.44   15.51   2.94  65.25
sdf             393.10     0.00  219.90    1.10 39271.20     2.30   355.42     3.56   16.18   3.03  67.00
md0               0.00     0.00 3702.30    0.20 235110.40     0.85   127.00     0.00    0.00   0.00   0.00
dm-6              0.00     0.00 3702.30    0.20 235110.40     0.85   127.00    73.59   19.92   0.27  98.68

 

BONNIE++(1):

Version  1.96       ——Sequential Output—— –Sequential Input- –Random-
Concurrency   1     -Per Chr- –Block– -Rewrite- -Per Chr- –Block– –Seeks–
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
socrates      8000M   493  95 267924  34 114285  23  2259  89 267171  26 414.7  21
Latency             21284us     228ms     326ms   12583us   97254us     123ms
Version  1.96       ——Sequential Create—— ——–Random Create——–
socrates            -Create– –Read— -Delete– -Create– –Read— -Delete–
files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
    16:100000:16/64    75   1 +++++ +++  7349  33    73   0 +++++ +++  8331  47
Latency             32326ms     126us   10152us   45928ms     119us     211us
1.96,1.96,socrates,1,1333430853,8000M,,493,95,267924,34,114285,23,2259,89,267171,26,414.7,
21,16,100000,16,,64,75,1,+++++,+++,7349,33,73,0,+++++,+++,8331,47,21284us,228ms,326ms,
12583us,97254us,123ms,32326ms,126us,10152us,45928ms,119us,211us

 

BONNIE++(2):

Version  1.96       ——Sequential Output—— –Sequential Input- –Random-
Concurrency   1     -Per Chr- –Block– -Rewrite- -Per Chr- –Block– –Seeks–
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
socrates      8000M   483  95 270530  35 115589  23  2492  97 264919  26 398.9  24
Latency             18683us     328ms     227ms   17678us   88471us     142ms
Version  1.96       ——Sequential Create—— ——–Random Create——–
socrates            -Create– –Read— -Delete– -Create– –Read— -Delete–
files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
    16:100000:16/64    73   0 +++++ +++  8478  39    71   0 +++++ +++  8178  46
Latency             44327ms     169us    1430us   45429ms     130us     226us
1.96,1.96,socrates,1,1333443464,8000M,,483,95,270530,35,115589,23,2492,97,264919,26,398.9,24,
16,100000,16,,64,73,0,+++++,+++,8478,39,71,0,+++++,+++,8178,46,18683us,328ms,227ms,17678us,
88471us,142ms,44327ms,169us,1430us,45429ms,130us,226us

 

NOTE: For this test, we also checked the temperature of the various disks participating in the array during the bonnie++ tests.  If the utility is not available to you, try getting it with yum install hddtemp:

# for rdn in $(ls /dev/rsd*); do echo $(hddtemp $rdn); done
/dev/rsda: ST31000520AS: 34°C
/dev/rsdb: ST31000520AS: 35°C
/dev/rsdc: ST31000520AS: 36°C
/dev/rsdd: ST31000520AS: 36°C
/dev/rsde: ST31000520AS: 38°C
/dev/rsdf: ST1500DL003-9VT16L: 41°C
#

 

4

ALTERNATIVE COMMAND:

Revert all settings to those recommended in the summary.

IOSTAT:

Intelligent Writes (1):

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda               4.80 19011.40    1.80  199.20    26.40 77937.50   775.76    10.63   53.21   4.15  83.37
sdc               1.90 19007.10    0.40  202.70     9.60 77753.50   765.76    13.20   65.24   4.28  86.89
sdb               2.40 18997.10    1.00  212.90    13.60 77704.70   726.68    12.26   57.58   3.84  82.24
sdd               2.80 19024.40    0.60  185.30    13.60 77553.90   834.51    13.03   70.18   4.86  90.27
sde               3.50 19048.20    0.90  160.40    18.00 76994.30   954.90    15.38   94.55   5.61  90.49
sdf               1.10 19059.30    1.00  150.90     8.80 78187.10  1029.57    17.37  115.77   6.33  96.15
md0               0.00     0.00    0.00 4899.70     0.00 309016.05   126.14     0.00    0.00   0.00   0.00
dm-6              0.00     0.00    0.00 4899.60     0.00 309009.65   126.14  1202.49  250.24   0.20 100.02

Intelligent Writes (2):

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda              11.00 18902.50    2.10  182.40    52.40 76089.90   825.39    10.69   57.15   4.79  88.29
sdc              10.10 18892.30    1.80  193.80    47.60 76286.05   780.51    13.72   69.74   4.61  90.26
sdb               6.40 18873.80    1.00  212.50    29.60 76447.50   716.41    11.43   53.73   3.99  85.16
sdd               1.90 18908.20    0.70  178.60    10.40 77126.70   860.42    12.59   71.30   5.13  92.06
sde               5.40 18920.90    1.30  167.00    26.80 77682.30   923.46    15.60   95.09   5.61  94.34
sdf               1.80 18952.90    0.70  131.70    10.00 76095.90  1149.64    17.27  128.77   7.30  96.67
md0               0.00     0.00    0.00 4889.70     0.00 308450.80   126.16     0.00    0.00   0.00   0.00
dm-6              0.00     0.00    0.00 4889.70     0.00 308450.80   126.16  1198.09  245.80   0.20 100.02

 

Intelligent Reads(1):

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda             488.50     0.00  243.60    0.70 46470.80     1.45   380.45    12.07   49.15   3.79  92.47
sdc             492.10     0.00  246.20    0.70 46584.80     1.45   377.37     9.48   38.28   3.06  75.62
sdb             489.70     0.00  248.40    0.90 46603.20     2.25   373.89     8.71   34.83   2.90  72.36
sdd             489.70     0.00  248.40    0.90 46544.00     2.25   373.42     9.13   36.45   3.02  75.38
sde             486.90     0.00  245.40    0.90 46568.80     2.25   378.17     9.00   36.39   3.06  75.43
sdf             484.30     0.00  248.40    0.70 46664.40     1.45   374.68     9.56   38.35   3.15  78.51
md0               0.00     0.00 4420.50    0.10 281749.20     0.80   127.47     0.00    0.00   0.00   0.00
dm-6              0.00     0.00 4420.50    0.20 281749.20     0.85   127.47   171.64   38.67   0.22  95.07

 

Intelligent Reads(2):

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda             502.60     0.00  254.20    0.60 47941.20     1.20   376.31    12.84   50.23   3.73  94.94
sdc             498.30     0.00  252.40    0.60 48003.20     1.20   379.48     9.87   38.85   3.15  79.74
sdb             496.10     0.00  254.90    0.60 47968.40     1.20   375.50     9.28   36.20   3.00  76.57
sdd             496.70     0.00  253.70    0.70 47929.60     1.60   376.82     9.56   37.39   3.09  78.71
sde             502.80     0.00  253.30    0.70 47916.80     1.60   377.31     9.53   37.36   3.09  78.55
sdf             503.80     0.00  253.40    0.70 48010.80     1.60   377.90    10.11   39.66   3.23  82.09
md0               0.00     0.00 4525.60    0.10 288485.20     0.05   127.49     0.00    0.00   0.00   0.00
dm-6              0.00     0.00 4525.60    0.10 288485.20     0.05   127.49   181.37   39.94   0.21  96.54

 

BONNIE++(1):

Version  1.96       ——Sequential Output—— –Sequential Input- –Random-
Concurrency   1     -Per Chr- –Block– -Rewrite- -Per Chr- –Block– –Seeks–
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
socrates      8000M   477  94 294472  38 133467  24  2561  99 318752  24 428.3  21
Latency             27440us     326ms     231ms   17005us     139ms   65182us
Version  1.96       ——Sequential Create—— ——–Random Create——–
socrates            -Create– –Read— -Delete– -Create– –Read— -Delete–
files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
     16:10000:16/64  1539  11 +++++ +++  8431  30  1429  10 +++++ +++  7797  35
Latency             12781us     204us     165us    2418us     171us     169us
1.96,1.96,socrates,1,1333429574,8000M,,477,94,294472,38,133467,24,2561,99,318752,24,
428.3,21,16,10000,16,,64,1539,11,+++++,+++,8431,30,1429,10,+++++,+++,7797,35,27440us,
326ms,231ms,17005us,139ms,65182us,12781us,204us,165us,2418us,171us,169us

 

BONNIE++(2):

Version  1.96       ——Sequential Output—— –Sequential Input- –Random-
Concurrency   1     -Per Chr- –Block– -Rewrite- -Per Chr- –Block– –Seeks–
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
socrates      8000M   494  96 286757  37 125036  23  2570  99 323170  24 405.2  24
Latency             20859us     228ms     229ms   18650us     107ms     141ms
Version  1.96       ——Sequential Create—— ——–Random Create——–
socrates            -Create– –Read— -Delete– -Create– –Read— -Delete–
files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
     16:10000:16/64  1616  12 +++++ +++  8539  31  1612  12 +++++ +++  6608  29
Latency              1851us     122us    4279us     230us     171us     216us
1.96,1.96,socrates,1,1333429992,8000M,,494,96,286757,37,125036,23,2570,99,323170,24,405.2,
24,16,10000,16,,64,1616,12,+++++,+++,8539,31,1612,12,+++++,+++,6608,29,20859us,228ms,
229ms,18650us,107ms,141ms,1851us,122us,4279us,230us,171us,216us

NOTABLES: Able to squeeze about 10%+ more out of the array configuration.

5

ALTERNATIVE COMMAND:

Let's enable the write cache on the individual disks to see what the result would be then if it were safe to have the Write Cache enabled on XFS:

# hdparm -W 1 /dev/rsda; hdparm -W 1 /dev/rsdb; hdparm -W 1 /dev/rsdc; hdparm -W 1 /dev/rsdd; hdparm -W 1 /dev/rsde; hdparm -W 1 /dev/rsdf;

Check with:

# for rdn in $(ls /dev/rsd*); do hdparm -I $rdn|grep -i "write cache"; done
       *    Write cache
       *    Write cache
       *    Write cache
       *    Write cache
       *    Write cache
       *    Write cache
#

Set it back with:

# hdparm -W 0 /dev/rsda; hdparm -W 0 /dev/rsdb; hdparm -W 0 /dev/rsdc; hdparm -W 0 /dev/rsdd; hdparm -W 0 /dev/rsde; hdparm -W 0 /dev/rsdf;

IOSTAT:

Intelligent Writes:

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda               3.00 16860.50    0.70  181.10    14.80 68326.30   751.83     5.90   32.48   3.70  67.25
sdc               7.00 16876.70    1.10  168.10    32.40 69799.50   825.44    13.19   79.55   4.85  82.06
sdb               0.90 16873.00    0.20  170.90     4.40 69331.50   810.47    10.51   61.82   4.52  77.29
sdd               0.90 16878.80    0.40  164.60    10.40 69452.70   841.98    13.25   81.31   4.79  79.02
sde               3.70 16897.90    0.90  145.10    22.00 69181.10   947.99    12.42   85.30   5.90  86.21
sdf               1.00 16894.30    0.50  147.60    11.20 68465.90   924.74    12.78   86.36   5.95  88.09
md0               0.00     0.00    0.00 4453.60     0.00 280896.00   126.14     0.00    0.00   0.00   0.00
dm-6              0.00     0.00    0.00 4453.60     0.00 280896.00   126.14  1308.93  289.12   0.22 100.03

Intelligent Reads:

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda             497.10     0.00  254.30    0.10 48160.40     0.25   378.62    12.68   50.02   3.79  96.35
sdc             500.20     0.00  256.70    0.10 48067.20     0.25   374.36     9.55   37.20   3.09  79.45
sdb             502.80     0.00  254.40    0.10 48074.00     0.25   377.79     9.25   36.35   3.07  78.14
sdd             499.80     0.00  257.10    0.10 48061.20     0.25   373.73     9.12   35.47   3.00  77.17
sde             493.40     0.00  257.70    0.10 48069.60     0.25   372.92     9.42   36.56   3.07  79.06
sdf             496.00     0.00  255.20    0.10 48072.00     0.25   376.59     9.78   38.32   3.21  81.97
md0               0.00     0.00 4524.30    0.00 288402.00     0.00   127.49     0.00    0.00   0.00   0.00
dm-6              0.00     0.00 4524.30    0.00 288402.00     0.00   127.49   176.14   38.97   0.21  97.26

 

 

BONNIE++(1):

Version  1.96       ——Sequential Output—— –Sequential Input- –Random-
Concurrency   1     -Per Chr- –Block– -Rewrite- -Per Chr- –Block– –Seeks–
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
socrates      8000M   488  94 274661  34 118797  23  2491  94 331515  25 415.9  22
Latency             24935us     428ms    2466ms   26596us     127ms   88709us
Version  1.96       ——Sequential Create—— ——–Random Create——–
socrates            -Create– –Read— -Delete– -Create– –Read— -Delete–
files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
     16:10000:16/64  1640  12 +++++ +++  5689  21  1395  11 +++++ +++  4730  22
Latency               336us     542us     407us    2073us     205us    1019us
1.96,1.96,socrates,1,1333431791,8000M,,488,94,274661,34,118797,23,2491,94,331515,25,
415.9,22,16,10000,16,,64,1640,12,+++++,+++,5689,21,1395,11,+++++,+++,4730,22,24935us,
428ms,2466ms,26596us,127ms,88709us,336us,542us,407us,2073us,205us,1019us

 

BONNIE++(2):

Version  1.96       ——Sequential Output—— –Sequential Input- –Random-
Concurrency   1     -Per Chr- –Block– -Rewrite- -Per Chr- –Block– –Seeks–
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
socrates      8000M   476  94 264014  33 113425  23  2594  98 318977  26 434.4  22
Latency             21953us     527ms     343ms   16113us     116ms   86343us
Version  1.96       ——Sequential Create—— ——–Random Create——–
socrates            -Create– –Read— -Delete– -Create– –Read— -Delete–
files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
     16:10000:16/64  1325   9 +++++ +++ 10371  37  1458  10 +++++ +++  8470  38
Latency               369us     201us     164us     234us     179us     217us
1.96,1.96,socrates,1,1333432037,8000M,,476,94,264014,33,113425,23,2594,98,318977,26,
434.4,22,16,10000,16,,64,1325,9,+++++,+++,10371,37,1458,10,+++++,+++,8470,38,21953us,
527ms,343ms,16113us,116ms,86343us,369us,201us,164us,234us,179us,217us

NOTABLES: The Write Cache option actually degraded SDD performance slightly for reads in this test, but improved read performance slightly at the same time.

WARNING: Disable the Write Cache when done.  Power failures could cause data loss when used in conjunction with Journaled filesystems like XFS.  See the notes and links above.

6

No test would be complete without a real world copy test using standard utilities available on Linux.  For this test we'll do just that using cp to copy a bunch of files (text / binary) to the array.  Now for the same file the Linux OS will cache it in memory so subsequent copies should not be directly from source to target:

64MB text files written repeatedly one after another:

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda              26.60  7543.40    2.80   59.20   120.40 27978.25   906.41    30.61  459.30  11.54  71.53
sdc              26.10  7540.90    3.40   55.50   118.80 28079.60   957.50    33.57  538.13  12.51  73.68
sdd              24.40  7546.50    2.70   54.60   109.60 27607.20   967.43    41.96  678.57  13.99  80.19
sdb              24.40  7545.80    2.10   60.80   108.00 29134.55   929.81    27.58  427.06  10.50  66.06
sde              26.10  7541.50    3.70   55.40   119.60 27770.95   943.84    31.77  493.02  11.88  70.22
sdf              22.00  7549.30    2.20   52.30    98.00 26650.40   981.59    45.59  729.74  15.19  82.77
sdg               1.90     4.30    6.90    2.60   112.00    26.80    29.22     0.14   14.82   5.34   5.07
dm-0              0.00     0.00    6.40    6.70   102.40    26.80    19.73     0.15   11.24   3.70   4.85
dm-1              0.00     0.00    2.40    0.00     9.60     0.00     8.00     0.02    7.33   0.92   0.22
dm-2              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
md0               0.00     0.00    0.00 2029.10     0.00 127860.40   126.03     0.00    0.00   0.00   0.00
dm-3              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-4              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-5              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-6              0.00     0.00    0.00 2029.20     0.00 127866.80   126.03  2481.69 1057.88   0.45  91.53

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda              55.30 10072.60    8.00   85.20   256.40 38919.80   840.69    26.82  273.36   8.73  81.39
sdc              49.50 10072.40    5.70   75.70   224.40 38342.85   947.60    42.91  504.35  10.28  83.69
sdd              46.30 10074.50    5.50   77.20   211.20 38181.65   928.48    40.18  459.87  10.16  84.02
sdb              42.30 10075.80    5.30   84.40   194.40 39272.05   879.97    25.64  273.48   9.03  81.03
sde              54.80 10076.60    7.50   76.90   251.60 38155.40   910.12    43.23  480.73   9.91  83.66
sdf              50.90 10079.90    6.90   72.80   234.00 36368.45   918.51    50.99  570.62  10.64  84.84
sdg               0.00     0.30    0.60    2.10     8.40     9.20    13.04     0.01    2.00   1.56   0.42
dm-0              0.00     0.00    0.60    2.30     8.40     9.20    12.14     0.01    2.00   1.45   0.42
dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-2              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
md0               0.00     0.00    0.00 2665.40     0.00 167865.20   125.96     0.00    0.00   0.00   0.00
dm-3              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-4              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-5              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-6              0.00     0.00    0.00 2665.50     0.00 167871.60   125.96  2714.01  915.61   0.34  91.95

Binary file copy (64MB sized files):

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda              42.70 11305.20    4.10  101.30   188.00 43886.10   836.32    18.08  166.91   6.45  67.98
sdc              37.20 11326.30    4.80   80.50   171.60 43631.70  1027.04    25.09  287.91   8.47  72.29
sdd              46.50 11322.50    4.10   81.30   206.80 43414.90  1021.59    30.03  343.35   8.40  71.77
sdb              45.30 11315.90    4.80   95.90   202.80 43580.50   869.58    16.97  163.78   6.96  70.04
sde              43.10 11318.50    3.70   89.90   191.20 42855.70   919.81    20.42  210.11   7.38  69.07
sdf              39.30 11326.30    4.10   82.00   176.40 43772.90  1020.89    28.61  325.39   8.63  74.34
sdg               3.80     1.50   10.00    1.90   256.80    13.20    45.38     0.08    6.49   4.22   5.02
dm-0              0.00     0.00   13.90    3.30   258.40    13.20    31.58     0.10    5.62   2.91   5.01
dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-2              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
md0               0.00     0.00    0.00 2926.10     0.00 182208.40   124.54     0.00    0.00   0.00   0.00
dm-3              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-4              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-5              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-6              0.00     0.00    0.00 2926.10     0.00 182208.40   124.54  1605.75  523.53   0.26  76.64

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda              17.80 14908.20    2.40  123.80    81.60 60010.40   952.33    16.49  130.57   6.86  86.62
sdc              16.90 14926.10    2.40   98.10    78.00 58682.80  1169.37    33.15  326.38   9.54  95.85
sdd              19.70 14935.10    2.30   90.50    80.80 56770.40  1225.24    54.14  560.89  10.40  96.49
sdb              21.20 14910.50    2.80  114.10    95.20 58884.80  1009.07    22.37  189.10   7.48  87.40
sde              20.80 14924.40    2.50   93.80    91.60 56775.60  1181.04    43.71  430.05   9.86  94.94
sdf              19.70 14933.40    2.70   90.70    89.20 55640.40  1193.35    40.01  389.90  10.61  99.08
md0               0.00     0.00    1.20 3562.70     4.80 224597.60   126.04     0.00    0.00   0.00   0.00
dm-6              0.00     0.00    1.20 3562.70     4.80 224597.60   126.04  2867.68  811.34   0.28 100.02

Fastest 10 second average (64MB binary file):

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda              24.10 17271.30    2.50  124.30   108.00 67876.40  1072.31    22.97  171.41   7.13  90.45
sdc              23.80 17275.00    2.40  116.90   107.60 67625.20  1135.50    33.43  269.55   8.03  95.81
sdd              18.80 17282.80    1.70  114.40    80.00 69226.80  1193.92    39.28  334.30   8.28  96.14
sdb              16.70 17276.90    1.60  120.40    63.60 68342.00  1121.40    27.38  215.10   7.50  91.48
sde              25.40 17276.30    2.60  116.40   113.60 67750.00  1140.56    34.27  278.05   7.94  94.52
sdf              15.30 17282.30    1.60  115.20    62.80 68396.80  1172.25    41.17  344.03   8.39  97.97
md0               0.00     0.00    0.00 4386.00     0.00 276576.40   126.12     0.00    0.00   0.00   0.00
dm-6              0.00     0.00    0.00 4385.90     0.00 276570.00   126.12  2704.92  605.61   0.23 100.00

 

NOTE: The above target and source was the RAID6 array itself.  Caching is suspected of helping here because there are not significant reads being observed while the file is being written to the array.

 

Here are some of the notable alternative configurations as suggested by our readers:

 

TEST

ALTERNATIVE SETUP COMMAND + RESULTS:

bonnie++
iostat -xkd 10

1

ALTERNATIVE COMMAND:

This is a variation of the mount command including the logbsize as suggested by one of our readers:

mount -t xfs -o logbsize=128k,logbufs=8,noatime,nodiratime,allocsize=512m /dev/MBPCStorage/MBPCBackup MBPCBackupx/

 

IOSTAT:

Intelligent Writes:

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda               5.20 18681.20    1.70  180.10    27.60 75268.80   828.34     9.95   54.72   4.63  84.12
sdc               6.20 18680.50    1.10  180.90    29.20 75282.00   827.60    14.86   81.65   4.78  87.01
sdb               2.40 18671.20    1.00  189.80    13.60 75083.20   787.18    12.66   66.30   4.41  84.10
sdd               2.80 18694.40    0.60  167.30    13.60 75390.00   898.20    14.96   89.20   5.34  89.73
sde               3.10 18712.80    0.90  146.80    16.00 73985.60  1002.05    15.55  103.95   6.16  90.94
sdf               2.60 18718.30    1.30  142.50    15.60 74572.00  1037.38    17.45  120.77   6.57  94.41
md0               0.00     0.00    0.00 4679.60     0.00 295124.05   126.13     0.00    0.00   0.00   0.00
dm-6              0.00     0.00    0.00 4679.50     0.00 295117.65   126.13  1208.78  260.14   0.21 100.03

 

Intelligent Reads:

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda             493.80     0.00  251.00    0.70 47380.00     1.45   376.49    12.95   51.48   3.77  94.85
sdc             501.50     0.00  249.30    0.70 47456.80     1.45   379.67     9.65   38.53   3.13  78.33
sdb             500.00     0.10  250.30    0.80 47360.00     2.25   377.24     8.93   35.43   3.00  75.21
sdd             501.60     0.10  248.50    0.80 47336.00     2.25   379.77     9.12   36.46   3.03  75.42
sde             494.40     0.10  249.80    0.80 47366.40     2.25   378.04     9.57   38.05   3.09  77.51
sdf             497.20     0.00  247.80    0.70 47418.40     1.45   381.65    10.48   42.14   3.39  84.19
md0               0.00     0.00 4497.50    0.10 286720.00     0.80   127.50     0.00    0.00   0.00   0.00
dm-6              0.00     0.00 4497.50    0.10 286720.00     0.80   127.50   181.57   40.29   0.21  96.64

 

 

BONNIE++(1):

Version  1.96       ——Sequential Output—— –Sequential Input- –Random-
Concurrency   1     -Per Chr- –Block– -Rewrite- -Per Chr- –Block– –Seeks–
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
socrates      8000M   490  96 297953  39 132037  25  2578  97 325839  25 419.2  21
Latency             27478us     227ms     227ms   33920us     130ms     107ms
Version  1.96       ——Sequential Create—— ——–Random Create——–
socrates            -Create– –Read— -Delete– -Create– –Read— -Delete–
files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
     16:10000:16/64  1677  12 +++++ +++  9449  34  1477  10 +++++ +++  8512  38
Latency              8766us     206us     166us     198us     175us     292us
1.96,1.96,socrates,1,1333431309,8000M,,490,96,297953,39,132037,25,2578,97,325839,
25,419.2,21,16,10000,16,,64,1677,12,+++++,+++,9449,34,1477,10,+++++,+++,8512,38,27478us,
227ms,227ms,33920us,130ms,107ms,8766us,206us,166us,198us,175us,292us

 

BONNIE++(2):

Version  1.96       ——Sequential Output—— –Sequential Input- –Random-
Concurrency   1     -Per Chr- –Block– -Rewrite- -Per Chr- –Block– –Seeks–
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
socrates      8000M   488  95 287251  38 125903  24  2625  98 320884  25 411.2  24
Latency             26413us     229ms     232ms   17241us     114ms   67135us
Version  1.96       ——Sequential Create—— ——–Random Create——–
socrates            -Create– –Read— -Delete– -Create– –Read— -Delete–
files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
     16:10000:16/64  1745  13 +++++ +++  8733  31  1471  10 +++++ +++  8645  38
Latency               378us     278us    1327us    1407us     194us    2131us
1.96,1.96,socrates,1,1333431365,8000M,,488,95,287251,38,125903,24,2625,98,320884,
25,411.2,24,16,10000,16,,64,1745,13,+++++,+++,8733,31,1471,10,+++++,+++,8645,38,26413us,229ms,
232ms,17241us,114ms,67135us,378us,278us,1327us,1407us,194us,2131us

 

 


PERFORMANCE CONCLUSION

Though the individual tweaks to the XFS setup and mounting had negligeblie effect, the combination of the parameters used had overall greatest performance result in the tests ran.  This does not mean that this is the best tweaked configuration possible rather under the time and circumstances simply the best of the tests ran. 

 

Pages: 1 2 3 4 5

11 Responses to “HTPC / Backup Home Server Solution using Linux”

  1. Hi,

    Great post. You don’t need to specify the parameters when creating the XFS file system, see http://xfs.org/index.php/XFS_FAQ#Q:_I_want_to_tune_my_XFS_filesystems_for_.3Csomething.3E and http://www.spinics.net/lists/raid/msg38074.html . Of course, YMMV.

    Did you run those benchmarks while the array was resyncing?

  2. Hey Mathias,

    Thanks for posting. Just added the testing numbers so feel free to have a look and judge yourself.

    > logbsize and delaylog
    I ran another test with logbsize=128k (couldn’t find anything for delaylog in my mkfs.xfs man page so I’m not sure if that’ll do anything). Little to no difference in this case on first glance. Watch out for the results at some point for a closer look.

    One consideration here is that eventually I would grow the LVM and XFS to fill up to 4TB. I’ll be doing this soon Potentially in the future, I may try to grow this array as well to something well over 8TB (Yet to see how to do that). I’m not sure if XFS would auto-adjust in those cases for optimal values for those capacities and the link didn’t touch on that topic.

    All in all, I can still run tests on this thing recreating the FS if I need to so feel free to suggest numbers you’d be interested to see. I might leave this topic open for a week or two to see if I can think of anything else or if I’m missing anything. For my setup, having anything > 125MB/s is a bonus as the network is only 1GB/s with that theoretical max.

    Cheers!
    TK

  3. [...] could be done safely enough like this guy did and with RAID6 as well with SSD type R/W’s no less. Your size would be limited to the size of the [...]

  4. Thank you for posting this blog.  I was getting desparate.  I could not figure out why I could not stop the RAID1 device.  Even from Ubuntu Rescue Remix.  The LVM group was being assembled from the failed raid.  I removed the volume group and was finally able to gain exclusive access to the array to stop it, put in the new disk and rebuild the array.
     
    Nice job.
    Best,
    Dave.

  5. [...] we'll use for this is the APCUPSD daemon available in RPM format. We've set one up for our HTPCB server for a home redundancy / backup solution to protect against power surges and bridge the [...]

  6. [...] every time while transferring my files.  At the point, I not only lost connectivity with the HTPC+B but also my web access most of the time.  Here are the culprits and here's how we went [...]

  7. [...] removed the cable and the adapter and only used a 2 foot cable to my HTPC+B system I've just configured.  Voila!  Problem solved.  Ultimately, it's [...]

  8. [...] them from system to system to avoid choppy video / sound and also to accommodate the needs of our HTPC+B solution through file [...]

  9. [...] Linux Networking: Persistent naming rules based on MAC for eth0 and wlan0 Linux: HTPC / Home Backup: MDADM, RAID6, LVM, XFS, CIFS and NFS [...]

  10. [...] at this point and 4:15 minutes have passed).  While this was going on, we are referencing our HTPC page for [...]

  11. [...] HTPC, Backup & Storage [...]

Leave a Reply

You must be logged in to post a comment.


     
  Copyright © 2003 - 2013 Tom Kacperski (microdevsys.com). All rights reserved.

Creative Commons License
This work is licensed under a Creative Commons Attribution 3.0 Unported License