|
TEST
|
ALTERNATIVE SETUP COMMAND + RESULTS:
bonnie++
iostat -xkd 10
|
|
1
|
ALTERNATIVE COMMAND:
mount -t xfs /dev/MBPCStorage/MBPCBackup MBPCBackupx/
IOSTAT:
Intelligent Writes:
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 5.20 19297.80 1.50 185.10 26.80 77572.70 831.72 9.98 53.16 4.63 86.39
sdc 7.20 19287.30 1.50 195.50 34.80 77174.30 783.85 13.22 66.55 4.47 88.04
sdb 4.30 19271.90 0.70 210.60 20.00 77434.70 733.13 11.74 55.29 3.93 82.98
sdd 4.80 19289.00 1.20 195.60 24.00 78303.50 796.01 12.99 66.38 4.58 90.10
sde 7.10 19318.70 1.80 164.20 35.60 77358.30 932.46 15.49 93.15 5.58 92.69
sdf 5.40 19334.40 1.40 149.00 27.20 77870.70 1035.88 19.71 131.60 6.38 96.02
md0 0.00 0.00 0.00 4937.90 0.00 311456.00 126.15 0.00 0.00 0.00 0.00
dm-6 0.00 0.00 0.00 4937.90 0.00 311456.00 126.15 1196.34 243.40 0.20 100.00
Intelligent Reads:
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 488.50 0.00 255.30 0.70 47902.00 1.45 374.25 12.09 47.38 3.64 93.08
sdc 491.00 0.00 259.10 0.70 47911.20 1.45 368.84 8.94 34.52 2.98 77.49
sdb 492.20 0.00 258.40 0.70 47950.00 1.45 370.14 8.46 32.81 2.85 73.89
sdd 490.90 0.00 259.40 0.80 47920.00 1.85 368.35 8.88 34.25 2.99 77.90
sde 485.00 0.00 258.20 0.80 47821.60 1.85 369.29 8.88 34.38 2.97 76.93
sdf 483.40 0.00 259.80 0.80 47859.60 1.85 367.32 9.22 35.49 3.06 79.71
md0 0.00 0.00 4466.60 0.10 284558.00 0.05 127.41 0.00 0.00 0.00 0.00
dm-6 0.00 0.00 4466.60 0.10 284558.00 0.05 127.41 165.24 37.12 0.21 95.38
BONNIE++ (1):
Version 1.96 ——Sequential Output—— –Sequential Input– –Random-
Concurrency 1 -Per Chr- –Block– -Rewrite- -Per Chr- –Block– –Seeks–
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
socrates 8000M 489 95 293621 38 132637 25 2290 90 323067 25 391.3 22
Latency 30605us 230ms 326ms 33651us 128ms 360ms
Version 1.96 ——Sequential Create—— ——–Random Create——–
socrates -Create– –Read— -Delete– -Create– –Read— -Delete–
files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16:100000:16/64 71 0 +++++ +++ 8378 38 68 0 +++++ +++ 7296 40
Latency 33726ms 126us 201us 43328ms 140us 228us
1.96,1.96,socrates,1,1333399320,8000M,,489,95,293621,38,132637,25,2290,90,323067,25,391.
3,22,16,100000,16,,64,71,0,+++++,+++,8378,38,68,0,+++++,+++,7296,40,30605us,230ms,326ms,
33651us,128ms,360ms,33726ms,126us,201us,43328ms,140us,228us
BONNIE++ (2):
Version 1.96 ——Sequential Output—— –Sequential Input– –Random-
Concurrency 1 -Per Chr- –Block– -Rewrite- -Per Chr- –Block– –Seeks–
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
socrates 8000M 488 96 280658 36 123484 24 2142 85 310711 24 374.4 21
Latency 17704us 327ms 327ms 19081us 132ms 589ms
Version 1.96 ——Sequential Create—— ——–Random Create——–
socrates -Create– –Read— -Delete– -Create– –Read— -Delete–
files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16:100000:16/64 70 0 +++++ +++ 8194 37 67 0 +++++ +++ 7165 39
Latency 38230ms 113us 397us 44328ms 92us 436us
1.96,1.96,socrates,1,1333397907,8000M,,488,96,280658,36,123484,24,2142,85,310711,
24,374.4,21,16,100000,16,,64,70,0,+++++,+++,8194,37,67,0,+++++,+++,7165,39,17704us,
327ms,327ms,19081us,132ms,589ms,38230ms,113us,397us,
44328ms,92us,436us
|
|
2
|
ALTERNATIVE COMMAND:
# mkfs.xfs -L MBPCBackupx /dev/MBPCStorage/MBPCBackup -f
meta-data=/dev/MBPCStorage/MBPCBackup isize=256 agcount=32, agsize=15258800 blks
= sectsz=512 attr=2
data = bsize=4096 blocks=488281600, imaxpct=5
= sunit=16 swidth=64 blks
naming =version 2 bsize=4096 ascii-ci=0
log =internal log bsize=4096 blocks=238432, version=2
= sectsz=512 sunit=16 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
#
IOSTAT:
Intelligent Writes:
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 2.20 18679.10 1.00 180.10 12.80 75371.85 832.52 11.12 61.21 4.74 85.86
sdc 2.60 18669.30 0.70 190.30 13.20 75435.05 790.03 12.40 64.92 4.45 85.00
sdb 1.70 18656.80 0.50 201.20 8.80 75428.65 748.02 11.36 56.34 3.95 79.76
sdd 4.20 18663.70 1.20 194.60 20.90 75429.85 770.69 12.55 64.11 4.32 84.50
sde 6.20 18698.20 1.40 158.20 30.40 75422.65 945.53 14.14 88.58 5.56 88.79
sdf 2.20 18702.30 0.90 157.50 12.40 74483.20 940.60 16.09 100.19 5.75 91.15
md0 0.00 0.00 0.40 4885.30 0.90 308108.85 126.13 0.00 0.00 0.00 0.00
dm-6 0.00 0.00 0.40 4885.30 0.90 308115.25 126.13 1130.71 226.76 0.20 96.45
Intelligent Reads:
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 478.70 1.50 239.00 1.20 45558.80 8.55 379.41 12.47 52.17 3.94 94.54
sdc 472.00 0.00 241.60 1.10 45485.20 2.15 374.84 9.74 40.19 3.23 78.29
sdb 471.00 0.00 240.90 2.90 45491.20 9.35 373.26 8.37 34.40 2.93 71.53
sdd 474.10 0.00 239.30 1.30 45446.40 2.95 377.80 8.70 36.26 3.07 73.95
sde 475.70 0.00 243.40 1.30 45452.80 2.95 371.52 8.55 35.08 3.04 74.46
sdf 474.70 0.00 242.10 2.70 45325.20 8.55 370.37 8.85 36.11 3.07 75.19
md0 0.00 0.00 4291.90 0.30 273612.80 7.20 127.50 0.00 0.00 0.00 0.00
dm-6 0.00 0.00 4291.90 0.20 273612.80 7.20 127.50 168.01 39.24 0.22 96.26
NOTABLES:
iostat await metric going ballistic:
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 179.90 0.00 58.10 0.00 1050.40 36.16 44.12 800.78 17.21 100.00
sdc 0.00 171.20 0.00 65.20 0.00 967.60 29.68 16.83 259.36 13.41 87.44
sdb 0.00 168.50 0.00 68.70 0.00 991.20 28.86 13.25 198.22 10.97 75.38
sdd 0.00 190.90 0.00 43.50 0.00 975.20 44.84 140.54 3185.73 22.99 100.00
sde 0.00 170.70 0.00 66.80 0.00 998.00 29.88 11.88 186.03 10.91 72.88
sdf 0.00 190.20 0.00 44.10 0.00 992.80 45.02 130.85 2948.80 22.68 100.00
md0 0.00 0.00 0.00 218.10 0.00 7852.00 72.00 0.00 0.00 0.00 0.00
dm-6 0.00 0.00 0.00 218.20 0.00 7858.40 72.03 8414.55 8454053194093797.00 4.58 100.00
BONNIE++ (1):
Version 1.96 ——Sequential Output—— –Sequential Input– –Random-
Concurrency 1 -Per Chr- –Block– -Rewrite- -Per Chr- –Block– –Seeks–
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
socrates 8000M 483 95 296433 38 128335 24 2184 84 322603 25 399.4 23
Latency 20200us 227ms 326ms 16633us 129ms 276ms
Version 1.96 ——Sequential Create—— ——–Random Create——–
socrates -Create– –Read— -Delete– -Create– –Read— -Delete–
files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16:100000:16/64 79 1 +++++ +++ 8971 41 76 1 +++++ +++ 8383 46
Latency 30627ms 533us 170us 42027ms 99us 213us
1.96,1.96,socrates,1,1333397740,8000M,,483,95,296433,38,128335,24,2184,84,322603,25,399.4,
23,16,100000,16,,64,79,1,+++++,+++,8971,41,76,1,+++++,+++,8383,46,20200us,227ms,326ms,
16633us,129ms,276ms,30627ms,533us,170us,42027ms,99us,213us
BONNIE++ (2):
Version 1.96 ——Sequential Output—— –Sequential Input– –Random-
Concurrency 1 -Per Chr- –Block– -Rewrite- -Per Chr- –Block– –Seeks–
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
socrates 8000M 490 95 277994 35 119279 23 2079 81 304914 25 401.3 20
Latency 33137us 233ms 228ms 121ms 122ms 401ms
Version 1.96 ——Sequential Create—— ——–Random Create——–
socrates -Create– –Read— -Delete– -Create– –Read— -Delete–
files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16:100000:16/64 77 1 +++++ +++ 9243 42 73 0 +++++ +++ 7274 40
Latency 31526ms 102us 158us 43727ms 98us 221us
1.96,1.96,socrates,1,1333399367,8000M,,490,95,277994,35,119279,23,2079,81,304914,25,401.
3,20,16,100000,16,,64,77,1,+++++,+++,9243,42,73,0,+++++,+++,7274,40,33137us,233ms,228ms,
121ms,122ms,401ms,31526ms,102us,158us,43727ms,98us,221us
|
|
3
|
ALTERNATIVE COMMAND:
echo 256 > /sys/block/md0/md/stripe_cache_size
blockdev –setra 256 $(echo $(ls -al /dev/rsd*|awk '{ print "/dev/"$NF }'))
blockdev –setra 1024 /dev/md0
blockdev –setra 1024 /dev/MBPCStorage/MBPCBackup
for mskb in $(ls -al /dev/rsd*|awk '{ print $NF }'); do echo 512 > /sys/block/$mskb/queue/max_sectors_kb; done
for qdepth in $(ls -al /dev/rsd*|awk '{ print $NF }'); do echo 31 > /sys/block/$qdepth/device/queue_depth; done
IOSTAT:
Intelligent Writes:
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 12.80 9791.50 2.00 141.20 59.20 39643.60 554.51 1.63 11.33 5.43 77.70
sdc 9.80 9772.60 1.60 143.30 40.00 39640.80 547.70 1.73 11.90 5.43 78.70
sdb 13.10 9774.10 1.80 141.20 59.60 39669.60 555.65 1.64 11.47 5.35 76.51
sdd 7.20 9773.00 1.20 142.20 33.60 39708.00 554.28 1.67 11.65 5.38 77.11
sde 6.80 9773.80 1.50 141.10 33.20 39643.60 556.48 1.68 11.79 5.41 77.09
sdf 10.20 9774.00 1.50 140.30 46.80 39640.80 559.77 1.71 12.01 5.58 79.17
md0 0.00 0.00 0.00 2515.20 0.00 158643.20 126.15 0.00 0.00 0.00 0.00
dm-6 0.00 0.00 0.00 2515.20 0.00 158643.20 126.15 60.06 23.85 0.40 99.99
Intelligent Reads:
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 5.60 0.00 964.30 0.70 61838.00 1.45 128.16 0.61 0.63 0.43 41.35
sdc 80.60 0.00 1118.50 0.70 61831.20 1.45 110.49 0.62 0.55 0.42 47.12
sdb 62.70 0.10 1136.90 0.80 61798.00 2.25 108.64 0.53 0.47 0.36 41.41
sdd 92.10 0.10 1108.20 0.80 61841.60 2.25 111.53 0.63 0.57 0.42 46.91
sde 19.90 0.10 949.40 0.80 61848.80 2.25 130.19 0.71 0.75 0.51 48.69
sdf 26.80 0.00 942.10 0.70 61818.40 1.45 131.14 0.97 1.03 0.63 59.24
md0 0.00 0.00 6506.00 0.10 370941.60 0.80 114.03 0.00 0.00 0.00 0.00
dm-6 0.00 0.00 6506.00 0.10 370941.60 0.80 114.03 4.50 0.69 0.15 96.76
BONNIE++(1):
Version 1.96 ——Sequential Output—— –Sequential Input– –Random-
Concurrency 1 -Per Chr- –Block– -Rewrite- -Per Chr- –Block– –Seeks–
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
socrates 8000M 486 95 148258 28 72908 22 2199 86 379404 51 411.3 21
Latency 17967us 228ms 185ms 35547us 51719us 385ms
Version 1.96 ——Sequential Create—— ——–Random Create——–
socrates -Create– –Read— -Delete– -Create– –Read— -Delete–
files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16:100000:16/64 43 0 +++++ +++ 7566 34 42 0 +++++ +++ 6701 36
Latency 6650ms 272us 168us 8276ms 106us 194us
1.96,1.96,socrates,1,1333397055,8000M,,486,95,148258,28,72908,22,2199,86,379404,
51,411.3,21,16,100000,16,,64,43,0,+++++,+++,7566,34,42,0,+++++,+++,6701,36,17967us,
228ms,185ms,35547us,51719us,385ms,6650ms,272us,168us,8276ms,106us,194us
BONNIE++(2):
Version 1.96 ——Sequential Output—— –Sequential Input– –Random-
Concurrency 1 -Per Chr- –Block– -Rewrite- -Per Chr- –Block– –Seeks–
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
socrates 8000M 485 95 145075 27 71573 22 2239 87 369472 55 401.3 22
Latency 22517us 226ms 362ms 30212us 107ms 279ms
Version 1.96 ——Sequential Create—— ——–Random Create——–
socrates -Create– –Read— -Delete– -Create– –Read— -Delete–
files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16:100000:16/64 42 0 +++++ +++ 7508 34 41 0 +++++ +++ 6637 36
Latency 7638ms 130us 176us 8339ms 137us 182us
1.96,1.96,socrates,1,1333427981,8000M,,485,95,145075,27,71573,22,2239,87,369472,55,401.3,
22,16,100000,16,,64,42,0,+++++,+++,7508,34,41,0,+++++,+++,6637,36,22517us,226ms,362ms,
30212us,107ms,279ms,7638ms,130us,176us,8339ms,137us,182us
NOTABLES: Writes were degraded to about 1/2 of their former value.
NOTABLES: Reads were up by approx 50MB/s.
|
|
3.1
|
Test #3 was very interesting. So let's expand on it:
ALTERNATIVE COMMAND:
We'll put everything back with the exception of stripe_cache_size which we'll set to match for all the individual components:
echo 32768 > /sys/block/md0/md/stripe_cache_size
blockdev –setra 32768 $(echo $(ls -al /dev/rsd*|awk '{ print "/dev/"$NF }'))
blockdev –setra 32768 /dev/md0
blockdev –setra 32768 /dev/MBPCStorage/MBPCBackup
for mskb in $(ls -al /dev/rsd*|awk '{ print $NF }'); do echo 8192 > /sys/block/$mskb/queue/max_sectors_kb; done
for qdepth in $(ls -al /dev/rsd*|awk '{ print $NF }'); do echo 31 > /sys/block/$qdepth/device/queue_depth; done
IOSTAT:
Intelligent Writes:
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 142.38 12806.91 15.83 128.16 617.23 50336.47 707.75 41.01 271.73 6.58 94.68
sdc 172.85 12729.46 16.63 162.32 759.12 49745.29 564.43 30.91 165.18 5.19 92.93
sdb 170.44 12780.56 16.33 156.31 723.05 49380.96 580.43 35.64 193.64 5.35 92.33
sdd 192.28 12776.05 18.04 134.27 808.02 50432.67 672.87 43.75 272.42 6.27 95.48
sde 181.16 12765.73 19.14 129.16 806.41 49708.02 681.26 38.57 246.31 6.49 96.29
sdf 180.26 12818.14 17.33 113.23 758.32 50658.72 787.63 43.87 323.79 7.60 99.27
md0 0.00 0.00 0.00 3434.87 0.00 204659.77 119.17 0.00 0.00 0.00 0.00
dm-6 0.00 0.00 0.00 3434.87 0.00 204659.77 119.17 1578.39 444.59 0.29 100.02
Intelligent Reads:
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 492.30 0.00 244.90 0.70 47040.40 1.45 383.08 12.70 51.65 3.89 95.52
sdc 491.90 0.00 251.10 0.70 47091.60 1.45 374.05 9.79 38.79 3.24 81.47
sdb 494.10 0.00 249.30 0.70 47123.60 1.45 377.00 9.38 37.45 3.12 78.01
sdd 494.20 0.00 249.10 0.80 47139.20 1.85 377.28 9.81 39.20 3.25 81.29
sde 488.50 0.00 249.10 0.80 47148.80 1.85 377.36 9.81 39.22 3.23 80.78
sdf 488.70 0.00 249.00 0.80 47168.40 1.85 377.66 9.90 39.58 3.27 81.80
md0 0.00 0.00 4446.10 0.10 283443.20 0.05 127.50 0.00 0.00 0.00 0.00
dm-6 0.00 0.00 4446.10 0.10 283443.20 0.05 127.50 181.43 40.75 0.22 97.06
BONNIE++(1):
Version 1.96 ——Sequential Output—— –Sequential Input– –Random-
Concurrency 1 -Per Chr- –Block– -Rewrite- -Per Chr- –Block– –Seeks–
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
socrates 8000M 493 96 206133 28 111493 21 2244 87 297317 24 358.2 21
Latency 27737us 356ms 232ms 21396us 119ms 341ms
Version 1.96 ——Sequential Create—— ——–Random Create——–
socrates -Create– –Read— -Delete– -Create– –Read— -Delete–
files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16:100000:16/64 75 0 4685 15 11215 53 75 0 +++++ +++ 8131 45
Latency 66627ms 76193us 181us 66132ms 112us 193us
1.96,1.96,socrates,1,1333430597,8000M,,493,96,206133,28,111493,21,2244,87,297317,
24,358.2,21,16,100000,16,,64,75,0,4685,15,11215,53,75,0,+++++,+++,8131,45,27737us,
356ms,232ms,21396us,119ms,341ms,66627ms,76193us,181us,66132ms,112us,193us
BONNIE++(2):
Version 1.96 ——Sequential Output—— –Sequential Input– –Random-
Concurrency 1 -Per Chr- –Block– -Rewrite- -Per Chr- –Block– –Seeks–
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
socrates 8000M 488 96 174456 25 109809 21 2196 88 278683 21 374.1 22
Latency 24236us 453ms 327ms 19331us 162ms 133ms
Version 1.96 ——Sequential Create—— ——–Random Create——–
socrates -Create– –Read— -Delete– -Create– –Read— -Delete–
files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16:100000:16/64 76 0 +++++ +++ 8931 41 75 0 +++++ +++ 7209 39
Latency 65126ms 118us 161us 63427ms 123us 180us
1.96,1.96,socrates,1,1333430451,8000M,,488,96,174456,25,109809,21,2196,88,278683,
21,374.1,22,16,100000,16,,64,76,0,+++++,+++,8931,41,75,0,+++++,+++,7209,39,24236us,
453ms,327ms,19331us,162ms,133ms,65126ms,118us,161us,63427ms,123us,180us
NOTABLES: Better then test 3.0 but still below our max achieved.
|
|
3.2
|
ALTERNATIVE COMMAND:
echo 16384 > /sys/block/md0/md/stripe_cache_size
blockdev –setra 16384 $(echo $(ls -al /dev/rsd*|awk '{ print "/dev/"$NF }'))
blockdev –setra 16384 /dev/md0
blockdev –setra 16384 /dev/MBPCStorage/MBPCBackup
for mskb in $(ls -al /dev/rsd*|awk '{ print $NF }'); do echo 8192 > /sys/block/$mskb/queue/max_sectors_kb; done
for qdepth in $(ls -al /dev/rsd*|awk '{ print $NF }'); do echo 31 > /sys/block/$qdepth/device/queue_depth; done
IOSTAT:
Intelligent Writes:
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 7.50 18947.50 1.50 172.60 36.00 75815.50 871.36 12.00 68.38 5.00 87.01
sdc 5.20 18926.90 1.30 192.40 26.40 75673.50 781.62 13.73 70.46 4.56 88.37
sdb 2.90 18912.00 0.50 208.60 13.60 76045.10 727.49 11.01 52.47 3.93 82.17
sdd 2.60 18913.80 0.40 206.60 12.00 75798.30 732.47 13.06 63.05 4.15 85.86
sde 1.10 18968.60 0.80 149.60 7.60 74913.10 996.29 14.87 97.27 6.18 92.95
sdf 4.40 18962.30 1.00 156.20 21.60 75107.50 955.84 17.38 109.55 6.02 94.60
md0 0.00 0.00 0.00 4782.90 0.00 301676.80 126.15 0.00 0.00 0.00 0.00
dm-6 0.00 0.00 0.00 4783.00 0.00 301683.20 126.15 1222.78 253.83 0.21 100.11
Intelligent Reads:
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 419.70 0.00 216.10 1.20 40678.80 2.40 374.42 7.09 32.66 4.46 96.85
sdc 419.90 0.00 225.40 1.20 40666.00 2.40 358.94 3.96 17.54 2.98 67.64
sdb 421.00 0.00 224.60 1.40 40686.80 3.20 360.09 3.59 15.95 2.82 63.84
sdd 421.50 0.00 222.10 3.00 40601.60 9.60 360.83 3.93 17.47 2.98 67.10
sde 410.10 0.00 223.50 3.00 40620.40 9.60 358.76 3.76 16.64 2.91 65.84
sdf 412.10 0.00 221.90 2.80 40662.40 8.80 362.00 4.03 17.95 3.05 68.49
md0 0.00 0.00 3828.60 0.30 243119.60 7.20 127.00 0.00 0.00 0.00 0.00
dm-6 0.00 0.00 3828.60 0.20 243119.60 7.20 127.00 77.81 20.35 0.25 97.49
BONNIE++(1):
Version 1.96 ——Sequential Output—— –Sequential Input– –Random-
Concurrency 1 -Per Chr- –Block– -Rewrite- -Per Chr- –Block– –Seeks–
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
socrates 8000M 487 95 283778 38 121539 25 2261 88 280154 26 402.2 24
Latency 20726us 228ms 226ms 9735us 137ms 125ms
Version 1.96 ——Sequential Create—— ——–Random Create——–
socrates -Create– –Read— -Delete– -Create– –Read— -Delete–
files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16:100000:16/64 78 1 +++++ +++ 9078 42 74 0 +++++ +++ 7914 45
Latency 33726ms 153us 154us 45029ms 1112us 240us
1.96,1.96,socrates,1,1333434643,8000M,,487,95,283778,38,121539,25,2261,88,280154,
26,402.2,24,16,100000,16,,64,78,1,+++++,+++,9078,42,74,0,+++++,+++,7914,45,20726us,
228ms,226ms,9735us,137ms,125ms,33726ms,153us,154us,45029ms,1112us,240us
BONNIE++(2):
Version 1.96 ——Sequential Output—— –Sequential Input– –Random-
Concurrency 1 -Per Chr- –Block– -Rewrite- -Per Chr- –Block– –Seeks–
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
socrates 8000M 482 96 268911 34 119235 25 2289 90 256339 26 396.5 23
Latency 23678us 228ms 230ms 12734us 118ms 113ms
Version 1.96 ——Sequential Create—— ——–Random Create——–
socrates -Create– –Read— -Delete– -Create– –Read— -Delete–
files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16:100000:16/64 77 1 +++++ +++ 8508 39 74 0 +++++ +++ 10431 58
Latency 40828ms 149us 170us 43527ms 131us 214us
1.96,1.96,socrates,1,1333436104,8000M,,482,96,268911,34,119235,25,2289,90,256339,
26,396.5,23,16,100000,16,,64,77,1,+++++,+++,8508,39,74,0,+++++,+++,10431,58,23678us,
228ms,230ms,12734us,118ms,113ms,40828ms,149us,170us,43527ms,131us,214us
NOTABLES: We are slowly moving into the black once again and seeing R/W speeds closer to what we started with.
WARNING: At this point I noticed that my North Bridge controller (AMD 790X) was hot enough to cook my breakfast on and could very well be affecting my statistics. A little makeshift fan suspended by some elastics helped but a more permanent solution is in order. A few moments to cool off and we're ready for the next test.
|
|
3.3
|
ALTERNATIVE COMMAND:
Set everything with the exception of the queue_depth above to 16384:
for mskb in $(ls -al /dev/rsd*|awk '{ print $NF }'); do echo 16384 > /sys/block/$mskb/queue/max_sectors_kb; done
IOSTAT:
Intelligent Writes:
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 5.60 18585.50 1.80 180.40 29.60 75505.20 829.14 9.68 53.18 4.69 85.41
sdc 3.50 18588.70 1.10 175.70 18.40 74504.00 843.01 12.52 68.54 5.04 89.19
sdb 5.20 18588.10 1.30 178.90 26.00 75592.00 839.27 12.02 66.92 4.70 84.70
sdd 4.10 18601.70 1.40 167.10 22.00 75848.40 900.54 14.06 83.98 5.45 91.80
sde 3.80 18618.90 1.10 147.30 19.60 75531.60 1018.21 13.80 92.83 6.31 93.64
sdf 1.40 18626.80 1.40 141.20 11.20 76698.00 1075.87 16.07 113.86 6.85 97.69
md0 0.00 0.00 0.00 4764.30 0.00 300468.55 126.13 0.00 0.00 0.00 0.00
dm-6 0.00 0.00 0.00 4764.30 0.00 300468.55 126.13 1195.13 248.20 0.21 100.03
Intelligent Reads:
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 408.90 0.00 211.30 0.70 39609.60 1.45 373.69 7.01 33.06 4.54 96.33
sdc 413.50 0.00 216.60 0.70 39680.00 1.45 365.22 3.96 18.23 3.01 65.51
sdb 412.60 0.00 217.30 0.70 39616.00 1.45 363.46 3.63 16.66 2.90 63.27
sdd 414.70 1.50 215.10 0.80 39609.60 7.85 367.00 3.89 18.01 3.00 64.74
sde 402.30 1.50 218.00 0.80 39622.40 7.85 362.25 3.63 16.57 2.94 64.41
sdf 403.10 1.50 216.90 0.80 39603.20 7.85 363.90 3.84 17.62 3.06 66.56
md0 0.00 0.00 3753.90 0.20 238387.20 6.40 127.00 0.00 0.00 0.00 0.00
dm-6 0.00 0.00 3753.90 0.10 238387.20 6.40 127.01 76.42 20.34 0.26 96.84
BONNIE++(1):
Version 1.96 ——Sequential Output—— –Sequential Input– –Random-
Concurrency 1 -Per Chr- –Block– -Rewrite- -Per Chr- –Block– –Seeks–
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
socrates 8000M 488 95 279460 36 119496 24 2241 90 265914 26 397.5 21
Latency 23270us 231ms 326ms 12677us 157ms 163ms
Version 1.96 ——Sequential Create—— ——–Random Create——–
socrates -Create– –Read— -Delete– -Create– –Read— -Delete–
files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16:100000:16/64 78 1 +++++ +++ 7993 37 73 0 +++++ +++ 7838 44
Latency 35326ms 149us 182us 45826ms 536us 201us
1.96,1.96,socrates,1,1333429965,8000M,,488,95,279460,36,119496,24,2241,90,265914,
26,397.5,21,16,100000,16,,64,78,1,+++++,+++,7993,37,73,0,+++++,+++,7838,44,23270us,
231ms,326ms,12677us,157ms,163ms,35326ms,149us,182us,45826ms,536us,201us
BONNIE++(2):
Version 1.96 ——Sequential Output—— –Sequential Input– –Random-
Concurrency 1 -Per Chr- –Block– -Rewrite- -Per Chr- –Block– –Seeks–
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
socrates 8000M 489 96 259889 34 107053 22 2244 87 243427 23 408.5 22
Latency 26346us 227ms 226ms 20533us 102ms 116ms
Version 1.96 ——Sequential Create—— ——–Random Create——–
socrates -Create– –Read— -Delete– -Create– –Read— -Delete–
files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16:100000:16/64 75 1 +++++ +++ 8628 40 73 1 +++++ +++ 8316 47
Latency 39128ms 339us 164us 45727ms 231us 203us
1.96,1.96,socrates,1,1333431841,8000M,,489,96,259889,34,107053,22,2244,87,243427,23,
408.5,22,16,100000,16,,64,75,1,+++++,+++,8628,40,73,1,+++++,+++,8316,47,26346us,227ms,
226ms,20533us,102ms,116ms,39128ms,339us,164us,45727ms,231us,203us
NOTABLES: Performance was worse by about 10%.
|
|
3.4
|
ALTERNATIVE COMMAND:
Let's try a smaller value:
for mskb in $(ls -al /dev/rsd*|awk '{ print $NF }'); do echo 4096 > /sys/block/$mskb/queue/max_sectors_kb; done
IOSTAT:
Intelligent Writes:
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 4.60 17788.90 1.70 176.50 25.20 72684.00 816.04 9.96 56.37 4.76 84.81
sdc 3.00 17791.00 0.90 168.60 15.60 71544.00 844.36 13.30 78.14 5.20 88.14
sdb 1.40 17770.20 0.80 191.90 8.80 71689.60 744.15 12.98 67.18 4.31 83.08
sdd 4.10 17792.30 1.10 170.00 20.80 71998.40 841.84 12.60 73.62 5.29 90.43
sde 3.90 17822.80 1.20 138.50 20.40 71908.40 1029.76 16.47 118.76 6.60 92.22
sdf 6.70 17811.60 2.00 146.80 34.80 71365.60 959.68 15.52 103.18 6.17 91.74
md0 0.00 0.00 0.00 4564.70 0.00 287891.65 126.14 0.00 0.00 0.00 0.00
dm-6 0.00 0.00 0.00 4564.70 0.00 287898.05 126.14 1199.93 262.78 0.22 100.00
Intelligent Reads:
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 406.80 0.00 205.80 1.00 39226.40 1.90 379.38 7.13 34.43 4.75 98.29
sdc 402.30 0.00 220.30 1.00 39316.00 1.90 355.34 3.72 16.86 3.01 66.58
sdb 405.40 0.10 216.80 1.10 39265.60 2.70 360.42 3.32 15.28 2.87 62.59
sdd 405.70 0.10 217.00 1.20 39340.80 3.10 360.62 3.77 17.35 3.12 68.12
sde 391.50 0.10 220.80 1.20 39187.20 3.10 353.07 3.44 15.51 2.94 65.25
sdf 393.10 0.00 219.90 1.10 39271.20 2.30 355.42 3.56 16.18 3.03 67.00
md0 0.00 0.00 3702.30 0.20 235110.40 0.85 127.00 0.00 0.00 0.00 0.00
dm-6 0.00 0.00 3702.30 0.20 235110.40 0.85 127.00 73.59 19.92 0.27 98.68
BONNIE++(1):
Version 1.96 ——Sequential Output—— –Sequential Input– –Random-
Concurrency 1 -Per Chr- –Block– -Rewrite- -Per Chr- –Block– –Seeks–
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
socrates 8000M 493 95 267924 34 114285 23 2259 89 267171 26 414.7 21
Latency 21284us 228ms 326ms 12583us 97254us 123ms
Version 1.96 ——Sequential Create—— ——–Random Create——–
socrates -Create– –Read— -Delete– -Create– –Read— -Delete–
files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16:100000:16/64 75 1 +++++ +++ 7349 33 73 0 +++++ +++ 8331 47
Latency 32326ms 126us 10152us 45928ms 119us 211us
1.96,1.96,socrates,1,1333430853,8000M,,493,95,267924,34,114285,23,2259,89,267171,26,414.7,
21,16,100000,16,,64,75,1,+++++,+++,7349,33,73,0,+++++,+++,8331,47,21284us,228ms,326ms,
12583us,97254us,123ms,32326ms,126us,10152us,45928ms,119us,211us
BONNIE++(2):
Version 1.96 ——Sequential Output—— –Sequential Input– –Random-
Concurrency 1 -Per Chr- –Block– -Rewrite- -Per Chr- –Block– –Seeks–
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
socrates 8000M 483 95 270530 35 115589 23 2492 97 264919 26 398.9 24
Latency 18683us 328ms 227ms 17678us 88471us 142ms
Version 1.96 ——Sequential Create—— ——–Random Create——–
socrates -Create– –Read— -Delete– -Create– –Read— -Delete–
files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16:100000:16/64 73 0 +++++ +++ 8478 39 71 0 +++++ +++ 8178 46
Latency 44327ms 169us 1430us 45429ms 130us 226us
1.96,1.96,socrates,1,1333443464,8000M,,483,95,270530,35,115589,23,2492,97,264919,26,398.9,24,
16,100000,16,,64,73,0,+++++,+++,8478,39,71,0,+++++,+++,8178,46,18683us,328ms,227ms,17678us,
88471us,142ms,44327ms,169us,1430us,45429ms,130us,226us
NOTE: For this test, we also checked the temperature of the various disks participating in the array during the bonnie++ tests. If the utility is not available to you, try getting it with yum install hddtemp:
# for rdn in $(ls /dev/rsd*); do echo $(hddtemp $rdn); done
/dev/rsda: ST31000520AS: 34°C
/dev/rsdb: ST31000520AS: 35°C
/dev/rsdc: ST31000520AS: 36°C
/dev/rsdd: ST31000520AS: 36°C
/dev/rsde: ST31000520AS: 38°C
/dev/rsdf: ST1500DL003-9VT16L: 41°C
#
|
|
4
|
ALTERNATIVE COMMAND:
Revert all settings to those recommended in the summary.
IOSTAT:
Intelligent Writes (1):
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 4.80 19011.40 1.80 199.20 26.40 77937.50 775.76 10.63 53.21 4.15 83.37
sdc 1.90 19007.10 0.40 202.70 9.60 77753.50 765.76 13.20 65.24 4.28 86.89
sdb 2.40 18997.10 1.00 212.90 13.60 77704.70 726.68 12.26 57.58 3.84 82.24
sdd 2.80 19024.40 0.60 185.30 13.60 77553.90 834.51 13.03 70.18 4.86 90.27
sde 3.50 19048.20 0.90 160.40 18.00 76994.30 954.90 15.38 94.55 5.61 90.49
sdf 1.10 19059.30 1.00 150.90 8.80 78187.10 1029.57 17.37 115.77 6.33 96.15
md0 0.00 0.00 0.00 4899.70 0.00 309016.05 126.14 0.00 0.00 0.00 0.00
dm-6 0.00 0.00 0.00 4899.60 0.00 309009.65 126.14 1202.49 250.24 0.20 100.02
Intelligent Writes (2):
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 11.00 18902.50 2.10 182.40 52.40 76089.90 825.39 10.69 57.15 4.79 88.29
sdc 10.10 18892.30 1.80 193.80 47.60 76286.05 780.51 13.72 69.74 4.61 90.26
sdb 6.40 18873.80 1.00 212.50 29.60 76447.50 716.41 11.43 53.73 3.99 85.16
sdd 1.90 18908.20 0.70 178.60 10.40 77126.70 860.42 12.59 71.30 5.13 92.06
sde 5.40 18920.90 1.30 167.00 26.80 77682.30 923.46 15.60 95.09 5.61 94.34
sdf 1.80 18952.90 0.70 131.70 10.00 76095.90 1149.64 17.27 128.77 7.30 96.67
md0 0.00 0.00 0.00 4889.70 0.00 308450.80 126.16 0.00 0.00 0.00 0.00
dm-6 0.00 0.00 0.00 4889.70 0.00 308450.80 126.16 1198.09 245.80 0.20 100.02
Intelligent Reads(1):
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 488.50 0.00 243.60 0.70 46470.80 1.45 380.45 12.07 49.15 3.79 92.47
sdc 492.10 0.00 246.20 0.70 46584.80 1.45 377.37 9.48 38.28 3.06 75.62
sdb 489.70 0.00 248.40 0.90 46603.20 2.25 373.89 8.71 34.83 2.90 72.36
sdd 489.70 0.00 248.40 0.90 46544.00 2.25 373.42 9.13 36.45 3.02 75.38
sde 486.90 0.00 245.40 0.90 46568.80 2.25 378.17 9.00 36.39 3.06 75.43
sdf 484.30 0.00 248.40 0.70 46664.40 1.45 374.68 9.56 38.35 3.15 78.51
md0 0.00 0.00 4420.50 0.10 281749.20 0.80 127.47 0.00 0.00 0.00 0.00
dm-6 0.00 0.00 4420.50 0.20 281749.20 0.85 127.47 171.64 38.67 0.22 95.07
Intelligent Reads(2):
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 502.60 0.00 254.20 0.60 47941.20 1.20 376.31 12.84 50.23 3.73 94.94
sdc 498.30 0.00 252.40 0.60 48003.20 1.20 379.48 9.87 38.85 3.15 79.74
sdb 496.10 0.00 254.90 0.60 47968.40 1.20 375.50 9.28 36.20 3.00 76.57
sdd 496.70 0.00 253.70 0.70 47929.60 1.60 376.82 9.56 37.39 3.09 78.71
sde 502.80 0.00 253.30 0.70 47916.80 1.60 377.31 9.53 37.36 3.09 78.55
sdf 503.80 0.00 253.40 0.70 48010.80 1.60 377.90 10.11 39.66 3.23 82.09
md0 0.00 0.00 4525.60 0.10 288485.20 0.05 127.49 0.00 0.00 0.00 0.00
dm-6 0.00 0.00 4525.60 0.10 288485.20 0.05 127.49 181.37 39.94 0.21 96.54
BONNIE++(1):
Version 1.96 ——Sequential Output—— –Sequential Input– –Random-
Concurrency 1 -Per Chr- –Block– -Rewrite- -Per Chr- –Block– –Seeks–
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
socrates 8000M 477 94 294472 38 133467 24 2561 99 318752 24 428.3 21
Latency 27440us 326ms 231ms 17005us 139ms 65182us
Version 1.96 ——Sequential Create—— ——–Random Create——–
socrates -Create– –Read— -Delete– -Create– –Read— -Delete–
files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16:10000:16/64 1539 11 +++++ +++ 8431 30 1429 10 +++++ +++ 7797 35
Latency 12781us 204us 165us 2418us 171us 169us
1.96,1.96,socrates,1,1333429574,8000M,,477,94,294472,38,133467,24,2561,99,318752,24,
428.3,21,16,10000,16,,64,1539,11,+++++,+++,8431,30,1429,10,+++++,+++,7797,35,27440us,
326ms,231ms,17005us,139ms,65182us,12781us,204us,165us,2418us,171us,169us
BONNIE++(2):
Version 1.96 ——Sequential Output—— –Sequential Input– –Random-
Concurrency 1 -Per Chr- –Block– -Rewrite- -Per Chr- –Block– –Seeks–
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
socrates 8000M 494 96 286757 37 125036 23 2570 99 323170 24 405.2 24
Latency 20859us 228ms 229ms 18650us 107ms 141ms
Version 1.96 ——Sequential Create—— ——–Random Create——–
socrates -Create– –Read— -Delete– -Create– –Read— -Delete–
files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16:10000:16/64 1616 12 +++++ +++ 8539 31 1612 12 +++++ +++ 6608 29
Latency 1851us 122us 4279us 230us 171us 216us
1.96,1.96,socrates,1,1333429992,8000M,,494,96,286757,37,125036,23,2570,99,323170,24,405.2,
24,16,10000,16,,64,1616,12,+++++,+++,8539,31,1612,12,+++++,+++,6608,29,20859us,228ms,
229ms,18650us,107ms,141ms,1851us,122us,4279us,230us,171us,216us
NOTABLES: Able to squeeze about 10%+ more out of the array configuration.
|
|
5
|
ALTERNATIVE COMMAND:
Let's enable the write cache on the individual disks to see what the result would be then if it were safe to have the Write Cache enabled on XFS:
# hdparm -W 1 /dev/rsda; hdparm -W 1 /dev/rsdb; hdparm -W 1 /dev/rsdc; hdparm -W 1 /dev/rsdd; hdparm -W 1 /dev/rsde; hdparm -W 1 /dev/rsdf;
Check with:
# for rdn in $(ls /dev/rsd*); do hdparm -I $rdn|grep -i "write cache"; done
* Write cache
* Write cache
* Write cache
* Write cache
* Write cache
* Write cache
#
Set it back with:
# hdparm -W 0 /dev/rsda; hdparm -W 0 /dev/rsdb; hdparm -W 0 /dev/rsdc; hdparm -W 0 /dev/rsdd; hdparm -W 0 /dev/rsde; hdparm -W 0 /dev/rsdf;
IOSTAT:
Intelligent Writes:
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 3.00 16860.50 0.70 181.10 14.80 68326.30 751.83 5.90 32.48 3.70 67.25
sdc 7.00 16876.70 1.10 168.10 32.40 69799.50 825.44 13.19 79.55 4.85 82.06
sdb 0.90 16873.00 0.20 170.90 4.40 69331.50 810.47 10.51 61.82 4.52 77.29
sdd 0.90 16878.80 0.40 164.60 10.40 69452.70 841.98 13.25 81.31 4.79 79.02
sde 3.70 16897.90 0.90 145.10 22.00 69181.10 947.99 12.42 85.30 5.90 86.21
sdf 1.00 16894.30 0.50 147.60 11.20 68465.90 924.74 12.78 86.36 5.95 88.09
md0 0.00 0.00 0.00 4453.60 0.00 280896.00 126.14 0.00 0.00 0.00 0.00
dm-6 0.00 0.00 0.00 4453.60 0.00 280896.00 126.14 1308.93 289.12 0.22 100.03
Intelligent Reads:
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 497.10 0.00 254.30 0.10 48160.40 0.25 378.62 12.68 50.02 3.79 96.35
sdc 500.20 0.00 256.70 0.10 48067.20 0.25 374.36 9.55 37.20 3.09 79.45
sdb 502.80 0.00 254.40 0.10 48074.00 0.25 377.79 9.25 36.35 3.07 78.14
sdd 499.80 0.00 257.10 0.10 48061.20 0.25 373.73 9.12 35.47 3.00 77.17
sde 493.40 0.00 257.70 0.10 48069.60 0.25 372.92 9.42 36.56 3.07 79.06
sdf 496.00 0.00 255.20 0.10 48072.00 0.25 376.59 9.78 38.32 3.21 81.97
md0 0.00 0.00 4524.30 0.00 288402.00 0.00 127.49 0.00 0.00 0.00 0.00
dm-6 0.00 0.00 4524.30 0.00 288402.00 0.00 127.49 176.14 38.97 0.21 97.26
BONNIE++(1):
Version 1.96 ——Sequential Output—— –Sequential Input– –Random-
Concurrency 1 -Per Chr- –Block– -Rewrite- -Per Chr- –Block– –Seeks–
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
socrates 8000M 488 94 274661 34 118797 23 2491 94 331515 25 415.9 22
Latency 24935us 428ms 2466ms 26596us 127ms 88709us
Version 1.96 ——Sequential Create—— ——–Random Create——–
socrates -Create– –Read— -Delete– -Create– –Read— -Delete–
files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16:10000:16/64 1640 12 +++++ +++ 5689 21 1395 11 +++++ +++ 4730 22
Latency 336us 542us 407us 2073us 205us 1019us
1.96,1.96,socrates,1,1333431791,8000M,,488,94,274661,34,118797,23,2491,94,331515,25,
415.9,22,16,10000,16,,64,1640,12,+++++,+++,5689,21,1395,11,+++++,+++,4730,22,24935us,
428ms,2466ms,26596us,127ms,88709us,336us,542us,407us,2073us,205us,1019us
BONNIE++(2):
Version 1.96 ——Sequential Output—— –Sequential Input– –Random-
Concurrency 1 -Per Chr- –Block– -Rewrite- -Per Chr- –Block– –Seeks–
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
socrates 8000M 476 94 264014 33 113425 23 2594 98 318977 26 434.4 22
Latency 21953us 527ms 343ms 16113us 116ms 86343us
Version 1.96 ——Sequential Create—— ——–Random Create——–
socrates -Create– –Read— -Delete– -Create– –Read— -Delete–
files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16:10000:16/64 1325 9 +++++ +++ 10371 37 1458 10 +++++ +++ 8470 38
Latency 369us 201us 164us 234us 179us 217us
1.96,1.96,socrates,1,1333432037,8000M,,476,94,264014,33,113425,23,2594,98,318977,26,
434.4,22,16,10000,16,,64,1325,9,+++++,+++,10371,37,1458,10,+++++,+++,8470,38,21953us,
527ms,343ms,16113us,116ms,86343us,369us,201us,164us,234us,179us,217us
NOTABLES: The Write Cache option actually degraded SDD performance slightly for reads in this test, but improved read performance slightly at the same time.
WARNING: Disable the Write Cache when done. Power failures could cause data loss when used in conjunction with Journaled filesystems like XFS. See the notes and links above.
|
|
6
|
No test would be complete without a real world copy test using standard utilities available on Linux. For this test we'll do just that using cp to copy a bunch of files (text / binary) to the array. Now for the same file the Linux OS will cache it in memory so subsequent copies should not be directly from source to target:
64MB text files written repeatedly one after another:
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 26.60 7543.40 2.80 59.20 120.40 27978.25 906.41 30.61 459.30 11.54 71.53
sdc 26.10 7540.90 3.40 55.50 118.80 28079.60 957.50 33.57 538.13 12.51 73.68
sdd 24.40 7546.50 2.70 54.60 109.60 27607.20 967.43 41.96 678.57 13.99 80.19
sdb 24.40 7545.80 2.10 60.80 108.00 29134.55 929.81 27.58 427.06 10.50 66.06
sde 26.10 7541.50 3.70 55.40 119.60 27770.95 943.84 31.77 493.02 11.88 70.22
sdf 22.00 7549.30 2.20 52.30 98.00 26650.40 981.59 45.59 729.74 15.19 82.77
sdg 1.90 4.30 6.90 2.60 112.00 26.80 29.22 0.14 14.82 5.34 5.07
dm-0 0.00 0.00 6.40 6.70 102.40 26.80 19.73 0.15 11.24 3.70 4.85
dm-1 0.00 0.00 2.40 0.00 9.60 0.00 8.00 0.02 7.33 0.92 0.22
dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
md0 0.00 0.00 0.00 2029.10 0.00 127860.40 126.03 0.00 0.00 0.00 0.00
dm-3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-6 0.00 0.00 0.00 2029.20 0.00 127866.80 126.03 2481.69 1057.88 0.45 91.53
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 55.30 10072.60 8.00 85.20 256.40 38919.80 840.69 26.82 273.36 8.73 81.39
sdc 49.50 10072.40 5.70 75.70 224.40 38342.85 947.60 42.91 504.35 10.28 83.69
sdd 46.30 10074.50 5.50 77.20 211.20 38181.65 928.48 40.18 459.87 10.16 84.02
sdb 42.30 10075.80 5.30 84.40 194.40 39272.05 879.97 25.64 273.48 9.03 81.03
sde 54.80 10076.60 7.50 76.90 251.60 38155.40 910.12 43.23 480.73 9.91 83.66
sdf 50.90 10079.90 6.90 72.80 234.00 36368.45 918.51 50.99 570.62 10.64 84.84
sdg 0.00 0.30 0.60 2.10 8.40 9.20 13.04 0.01 2.00 1.56 0.42
dm-0 0.00 0.00 0.60 2.30 8.40 9.20 12.14 0.01 2.00 1.45 0.42
dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
md0 0.00 0.00 0.00 2665.40 0.00 167865.20 125.96 0.00 0.00 0.00 0.00
dm-3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-6 0.00 0.00 0.00 2665.50 0.00 167871.60 125.96 2714.01 915.61 0.34 91.95
Binary file copy (64MB sized files):
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 42.70 11305.20 4.10 101.30 188.00 43886.10 836.32 18.08 166.91 6.45 67.98
sdc 37.20 11326.30 4.80 80.50 171.60 43631.70 1027.04 25.09 287.91 8.47 72.29
sdd 46.50 11322.50 4.10 81.30 206.80 43414.90 1021.59 30.03 343.35 8.40 71.77
sdb 45.30 11315.90 4.80 95.90 202.80 43580.50 869.58 16.97 163.78 6.96 70.04
sde 43.10 11318.50 3.70 89.90 191.20 42855.70 919.81 20.42 210.11 7.38 69.07
sdf 39.30 11326.30 4.10 82.00 176.40 43772.90 1020.89 28.61 325.39 8.63 74.34
sdg 3.80 1.50 10.00 1.90 256.80 13.20 45.38 0.08 6.49 4.22 5.02
dm-0 0.00 0.00 13.90 3.30 258.40 13.20 31.58 0.10 5.62 2.91 5.01
dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
md0 0.00 0.00 0.00 2926.10 0.00 182208.40 124.54 0.00 0.00 0.00 0.00
dm-3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-6 0.00 0.00 0.00 2926.10 0.00 182208.40 124.54 1605.75 523.53 0.26 76.64
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 17.80 14908.20 2.40 123.80 81.60 60010.40 952.33 16.49 130.57 6.86 86.62
sdc 16.90 14926.10 2.40 98.10 78.00 58682.80 1169.37 33.15 326.38 9.54 95.85
sdd 19.70 14935.10 2.30 90.50 80.80 56770.40 1225.24 54.14 560.89 10.40 96.49
sdb 21.20 14910.50 2.80 114.10 95.20 58884.80 1009.07 22.37 189.10 7.48 87.40
sde 20.80 14924.40 2.50 93.80 91.60 56775.60 1181.04 43.71 430.05 9.86 94.94
sdf 19.70 14933.40 2.70 90.70 89.20 55640.40 1193.35 40.01 389.90 10.61 99.08
md0 0.00 0.00 1.20 3562.70 4.80 224597.60 126.04 0.00 0.00 0.00 0.00
dm-6 0.00 0.00 1.20 3562.70 4.80 224597.60 126.04 2867.68 811.34 0.28 100.02
Fastest 10 second average (64MB binary file):
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 24.10 17271.30 2.50 124.30 108.00 67876.40 1072.31 22.97 171.41 7.13 90.45
sdc 23.80 17275.00 2.40 116.90 107.60 67625.20 1135.50 33.43 269.55 8.03 95.81
sdd 18.80 17282.80 1.70 114.40 80.00 69226.80 1193.92 39.28 334.30 8.28 96.14
sdb 16.70 17276.90 1.60 120.40 63.60 68342.00 1121.40 27.38 215.10 7.50 91.48
sde 25.40 17276.30 2.60 116.40 113.60 67750.00 1140.56 34.27 278.05 7.94 94.52
sdf 15.30 17282.30 1.60 115.20 62.80 68396.80 1172.25 41.17 344.03 8.39 97.97
md0 0.00 0.00 0.00 4386.00 0.00 276576.40 126.12 0.00 0.00 0.00 0.00
dm-6 0.00 0.00 0.00 4385.90 0.00 276570.00 126.12 2704.92 605.61 0.23 100.00
NOTE: The above target and source was the RAID6 array itself. Caching is suspected of helping here because there are not significant reads being observed while the file is being written to the array.
|
Hi,
Great post. You don’t need to specify the parameters when creating the XFS file system, see http://xfs.org/index.php/XFS_FAQ#Q:_I_want_to_tune_my_XFS_filesystems_for_.3Csomething.3E and http://www.spinics.net/lists/raid/msg38074.html . Of course, YMMV.
Did you run those benchmarks while the array was resyncing?
Hey Mathias,
Thanks for posting. Just added the testing numbers so feel free to have a look and judge yourself.
> logbsize and delaylog
I ran another test with logbsize=128k (couldn’t find anything for delaylog in my mkfs.xfs man page so I’m not sure if that’ll do anything). Little to no difference in this case on first glance. Watch out for the results at some point for a closer look.
One consideration here is that eventually I would grow the LVM and XFS to fill up to 4TB. I’ll be doing this soon Potentially in the future, I may try to grow this array as well to something well over 8TB (Yet to see how to do that). I’m not sure if XFS would auto-adjust in those cases for optimal values for those capacities and the link didn’t touch on that topic.
All in all, I can still run tests on this thing recreating the FS if I need to so feel free to suggest numbers you’d be interested to see. I might leave this topic open for a week or two to see if I can think of anything else or if I’m missing anything. For my setup, having anything > 125MB/s is a bonus as the network is only 1GB/s with that theoretical max.
Cheers!
TK
[…] could be done safely enough like this guy did and with RAID6 as well with SSD type R/W’s no less. Your size would be limited to the size of the […]
Thank you for posting this blog. I was getting desparate. I could not figure out why I could not stop the RAID1 device. Even from Ubuntu Rescue Remix. The LVM group was being assembled from the failed raid. I removed the volume group and was finally able to gain exclusive access to the array to stop it, put in the new disk and rebuild the array.
Nice job.
Best,
Dave.
[…] we'll use for this is the APCUPSD daemon available in RPM format. We've set one up for our HTPCB server for a home redundancy / backup solution to protect against power surges and bridge the […]
[…] every time while transferring my files. At the point, I not only lost connectivity with the HTPC+B but also my web access most of the time. Here are the culprits and here's how we went […]
[…] removed the cable and the adapter and only used a 2 foot cable to my HTPC+B system I've just configured. Voila! Problem solved. Ultimately, it's […]
[…] them from system to system to avoid choppy video / sound and also to accommodate the needs of our HTPC+B solution through file […]
[…] Linux Networking: Persistent naming rules based on MAC for eth0 and wlan0 Linux: HTPC / Home Backup: MDADM, RAID6, LVM, XFS, CIFS and NFS […]
[…] at this point and 4:15 minutes have passed). While this was going on, we are referencing our HTPC page for […]
[…] HTPC, Backup & Storage […]