Okay, here's (maybe) the last word on my RAID issues at work, if anyone's still reading these. :)<div><br></div><div>My RAID5 array is 8x 300 GB 15K RPM U320 drives, 7 active with 1 hot spare<br><div><div>md_d1 : active raid5 sdu1[6] sdv1[7](S) sdt1[5] sds1[4] sdr1[3] sdq1[2] sdp1[1] sdo1[0]</div>
<div> 1757804544 blocks level 5, 256k chunk, algorithm 2 [7/7] [UUUUUUU]</div></div><div><br></div><div>Just one drive from this array shows:</div><div><div>/dev/sdr:</div><div> Timing cached reads: 1438 MB in 2.00 seconds = 718.54 MB/sec</div>
<div> Timing buffered disk reads: 384 MB in 3.00 seconds = 127.97 MB/sec</div></div><div><br></div><div><div>/srv/d1# dd if=/dev/zero of=bigtestfile bs=2M count=16384</div><div>16384+0 records in</div><div>16384+0 records out</div>
<div>34359738368 bytes (34 GB) copied, 414.181 s, 83.0 MB/s [writing]</div><div><br></div><div>/srv/d1# dd of=/dev/null if=bigtestfile bs=2M</div><div>16384+0 records in</div><div>16384+0 records out</div><div>34359738368 bytes (34 GB) copied, 126.176 s, 272 MB/s [reading]</div>
</div><div><br></div><div><br></div><div>My RAID10 array is 14x 300 GB 10K RPM U320 drives, 12 active with two hot spares</div><div><div>md_d0 : active raid10 sdn1[12](S) sdm1[13](S) sdl1[11] sdk1[10] sdj1[9] sdi1[8] sdh1[7] sdg1[6] sdf1[5] sde1[4] sdd1[3] sdc1[2] sdb1[1] sda1[0]</div>
<div> 1757804544 blocks 512K chunks 2 near-copies [12/12] [UUUUUUUUUUUU]</div></div><div><div><br></div><div>Just one drive from this array shows:</div><div>/dev/sdl:</div><div> Timing cached reads: 1440 MB in 2.00 seconds = 719.94 MB/sec</div>
<div> Timing buffered disk reads: 254 MB in 3.00 seconds = 84.63 MB/sec</div></div><div><br></div><div><div>/srv/d0# dd if=/dev/zero of=bigtestfile bs=2M count=16384</div><div>16384+0 records in</div><div>16384+0 records out</div>
<div>34359738368 bytes (34 GB) copied, 280.52 s, 122 MB/s [writing]</div><div><br></div><div>/srv/d0# dd of=/dev/null if=bigtestfile bs=2M</div><div>16384+0 records in</div><div>16384+0 records out</div><div>34359738368 bytes (34 GB) copied, 126.134 s, 272 MB/s [reading]</div>
</div><div><br></div><div>Or one could say that reading is the same speed from either array, and pretty much at the maximum practical for an Ultra-320 bus; writing is faster (for my configuration) by 50% on the RAID10 array as compared to the RAID5 array, despite the RAID5 array sporting drives which are ~50% faster (15K RPM vs. 10K RPM).</div>
<div><br></div><div>Curiously, copying the 'bigtestfile' from d0 to d1 or d1 to d0 results in ~80 MB/s either way. I can't think of an explanation off the top of my head.</div><div><br></div><div>There is a lot more testing which could be done, and I'm not saying one configuration is better than the other. I think I may leave it as is for now.</div>
<div><br></div><div>Kevin</div><div><br><br><div class="gmail_quote">On Mon, May 23, 2011 at 5:33 PM, Kevin McGregor <span dir="ltr"><<a href="mailto:kevin.a.mcgregor@gmail.com">kevin.a.mcgregor@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">For yet another comparison, my RAID10 6x 750 GB SATA XFS gives:<div><div># dd if=/dev/zero of=bigfile bs=1M count=16384</div>
<div>16384+0 records in</div><div>16384+0 records out</div><div><a href="tel:17179869184" value="+17179869184" target="_blank">17179869184</a> bytes (17 GB) copied, 79.5334 s, 216 MB/s</div>
<div># dd of=/dev/null if=bigfile bs=1M</div><div>16384+0 records in</div><div>16384+0 records out</div><div><a href="tel:17179869184" value="+17179869184" target="_blank">17179869184</a> bytes (17 GB) copied, 31.4869 s, 546 MB/s</div>
<div><br></div><div>Maybe I should switch to RAID6. ;-)</div>
<div><br></div><font color="#888888"><div>Kevin</div><br></font><div class="gmail_quote"><div class="im">On Fri, May 20, 2011 at 4:54 PM, Trevor Cordes <span dir="ltr"><<a href="mailto:trevor@tecnopolis.ca" target="_blank">trevor@tecnopolis.ca</a>></span> wrote:<br>
</div><div><div></div><div class="h5"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
My RAID6 8x 2TB-drive SATA XFS gives:<br>
<br>
#dd if=/dev/zero of=/new/test bs=1M count=32768<br>
<div>32768+0 records in<br>
32768+0 records out<br>
</div>34359738368 bytes (34 GB) copied, 152.032 s, 226 MB/s<br>
<br>
#dd of=/dev/null if=/new/test bs=1M<br>
<div>32768+0 records in<br>
32768+0 records out<br>
</div>34359738368 bytes (34 GB) copied, 57.7027 s, 595 MB/s<br>
(wow!!)<br>
<br>
During the whole write time both CPU cores were 90-100%, mostly 95-99%!<br>
Glad to see RAID/XFS code is multi-core aware. For reading it was 1<br>
core at 100% and the other around 20%. The write limiting factor<br>
appears to be my piddly Pentium D on my file server. Still, this is<br>
3-4X the speed my old (1TB drives, crappy PCI SATA cards) array was<br>
giving me. The read is quite interesting in that the 100% CPU<br>
indicates it is probably doing parity checks on every read.<br>
<br>
I think a big part of the good speed is my new 8-port SATA card, an<br>
Intel PCI-Express x 8 in a x8 slot. If your SCSI card is just PCI,<br>
then the PCI MB/s speed limit is what's killing you. Even PCI-X may be<br>
limiting. And the Intel card was pretty cheap, under $200.<br>
<br>
BTW, I got stuck with two spare SATA card expander cables (1 card<br>
port to 4 SATA drives) if anyone wants some cheap. I can get in the<br>
Intel cards too, if anyone wants a complete package.<br>
<div><div></div><div>_______________________________________________<br>
Roundtable mailing list<br>
<a href="mailto:Roundtable@muug.mb.ca" target="_blank">Roundtable@muug.mb.ca</a><br>
<a href="http://www.muug.mb.ca/mailman/listinfo/roundtable" target="_blank">http://www.muug.mb.ca/mailman/listinfo/roundtable</a><br>
</div></div></blockquote></div></div></div><br></div>
</blockquote></div><br></div></div>