So far I made this<div><div>md_d0 : active raid10 sdn1[12](S) sdm1[13](S) sdl1[11] sdk1[10] sdj1[9] sdi1[8] sdh1[7] sdg1[6] sdf1[5] sde1[4] sdd1[3] sdc1[2] sdb1[1] sda1[0]</div><div> 1757804544 blocks 512K chunks 2 near-copies [12/12] [UUUUUUUUUUUU]</div>
<div>and got this</div><div><div># dd if=/dev/zero of=/srv/d0/bigzerofile bs=1M count=32768</div><div>32768+0 records in</div><div>32768+0 records out</div><div>34359738368 bytes (34 GB) copied, 281.034 s, 122 MB/s</div><div>
# dd of=/dev/null if=/srv/d0/bigzerofile bs=1M</div><div>32768+0 records in</div><div>32768+0 records out</div><div>34359738368 bytes (34 GB) copied, 126.21 s, 272 MB/s</div></div><div><br></div><div>I'm wondering if 12 drives would over-saturate one Ultra-320 channel. Doesn't Ultra-320 suggest a maximum usable (or theoretical) capacity of 320 MB/s? I could try setting up a stripe set/RAID0 of varying numbers of drives and compare. What do you think?</div>
<div><br></div><div>I don't think the external enclosure (HP MSA30?) allows for splitting the drives into two groups. Only one cable can be connected to it, although there may have been an option for a second at purchase.</div>
<br><div class="gmail_quote">On Thu, May 19, 2011 at 9:27 PM, Adam Thompson <span dir="ltr"><<a href="mailto:athompso@athompso.net">athompso@athompso.net</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div lang="EN-CA" link="blue" vlink="purple"><div><p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">Well, I did say there was more than one variable involved! 25MB/sec is a bit slow, but I don’t know how efficiently that LSI chip is at managing bus contention, or how efficient the kernel driver for that chip is. I do know from experience that 8-disk arrays are well into the land of diminishing returns from a speed perspective: RAID-1 on two disks or RAID-10 on four seems to be the sweet spot for speed.</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"> </span></p><p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">Once your rebuild has finished, I would recommend doing some throughput tests (both reading and writing) on the array; something perhaps like:</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"># sync; time sh –c ‘dd if=/dev/zero of=/mnt/raidarray/BIGFILE.ZERO bs=1M count=1024;sync’</span></p><p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">followed by</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"># time dd if=/dev/md0 of=/dev/null bs=1M count=1024</span></p><p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">Those are both very naïve approaches, but should give you a feel for the maximum read and write speeds of your array. I strongly suspect those numbers will be much higher than the raid re-sync rate, again, mostly due to write-flush barriers in the md code.</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"> </span></p><p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">I’m interested in knowing how this ends, personally… please let us know.</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"> </span></p><p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">-Adam</span></p><p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"> </span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">(P.S. does anyone know how to avoid top-posting in Outlook 2010?)</span></p><p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"> </span></p><p class="MsoNormal">
<a name="1300b3ac1092edb0__MailEndCompose"><span style="font-size:11.0pt;color:#1F497D"> </span></a></p><div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt"><div><div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:10.0pt">From:</span></b><span lang="EN-US" style="font-size:10.0pt"> <a href="mailto:roundtable-bounces@muug.mb.ca" target="_blank">roundtable-bounces@muug.mb.ca</a> [mailto:<a href="mailto:roundtable-bounces@muug.mb.ca" target="_blank">roundtable-bounces@muug.mb.ca</a>] <b>On Behalf Of </b>Kevin McGregor<br>
<b>Sent:</b> Thursday, May 19, 2011 14:52<br><b>To:</b> Continuation of Round Table discussion<br><b>Subject:</b> Re: [RndTbl] RAID5 rebuild performance</span></p></div></div><div><div></div><div class="h5"><p class="MsoNormal">
</p><p class="MsoNormal" style="margin-bottom:12.0pt">raid6: using algorithm sse2x2 (2883 MB/s). So, 1% of that is reasonable? :-) Oh well, I guess I can wait until tomorrow for the rebuild to finish.</p><div><p class="MsoNormal">
On Thu, May 19, 2011 at 2:12 PM, Adam Thompson <<a href="mailto:athompso@athompso.net" target="_blank">athompso@athompso.net</a>> wrote:</p><p class="MsoNormal">Look back in dmesg output for the RAID module speed tests, notice which one was selected, and divide that number by 2. There's your theoretical bottleneck on the CPU.<br>
Take the minimum sustained disk/channel/controller throughput, factor in interrupt latency, device driver efficiency, etc. and make a rough guess as to the overall throughput.<br>Consider that md code seems to have a lot of write barriers for safety - so even a rebuild will aspens much of its time waiting for the disk to sync().<br>
All in all, I think your numbers are probably reasonable.<br><span style="color:#888888">-Adam</span></p><div><div><p class="MsoNormal"><br><br>Kevin McGregor <<a href="mailto:kevin.a.mcgregor@gmail.com" target="_blank">kevin.a.mcgregor@gmail.com</a>> wrote:<br>
<br>>I installed Ubuntu Server 10.04.2 LTS AMD64 on a HP ProLiant ML370 G3 (4 x<br>>dual-core/hyperthreaded Xeon 2.66 GHz, 8 GB RAM) and I used the on-board<br>>SCSI controller to manage 8 x 300 GB 15K RPM SCSI drives in a software RAID<br>
>5 set up as a 7-drive array with 1 hot-spare drive. All drives are the exact<br>>same model with the same firmware version.<br>><br>>It's currently rebuilding the array (because I just created the array) and<br>
>/proc/mdstat is reporting "finish=165.7min speed=25856K/sec". Does that<br>>sound "right" in the sense that it's the right order of magnitude? I though<br>>it should be higher, but I haven't set up such an array before, so I don't<br>
>have anything to compare it to.<br>><br>>If it's slow, does anyone have a suggestion for speeding it up?<br>><br>>Kevin<br>><br>>_______________________________________________<br>>Roundtable mailing list<br>
><a href="mailto:Roundtable@muug.mb.ca" target="_blank">Roundtable@muug.mb.ca</a><br>><a href="http://www.muug.mb.ca/mailman/listinfo/roundtable" target="_blank">http://www.muug.mb.ca/mailman/listinfo/roundtable</a><br>
<br>_______________________________________________<br>Roundtable mailing list<br><a href="mailto:Roundtable@muug.mb.ca" target="_blank">Roundtable@muug.mb.ca</a><br><a href="http://www.muug.mb.ca/mailman/listinfo/roundtable" target="_blank">http://www.muug.mb.ca/mailman/listinfo/roundtable</a></p>
</div></div></div><p class="MsoNormal"> </p></div></div></div></div></div><br>_______________________________________________<br>
Roundtable mailing list<br>
<a href="mailto:Roundtable@muug.mb.ca">Roundtable@muug.mb.ca</a><br>
<a href="http://www.muug.mb.ca/mailman/listinfo/roundtable" target="_blank">http://www.muug.mb.ca/mailman/listinfo/roundtable</a><br>
<br></blockquote></div><br></div>