When the SCSI controller BIOS is initialized, it lists all of the drives on both channels, and most of them (as I recall) are described as being set to 80 MB/s with a note that the OS will probably set them to 320 MB/s. Am I recalling this correctly? Does this make any sense? Does Linux do this? Is there anything I have to do to enable this? I didn't see any messages in dmesg to this effect; I don't know if it's relevant.<div>
<br></div><div>Kevin<br><br><div class="gmail_quote">On Fri, May 20, 2011 at 2:05 PM, Adam Thompson <span dir="ltr"><<a href="mailto:athompso@athompso.net">athompso@athompso.net</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
I can tell you that those numbers are pretty much bang on. Writes would be faster if you enabled the drives' write caches, the HP controller will leave them at default, which is disabled.<br>
The RAID rebuild speed should have been about 90% of that 122Mb number, which it obviously wasn't. I don't have any good theories why at the moment.<br>
<div class="im">-Adam<br>
<br>
<br>
Kevin McGregor <<a href="mailto:kevin.a.mcgregor@gmail.com">kevin.a.mcgregor@gmail.com</a>> wrote:<br>
<br>
</div><div><div></div><div class="h5">>So far I made this<br>
>md_d0 : active raid10 sdn1[12](S) sdm1[13](S) sdl1[11] sdk1[10] sdj1[9]<br>
>sdi1[8] sdh1[7] sdg1[6] sdf1[5] sde1[4] sdd1[3] sdc1[2] sdb1[1] sda1[0]<br>
> 1757804544 blocks 512K chunks 2 near-copies [12/12] [UUUUUUUUUUUU]<br>
>and got this<br>
># dd if=/dev/zero of=/srv/d0/bigzerofile bs=1M count=32768<br>
>32768+0 records in<br>
>32768+0 records out<br>
>34359738368 bytes (34 GB) copied, 281.034 s, 122 MB/s<br>
># dd of=/dev/null if=/srv/d0/bigzerofile bs=1M<br>
>32768+0 records in<br>
>32768+0 records out<br>
>34359738368 bytes (34 GB) copied, 126.21 s, 272 MB/s<br>
><br>
>I'm wondering if 12 drives would over-saturate one Ultra-320 channel.<br>
>Doesn't Ultra-320 suggest a maximum usable (or theoretical) capacity of 320<br>
>MB/s? I could try setting up a stripe set/RAID0 of varying numbers of drives<br>
>and compare. What do you think?<br>
><br>
>I don't think the external enclosure (HP MSA30?) allows for splitting the<br>
>drives into two groups. Only one cable can be connected to it, although<br>
>there may have been an option for a second at purchase.<br>
><br>
>On Thu, May 19, 2011 at 9:27 PM, Adam Thompson <<a href="mailto:athompso@athompso.net">athompso@athompso.net</a>>wrote:<br>
><br>
>> Well, I did say there was more than one variable involved! 25MB/sec is a<br>
>> bit slow, but I don’t know how efficiently that LSI chip is at managing bus<br>
>> contention, or how efficient the kernel driver for that chip is. I do know<br>
>> from experience that 8-disk arrays are well into the land of diminishing<br>
>> returns from a speed perspective: RAID-1 on two disks or RAID-10 on four<br>
>> seems to be the sweet spot for speed.<br>
>><br>
>><br>
>><br>
>> Once your rebuild has finished, I would recommend doing some throughput<br>
>> tests (both reading and writing) on the array; something perhaps like:<br>
>><br>
>> # sync; time sh –c ‘dd if=/dev/zero of=/mnt/raidarray/BIGFILE.ZERO bs=1M<br>
>> count=1024;sync’<br>
>><br>
>> followed by<br>
>><br>
>> # time dd if=/dev/md0 of=/dev/null bs=1M count=1024<br>
>><br>
>> Those are both very naïve approaches, but should give you a feel for the<br>
>> maximum read and write speeds of your array. I strongly suspect those<br>
>> numbers will be much higher than the raid re-sync rate, again, mostly due to<br>
>> write-flush barriers in the md code.<br>
>><br>
>><br>
>><br>
>> I’m interested in knowing how this ends, personally… please let us know.<br>
>><br>
>><br>
>><br>
>> -Adam<br>
>><br>
>><br>
>><br>
>> (P.S. does anyone know how to avoid top-posting in Outlook 2010?)<br>
>><br>
>><br>
>><br>
>><br>
>><br>
>> *From:* <a href="mailto:roundtable-bounces@muug.mb.ca">roundtable-bounces@muug.mb.ca</a> [mailto:<br>
>> <a href="mailto:roundtable-bounces@muug.mb.ca">roundtable-bounces@muug.mb.ca</a>] *On Behalf Of *Kevin McGregor<br>
>> *Sent:* Thursday, May 19, 2011 14:52<br>
>> *To:* Continuation of Round Table discussion<br>
>> *Subject:* Re: [RndTbl] RAID5 rebuild performance<br>
>><br>
>><br>
>><br>
>> raid6: using algorithm sse2x2 (2883 MB/s). So, 1% of that is reasonable?<br>
>> :-) Oh well, I guess I can wait until tomorrow for the rebuild to finish.<br>
>><br>
>> On Thu, May 19, 2011 at 2:12 PM, Adam Thompson <<a href="mailto:athompso@athompso.net">athompso@athompso.net</a>><br>
>> wrote:<br>
>><br>
>> Look back in dmesg output for the RAID module speed tests, notice which one<br>
>> was selected, and divide that number by 2. There's your theoretical<br>
>> bottleneck on the CPU.<br>
>> Take the minimum sustained disk/channel/controller throughput, factor in<br>
>> interrupt latency, device driver efficiency, etc. and make a rough guess as<br>
>> to the overall throughput.<br>
>> Consider that md code seems to have a lot of write barriers for safety -<br>
>> so even a rebuild will aspens much of its time waiting for the disk to<br>
>> sync().<br>
>> All in all, I think your numbers are probably reasonable.<br>
>> -Adam<br>
>><br>
>><br>
>><br>
</div></div><div class="im">>> Kevin McGregor <<a href="mailto:kevin.a.mcgregor@gmail.com">kevin.a.mcgregor@gmail.com</a>> wrote:<br>
>><br>
>> >I installed Ubuntu Server 10.04.2 LTS AMD64 on a HP ProLiant ML370 G3 (4 x<br>
>> >dual-core/hyperthreaded Xeon 2.66 GHz, 8 GB RAM) and I used the on-board<br>
>> >SCSI controller to manage 8 x 300 GB 15K RPM SCSI drives in a software<br>
>> RAID<br>
>> >5 set up as a 7-drive array with 1 hot-spare drive. All drives are the<br>
>> exact<br>
>> >same model with the same firmware version.<br>
>> ><br>
>> >It's currently rebuilding the array (because I just created the array) and<br>
>> >/proc/mdstat is reporting "finish=165.7min speed=25856K/sec". Does that<br>
</div><div class="im">>> >sound "right" in the sense that it's the right order of magnitude? I<br>
>> though<br>
>> >it should be higher, but I haven't set up such an array before, so I don't<br>
>> >have anything to compare it to.<br>
>> ><br>
>> >If it's slow, does anyone have a suggestion for speeding it up?<br>
>> ><br>
>> >Kevin<br>
>> ><br>
>> >_______________________________________________<br>
</div><div><div></div><div class="h5">>> >Roundtable mailing list<br>
>> ><a href="mailto:Roundtable@muug.mb.ca">Roundtable@muug.mb.ca</a><br>
>> ><a href="http://www.muug.mb.ca/mailman/listinfo/roundtable" target="_blank">http://www.muug.mb.ca/mailman/listinfo/roundtable</a><br>
>><br>
>> _______________________________________________<br>
>> Roundtable mailing list<br>
>> <a href="mailto:Roundtable@muug.mb.ca">Roundtable@muug.mb.ca</a><br>
>> <a href="http://www.muug.mb.ca/mailman/listinfo/roundtable" target="_blank">http://www.muug.mb.ca/mailman/listinfo/roundtable</a><br>
>><br>
>><br>
>><br>
>> _______________________________________________<br>
>> Roundtable mailing list<br>
>> <a href="mailto:Roundtable@muug.mb.ca">Roundtable@muug.mb.ca</a><br>
>> <a href="http://www.muug.mb.ca/mailman/listinfo/roundtable" target="_blank">http://www.muug.mb.ca/mailman/listinfo/roundtable</a><br>
>><br>
>><br>
</div></div></blockquote></div><br></div>