<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 18px; font-family: Calibri, sans-serif; ">
<div>Hi,</div>
<div><br>
</div>
<div>Lustre is more scalable and performs better than NFS. </div>
<div><br>
</div>
<div>But it is mostly used in HPC/BigData context where you have lots of concurrent I/O, which is also bulky. Lustre is scalable because you'd have separate data and metadata storage and can scale to several data servers and storage targets. That means to get
most of Lustre you'd need to get several servers and a storage array ; usually the ones used in HPC are expensive. DDN I think would sell you 1/2 PB in 1/2 of rack space for something like 1/2M$ (Not that I have a direct exact quote from them). <a href="http://www.ddn.com/products/storage-platform-sfa12kx">http://www.ddn.com/products/storage-platform-sfa12kx</a>/
The DDN comes also with HA stuff preconfigured. (Lustre is free you could theoretically build a whitebox solution too; the hardest part is to have HA if you want it. )</div>
<div><br>
</div>
<div>Lustre does not like lots of small files at all because it hits metadata servers, although there were some improvements in the recent versions.</div>
<div><br>
</div>
<div>For Lustre you'll need Linux kernel modules for the clients (so if your clients are like, OpenBSD then you'd have to re-export LustreFS for them via NFS or something alike, losing most of the benefits of Lustre). Lustre will work over TCP but best speeds/most
often happens in HPC that it is used over Infiniband. </div>
<div><br>
</div>
<div>GlusterFS is userspace so it is slower but doesn't need any kernel modules and thus should work (any?)where you got FUSE. </div>
<div><br>
</div>
<div>For the homes, will something more like a replicated FS (Andrew, Coda, Inferno) be possibly also? (I have zero experience with any of these). </div>
<div><br>
</div>
<div>
<div>
<div>
<div>-- </div>
<div>Grigory Shamov</div>
</div>
<div>Westgrid/ComputeCanada Site Lead</div>
<div>University of Manitoba</div>
<div>E2-588 EITC Building, </div>
<div>(204) 474-9625</div>
<div><br>
</div>
<div><br>
</div>
</div>
</div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div style="font-family:Calibri; font-size:11pt; text-align:left; color:black; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<span style="font-weight:bold">From: </span>Adam Thompson <<a href="mailto:athompson@avant.ca">athompson@avant.ca</a>><br>
<span style="font-weight:bold">Reply-To: </span>Continuation of Round Table discussion <<a href="mailto:roundtable@muug.mb.ca">roundtable@muug.mb.ca</a>><br>
<span style="font-weight:bold">Date: </span>Monday, 20 April, 2015 4:24 PM<br>
<span style="font-weight:bold">To: </span>"<a href="mailto:roundtable@muug.mb.ca">roundtable@muug.mb.ca</a>" <<a href="mailto:roundtable@muug.mb.ca">roundtable@muug.mb.ca</a>><br>
<span style="font-weight:bold">Subject: </span>[RndTbl] Distributed/replicated home directories?<br>
</div>
<div><br>
</div>
<div>
<div>
<div dir="ltr">
<div class="gmail_default" style="font-family:monospace,monospace">I'm looking for the sweet spot between:</div>
<div class="gmail_default" style="font-family:monospace,monospace"><br>
</div>
<div class="gmail_default" style="font-family:monospace,monospace"> - NFS-mounted home directories, which allow easy central management and are portable across every workstation, but are (typically, and in my case) IOPS-limited,</div>
<div class="gmail_default" style="font-family:monospace,monospace"><br>
</div>
<div class="gmail_default" style="font-family:monospace,monospace">and<br>
</div>
<div class="gmail_default" style="font-family:monospace,monospace"><br>
</div>
<div class="gmail_default" style="font-family:monospace,monospace"> - local home directories, which are tied to the workstation but offer great performance</div>
<div class="gmail_default" style="font-family:monospace,monospace"><br>
</div>
<div class="gmail_default" style="font-family:monospace,monospace">At this point I'm seriously considering things like GlusterFS, Lustre, CephFS, etc., because the NFS performance sucks so badly for doing things like a) running Chrome, and b) running VirtualBox.</div>
<div class="gmail_default" style="font-family:monospace,monospace"><br>
</div>
<div class="gmail_default" style="font-family:monospace,monospace">Looking for ideas...</div>
<div class="gmail_default" style="font-family:monospace,monospace"><br>
</div>
<div>
<div class="gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<table>
<tbody>
<tr>
<td width="141px"><img src="http://avant.ca/av/wp-content/uploads/avant_2014_email-e1391110991529.jpg" alt="Avant logo" width="141px" height="45px"></td>
<td style="font-family:helvetica;font-size:11pt"><b>Adam Thompson</b><br>
Senior Systems Administrator<br>
<b><span style="color:#bed730">voice:</span></b> 204.789.9596 x24 | <b><span style="color:#00abbc">email:</span></b> <a href="mailto:athompson@avant.ca" target="_blank">athompson@avant.ca</a> | <b><span style="color:#b41e8e">web:</span></b> <a href="http://avant.ca/" target="_blank">avant.ca</a></td>
</tr>
</tbody>
</table>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</span>
</body>
</html>