<p dir="ltr">Tar it up and compress it. Scp it.</p>
<div class="gmail_quote">On Aug 24, 2016 17:19, "Trevor Cordes" <<a href="mailto:trevor@tecnopolis.ca">trevor@tecnopolis.ca</a>> wrote:<br type="attribution"><blockquote class="quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On 2016-08-24 Micah Garlich-Miller wrote:<br>
><br>
> - It's about 400 GB, but I dont know the data structure as I<br>
<div class="quoted-text">> haven't seen it yet. ie, I'm not sure if its naturally chunked.<br>
</div>> - It's of a sensitivity that it needs to be crypted before being<br>
> sent.<br>
> - Sending on a crypted external harddrive is not acceptable in this<br>
> situation.<br>
<br>
I find a very easy, secure way to give people large files from my linux<br>
box is via apache. If you already have a linux box you run apache on<br>
then this is a breeze. If you don't, you need apache (or any other WS)<br>
and have it serve up port 80/443 to the external internet (either<br>
directly, through DMZ, or port forwarding).<br>
<br>
Since you want encrypted, SSL is a must, even if you put in a<br>
self-signed ssl cert (just send the fingerprint to the other user to<br>
verify in that case).<br>
<br>
What I do is I have a directory on my web server that has directory<br>
indexing turned off in apache. Then I put a directory in it that is<br>
some uberlong (like 64 chars) random alphanum string like<br>
pYwRYezIOz6EIAi4HijyF2hVe70L2V<wbr>2hF3CR6UFMbvjCBgaI5XVsvDeiqzTJ<wbr>a307<br>
<br>
Then in that directory have directory indexing turned on (you can<br>
use .htaccess for this). Put your big files in there.<br>
<br>
Then email or otherwise get the link to your end user:<br>
<br>
<a href="https://foo.com/hidden/pYwRYezIOz6EIAi4HijyF2hVe70L2V2hF3CR6UFMbvjCBgaI5XVsvDeiqzTJa307/" rel="noreferrer" target="_blank">https://foo.com/hidden/<wbr>pYwRYezIOz6EIAi4HijyF2hVe70L2V<wbr>2hF3CR6UFMbvjCBgaI5XVsvDeiqzTJ<wbr>a307/</a><br>
<br>
Voila, secure, encrypted access to the files. For someone else on the<br>
net to gain access, they'd have to intercept your email/backchannel<br>
with the link, or guess the random string, which is basically<br>
impossible for this 62**64 string (or have local shell access to your<br>
server, though creative use of apache group permissions can mitigate<br>
this). If you wanted to you could even add .htpassword basic auth to<br>
the above. (Also, make sure you don't provide an un-SSL port 80<br>
http:// link to it!)<br>
<br>
Test that your no-dirindex is indeed working by going to:<br>
<a href="https://foo.com/hidden/" rel="noreferrer" target="_blank">https://foo.com/hidden/</a><br>
which should give you an error saying indexing denied.<br>
<br>
As for breaking up the files, you could use split to break them into<br>
chunks, which might be prudent, though many http downloaders/browsers<br>
will resume a broken connection so even a single 400G file might be ok<br>
(try using wget with resume options for it).<br>
<br>
To split you'd do:<br>
split -b 1000000000 infile dl-me<br>
for 1GB-ish chunks.<br>
<br>
On the other side it's just:<br>
cat dl-me* > original-file<br>
<br>
Even Windbloze might be able to do it, maybe with type?<br>
type dl-me*.* > original-file<br>
??? not sure on that one, maybe powershell gives us better options now.<br>
<br>
Final note, this transfer will saturate your modem's upload connection,<br>
so you might want to schedule it for a weekend or something.<br>
Ratelimiting with apache is kind of a pain, but the receiving end could<br>
use wget's rate-limiting options (aim for 75% your nominal u/l<br>
bandwidth maybe). Or you can setup qos / ratelimiting (using tc and<br>
iptables) for egress web traffic on your linux web server or linux<br>
router you control, but that gets complicated if you've never done it<br>
before.<br>
<br>
Final final note, you may tick off your ISP and/or exceed your monthly<br>
transfer limits (uploads are usually quite limited vs downloads).<br>
400GB is a huge amount to upload for normal home/SMB shaw/mts<br>
accounts. (You could bzip (or better) the file before starting the<br>
whole process to make it smaller, if both sides have enough disk<br>
space for 2 copies.)<br>
______________________________<wbr>_________________<br>
Roundtable mailing list<br>
<a href="mailto:Roundtable@muug.mb.ca">Roundtable@muug.mb.ca</a><br>
<a href="http://www.muug.mb.ca/mailman/listinfo/roundtable" rel="noreferrer" target="_blank">http://www.muug.mb.ca/mailman/<wbr>listinfo/roundtable</a><br>
</blockquote></div>