<html><head><meta http-equiv="Content-Type" content="text/html charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Apr 20, 2016, at 2:03 AM, Trevor Cordes <<a href="mailto:trevor@tecnopolis.ca" class="">trevor@tecnopolis.ca</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class="">When I run tests at the command line (dig) on the domains in question (the <br class="">ones I've seen in email bounces) I will often very quickly get a resolve <br class="">failure, but usually 5s later another dig to the same domain will resolve <br class="">100% ok!<br class=""><br class=""></div></div></blockquote><div><br class=""></div><div>I often use a utility called "check-soa" to check that each of the nameservers listed in the last NS response respond with an SOA</div><div><a href="https://github.com/bortzmeyer/check-soa" class="">https://github.com/bortzmeyer/check-soa</a></div><div>When running it against each of the 5 domains below I did experience occasional delays in command line output, I'm guessing since none of the latency values reported went up, that this delay was from the original NS response</div><br class=""><blockquote type="cite" class=""><div class=""><div class="">Every box I am testing on has a similar config with BIND named 9.10 (9.8 <br class="">on one box) running as the local recursive resolver. /etc/resolv.conf on <br class="">all is 127.0.0.1. So that means every lookup that isn't cached is going <br class="">to the root NS's.<br class=""><br class="">When it fails to resolve, named.log logs an entry like:<br class="">20-Apr-2016 00:37:28.276 query-errors: debug 1: client 127.0.0.1#33971 (<a href="http://artscouncil.mb.ca" class="">artscouncil.mb.ca</a>): query failed (SERVFAIL) for <a href="http://artscouncil.mb.ca/IN/A" class="">artscouncil.mb.ca/IN/A</a> at query.c:7769<br class="">20-Apr-2016 00:37:28.276 query-errors: debug 2: fetch completed at resolver.c:3658 for <a href="http://artscouncil.mb.ca/A" class="">artscouncil.mb.ca/A</a> in 0.215778: SERVFAIL/success [domain:artscouncil.mb.ca,referral:1,restart:3,qrysent:2,timeout:0,lame:0,neterr:2,badresp:0,adberr:0,findfail:0,valfail:0]<br class=""><br class="">For manual tests it only logs one, for the real-life sendmail problem <br class="">ones, I'll see dozen/hundreds of the same thing trying hour after hour <br class="">(usually around the sendmail queue retry times).<br class=""><br class="">One of my boxes is *extremely* well connected in the US, and while it <br class="">seems to have errors slightly less often, it still has them. All the rest <br class="">are on various levels of Shaw or MTS, res or business.<br class=""><br class="">This seems to have just started popping up maybe 6 months ago. It feels <br class="">like it's getting worse.<br class=""><br class="">I've setup an easy test, on the actual domains with the most problems:<br class=""><br class="">rndc flush<br class="">dig +short <a href="http://sportmanitoba.ca" class="">sportmanitoba.ca</a><br class="">dig +short <a href="http://gymcan.org" class="">gymcan.org</a><br class="">dig +short <a href="http://brandoneagles.ca" class="">brandoneagles.ca</a><br class="">dig +short <a href="http://interactivegym.org" class="">interactivegym.org</a><br class="">dig +short <a href="http://artscouncil.mb.ca" class="">artscouncil.mb.ca</a><br class=""><br class="">rndc flush<br class="">dig +trace <a href="http://sportmanitoba.ca" class="">sportmanitoba.ca</a><br class="">dig +trace <a href="http://gymcan.org" class="">gymcan.org</a><br class="">dig +trace <a href="http://brandoneagles.ca" class="">brandoneagles.ca</a><br class="">dig +trace <a href="http://interactivegym.org" class="">interactivegym.org</a><br class="">dig +trace <a href="http://artscouncil.mb.ca" class="">artscouncil.mb.ca</a><br class=""><br class="">Maybe some others can run those tests on their boxes (but only if you're <br class="">running BIND as caching resolver, which many/most people won't be).<br class=""><br class="">Here's where it gets interesting the +short tests I can get to fail at <br class="">least 1 of the domains (at random) about 1/8 of the time! On at least 5 <br class="">different boxes out there! But +trace has never failed once on any box or <br class="">any domain. It's like +trace does something different, maybe slowing the <br class="">process down or something, that allows it to always succeed. (Failure for <br class="">the +short is a missing line in output, +trace you have to look at the <br class="">domain/ip returned near the bottom.)<br class=""><br class=""></div></div></blockquote><div><br class=""></div><div>AFAIK +trace doesn't use your caching resolver except to get an answer for which nameservers/IPs to query the root/. at so this is definitely doing something different.</div><div><br class=""></div><br class=""><blockquote type="cite" class=""><div class=""><div class="">So, is it just these particular domains?? Something wrong on their (DNS) <br class="">side? Or is it more domains, not just these? Is there any way to <br class="">diagnose what exactly is failing? I find it bizarre that *all* of these <br class="">domains regularly go down for 4+ hours causing an email bounce!?! Or is <br class="">there something horribly wrong on my BIND caching DNS servers?<br class=""><br class=""></div></div></blockquote><div><br class=""></div><div>4 of 5 of these domains are on godaddy, the other has DNS handled by Westman.</div><br class=""><blockquote type="cite" class=""><div class=""><div class="">Perhaps they are slow, and I'm my BINDs are just not waiting long enough. <br class="">Is there a way to tell BIND to be more patient waiting for DNS packets to <br class="">come in?<br class=""><br class=""></div></div></blockquote><div><br class=""></div><div>Normally DNS has a multi-second timeout. I'm not sure of the technical details of how bind handles SRVFAILs but Bind does note which servers respond quicker and weights those. </div><br class=""><blockquote type="cite" class=""><div class=""><div class="">Maybe it's something regarding IPv6? (I'm doing this all in IPv4 and have <br class="">no current interest in 6. And I'm only looking for A records.)<br class=""><br class=""></div></div></blockquote><div><br class=""></div><div>If you don't have v6 addresses on things, generally they won't be asking for quad A records so I don't think this should be an issue for you.</div><br class=""><blockquote type="cite" class=""><div class=""><div class="">I have packet traces of the above sample commands when the lookups fail, <br class="">but I can't really figure out what it's doing, other than one boatload of <br class="">traffic for a tiny dns query. I can provide a trace privately on demand <br class="">if you think you can help.<br class=""><br class="">Even more odd, I never seem to have a problem with interactive things like <br class="">web browsing. If this was a problem with all domains, I should see this <br class="">in firefox all the time, but I don't. Maybe Firefox doesn't even obey <br class="">resolv.conf and does its own thing, or retries heavily itself?<br class=""><br class="">I also checked to ensure my iptables aren't dropping packets related to <br class="">this.<br class=""><br class="">Lastly, answers of "just use 8.8.8.8" aren't helpful because I also need <br class="">to handle dynamic local, and in some cases, external DNS (often with <br class="">multiple views), all in the same BIND/box (and I like uniformity across <br class="">boxes for ease of admin). Sure, I could try another resolver, but I see <br class="">no reason BIND can't be made to work, as it has for me for 20 years. And <br class="">if this is a BIND bug, I want to submit it to help solve it.<br class=""><br class="">Thanks guys!<br class="">_______________________________________________<br class="">Roundtable mailing list<br class=""><a href="mailto:Roundtable@muug.mb.ca" class="">Roundtable@muug.mb.ca</a><br class="">http://www.muug.mb.ca/mailman/listinfo/roundtable<br class=""></div></div></blockquote></div><br class=""><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><div class=""><br class="Apple-interchange-newline">Theodore Baschak - AS395089 - Hextet Systems<br class=""><a href="https://ciscodude.net/" class="">https://ciscodude.net/</a> - https://hextet.systems/<br class="">https://theodorebaschak.com/ - http://mbix.ca/</div></div><div class=""><br class=""></div><div class=""><br class=""></div></body></html>