June 2011
Burt Rosenberg
Until the summer of 2011, our DNS servers were diversified across the campus. This meant we were vulnerable to an entire-campus network outage. Until recently, that risk was acceptable. In particular, with our email servers on campus, a campus wide outage would stop our email as well — a much more serious concern operationally.
The department is experimenting with the cloud-based email service Google Apps. With cloud-based email, we are now in a position to demand higher availability, and we can build towards continued email functionality even during an entire-campus network outage. We need therefore to raise the bar on DNS availability.
To match cloud-level availability of email, we will join a cloud-base DNS service, namely Amazon Route53. This is paying service, based on usage, but I expect yearly costs to remain under $20.
Second, our existing network of DNS servers must be limited to DNS servers which we have administrative control. The primary-secondary discipline of a network of DNS servers is such that updates to the DNS information must originate at the one primary server and are propagated automatically to secondary servers. In the event of the loss of the primary server, the secondary servers cannot be updated by the primary-secondary approach.
During a prolonged outage, it might be necessary to update the DNS information: in particular, the update of MX records to redirect email towards a cloud-based email. Hence indiscriminately propagating DNS information to multiple secondaries can paradoxically lower availability by freezing outdated information.
The department has both internal and external DNS zones. All internal zones are unaffected by this project; and only one of the external DNS zones is affected. The external zones are cs.miami.edu and 89.31.192.in-addr.arpa. Only cs.miami.edu needs to be treated, as the reverse lookup domain is not obligatory for most functionality.
Since the internal domains are unaffected, no new administrative procedures are needed for internal domains.
The external domains will continue to be administrated as before, using Named, with new procedures for and isolated to Route53 administration. Route53 does not use the primary-secondary discipline. Hence, the new procedures will require individual updates to Route53 and Named.
Route53 is a cloud-deployed web service. It has two interfaces. It is queried using Bind (or tools such as dig) and that is how it is part of the DNS infrastructure. The other interface are XML documents submitted by https to an the Amazon URL https://route53.amazonaws.com/2011-05-05/{service}, where {service} selects for the service desired.
See: http://docs.amazonwebservices.com/Route53/latest/APIReference/
Route53 requires and Amazon Web Services (AWS) account. An account has been opened with the mail address amz-dns@cs.miami.edu. This AWS account has a associated with it an Access Key ID and a Secret Access Key. These are needed in the web requests as a sort of username/password pair. Although https is used to protect privacy, the Secret Access Key is never divulged in a straight forward manner, but is used to MAC sign requests.
This account is where billing occurs as well. The email amz-dns@cs.miami.edu would be used for support with Amazon for this product. We will share the password to this account. However, administration of the DNS database needs only the Access Key ID and Secret Access Key, and does not need to log onto the AWS account.
Various tools exist to manipulate the http interface. R53Fox is a graphical front end that allows inspection of the zone database and some limited changes.
The workhorse utility is the Perl script dnscurl. Dnscurl prepares the command line and executes curl including the special headers that authenticates the request, allowing change access to our zone data. Most changes are a POST of an xml file describing the change.
See: http://docs.amazonwebservices.com/Route53/latest/DeveloperGuide/
Dnscurl finds the AWS account Access Key ID and Secret Access Key in the file .aws-secrets. This file must be kept confidential.
Route53 uses xml. A GET request to a domain will return the information in xml. To delete, create, or change RR's requires preparing an xml document and using dnscurl to POST the document. The document returned by the GET is useful to fashion the document to POST.
A change is done by deleting the RR's and the creating the updated RR's. The POST works as a transaction. A document with multiple changes either succeeds entirely entirely or fails entirely. Therefore a change, which seems like a delete followed by a create, to the outside world seems like an update — it is impossible for the deletion to succeed but the create fail.
It is possible to hand edit the xml documents returned by a GET request, and submit these documents using dnscurl. It is easier to use the R53Fox GUI tool. Since this tool is built on Firefox technology, XUL and Javascript, it should run on any platform that has an xulrunner.
There are now three sources of DNS for cs.miami.edu. There is the primary/secondary public DNS hosted at the department, with various secondaries on campus. There is Route53, outsourced DNS that must be held synchronous with the on campus source. And there is the internal DNS, which has different information. In some cases, all three must be changed, and each has its own method to change.
An example where all three have to change is www.cs.miami.edu. This is a public RR with A record value 192.31.89.8 and is in the databases of by Route53 and the primary/secondary DNS. To update it in Route53 use R53Fox. Then log into pickett and change the master file and HUP named, remembering to increment the serial number so it can be picked up by secondaries.
We also use the external address internally. Even though www.cs.miami.edu is pickett, with internal IP 172.18.0.4, we want internal clients to browse www.cs.miami.edu the same as external clients. Therefore the internal database needs updating. Since we run dynamic DNS internally, to update use nsupdate on mcclellan.
[burt@mcclellan}: nsupdate > update add www.cs.miami.edu. 86400 IN A 192.31.89.8 > send > quit
I am looking for a student that is willing to take on as a project the automation of this triple work.
Last update: June 22, 2013.