How to setup a private DNS for your virtual cluster

One of the challenges I faced recently was building a virtual cluster based on Oracle 11g Release 2 on top of Oracle Enterprise Linux (OEL) running inside VMware server. Although I have an existing virtual Oracle 11g Release 1 cluster, I decided to build a new one in order to be able to teach both versions of the Oracle University RAC courses I teach. Also, my existing virtual cluster runs OEL4 and for 11gR2 I needed OEL5. I’ll save the story about building the actual cluster for another posting, because I need to perform additional research first, but I’d like to share my solution on how to get around the DNS requirements without making changes to the Oracle corporate DNS server.

Background information

Oracle 11gR2 Clusterware has many new features and two of them require adding resource records to DNS. The features in question are:

• Single Client Access Name (SCAN)
• Grid Naming Service (GNS)

Adding resource records to DNS is something that is handled by network administrators in most organizations. However, I consider this a DBA2.0 skill and I think you should gain knowledge in this area if you don’t already have it. A good start would be reading DNS and BIND published by O’Reilly.

Before continuing I will introduce some basic DNS vocabulary. The Domain Name System (DNS) is a client-server architecture which purpose is to translate names into ip-addresses and vice-versa. The server is called the DNS server and the client is called the resolver. The most well-known implementation is the Berkeley Internet Name Domain (BIND) software where the nameserver process is called “named”. As the name suggests DNS is based on domains which are divided into administrative regions which are called zones. A nameserver can be either the master or slave for a zone. The data, called resource records, for a zone is stored in so-called zone files which are maintained on the master nameserver for a particular zone. Slave servers retrieve their zone file from the master server in a process known as a zone transfer. The BIND nameserver is configured by the /etc/named.conf file, and the zone files are usually stored in /var/named.

Making changes to the Oracle corporate DNS server for my virtual cluster is unlikely to happen, nor advisable to do so, and therefore I decided to run a DNS server inside my virtual cluster. This sounds easier than it actually is. The problem is that nameservers are linked to form a tree, thus the nameserver above me links to my nameserver and maybe mine links to a nameserver underneath me. This linking between nameservers is called delegation. Since I am trying to avoid adding resource records to the Oracle corporate nameservers there will be no delegation to my nameserver from the Oracle nameservers. This isn’t a problem as long as clients who wish to connect to my cluster talk to my nameserver and not to the Oracle nameservers. Another problem is that, because of firewalls, my nameserver cannot talk to nameservers outside the corporate Oracle network which means that my nameserver can only resolve names which are within my own zones. Therefore my nameserver should forward any request that it cannot resolve by itself to another nameserver that can.

The implementation

The DNS implementation that I created for my virtual cluster has the following features:

• I use example.com as my domain.
• The master DNS server for example.comwill be on my first RAC node.
• A slave DNS server for example.com will be on my second RAC node.
• All DNS lookups that fall outside example.com will be forwarded.
• Each node uses its local nameserver as its primary.
• Each node uses the other node’s nameserver as its secondary.
• DNS security will not be implemented.

The first step in implementing the above setup is to install the bind rpm either using the rpm or yum command.

The next step is to configure the DNS server. Without the right tool this can be quite challenging. I found h2n To be the best tool to handle this job for me. Basically h2n generates all DNS configuration and zone files based on /etc/hosts or any other file with a similar structure. Mine is called hosts.dns and is shown below:

192.168.40.141  el5n1.example.com
192.168.40.142  el5n2.example.com
#
192.168.40.101  el5n1-vip.example.com
192.168.40.102  el5n2-vip.example.com
#
192.168.40.110  el5n-cluster-scan.example.com
192.168.40.111  el5n-cluster-scan.example.com
192.168.40.112  el5n-cluster-scan.example.com
#
192.168.180.135 el5n1-priv.example.com
192.168.180.136 el5n2-priv.example.com

My RAC nodes are called el5n1 and el5n2, and the IP addresses for the public network interface are shown on lines 1 and 2. Their Virtual IP addresses are on lines 4 and 5. Lines 7, 8 and 9 have the SCAN IP addresses. Finally lines 11 and 12 show the private interface IP addresses.

I created a ‘dns’ sub-directory in my home-directory and stored the above file in there to act as my dns setup area.

The next step is to let h2n generate the DNS configuration and zone files for me based on the above hosts file. I stored the required h2n command line arguments in a .conf file called h2n.conf, to prevent myself from entering them over and over again.

-H hosts.dns
-d example.com
-n 192.168.40
-n 192.168.180
-u root@example.com
-W /var/named
-M
-y
+O forwarders { 192.168.40.2; };
+O forward only;
-s el5n1
-s el5n2
-z 192.168.40.141

Line 1 tells h2n which host file to use, line 2 specifies the domain name, lines 3 and 4 specify the subnets in use. Line 5 is to specify who to contact if something is wrong with my DNS setup. Line 6 specifies where the working directory for the name server is. Lines 7 and 8 are to suppress the generation of MX records and to select a particular serial number format. Lines 9 and 10 are options that h2n places in the named.conf file. Lines 11, 12 and 13 tells h2n which machines act as nameservers and that I want h2n to generate config files for a slave nameserver that transfers its zones from the master DNS server at the specified IP address.

Now that I have the files to drive h2n, the actual generation of the DNS configuration file is straightforward.

$ h2n
Initializing new database files...
Reading host file `hosts.dns'...
Writing database files...
Generating boot and conf files...
Checking NS, MX, and other RRs for various improprieties...
Done.

There is one file h2n cannot generate and that is db.cache Which I downloaded and stored in my dns setup directory. The generated named.conf file should be copied to /etc and the db.* should be copied to /var/named. The setup is now complete and the next step is to start the nameserver and to check its status.

$ sudo service named start
Starting named: [  OK  ]

$ sudo service named status
number of zones: 4
debug level: 0
xfers running: 0
xfers deferred: 0
soa queries in progress: 0
query logging is OFF
recursive clients: 0/1000
tcp clients: 0/100
server is up and running
named (pid  8656) is running...

The above output shows that my nameserver started successfully and that it runs normally. The next step is to automate the nameserver startup each time the operating system starts using the chkconfig command as shown below:

$ sudo chkconfig named on

$ sudo chkconfig --list named
named           0:off   1:off   2:on    3:on    4:on    5:on    6:off

The DNS server configuration is now complete but before I can test my nameserver I first need to configure the resolver. This step involves editing /etc/resolv.conf and add the IP address of my two nameservers to the nameserver directives. To save typing fully qualified host names I use the search directive to specify my local domain name.

search example.com
nameserver 0.0.0.0
nameserver 192.168.40.142

The final step is to test my master nameserver by resolving several names and IP addresses inside my own domain and by resolving something outside my own domain to test the forwarding part of my setup.

$ host el5n1
el5n1.example.com has address 192.168.40.141

$ host 192.168.40.141
141.40.168.192.in-addr.arpa domain name pointer el5n1.example.com.

$ host el5n-cluster-scan
el5n-cluster-scan.example.com has address 192.168.40.110
el5n-cluster-scan.example.com has address 192.168.40.111
el5n-cluster-scan.example.com has address 192.168.40.112

$ host prutser.wordpress.com
prutser.wordpress.com is an alias for lb.wordpress.com.
lb.wordpress.com has address 72.233.2.58
lb.wordpress.com has address 72.233.2.59
lb.wordpress.com has address 74.200.243.251
lb.wordpress.com has address 74.200.243.253
lb.wordpress.com has address 76.74.254.123
lb.wordpress.com has address 76.74.255.123

According to the above output my master DNS server is working fine and I can now proceed to configure the slave DNS server on my second RAC node. All configuration and zone files for the slave nameserver were already generated by h2n and all I needed to do was to copy the conf.sec.save to /etc/named.conf, and the db.cache and db.127.0.0 to /var/named on my second RAC node. Before starting the slave nameserver I needed to add ENABLE_ZONE_WRITE=yes to the /etc/sysconfig/named file to allow it to save the transferred zone file to disk. The steps to start the slave nameserver manually and to automate the startup after a reboot are the same as for the master nameserver. The /etc/resolv.conf is of course different because it needs the IP address of the master nameserver.

Wrapping up

To add a new host to my DNS configuration all I need to do is to add the hostname and IP address to the hosts.dns file and to let h2n generate the zone files for me. Thereafter I need to copy the new zone files to /var/named on my first RAC node and tell named to re-read the zone files from disk using the rndc command. To ease this process I wrote the following Makefile which will do all the work for me after I edit the hosts.dns file.

named.conf:     hosts.dns h2n.conf
        h2n
        sudo cp named.conf /etc
        sudo cp db.* /var/named
        sudo rndc reload

The above described DNS configuration allows running my own nameservers inside my virtual RAC cluster giving me the possibility to explore the Oracle11gR2 Clusterware. Also, I gained a little bit of experience in setting up and maintaining DNS.
-Harald