The Dutch Prutser's Blog

By: Harald van Breederode

  • Disclaimer

    The views expressed on this blog are my own and do not necessarily reflect the views of Oracle.
  • Subscribe

  • Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 238 other followers

Archive for the ‘Linux’ Category

Ksplice in action

Posted by Harald van Breederode on September 24, 2011

On July 21, 2011 Oracle announced that it has aquired Ksplice. With Ksplice users can update the Linux kernel while it is running, so without a reboot or any other disruption. As of September 15, 2011 Ksplice is available, at no additional charge, to new and existing Oracle PremierSupport customers on the Unbreakable Linux Network (ULN).

Updating the Linux kernel while it is running sounded like an impossible mission to me, and I was really keen to see this in action with my own “eyes” ;-) Yesterday I gave it a try and in this posting I will share my first exprerience with you.

The installation of Ksplice is a very easy process which took only a few minutes and can be performed while the system is up and running. It does however require an ULN account for obvious reasons ;-)

Before updating my system lets have a look when the system was booted, which kernel it is running and show you that I have an Oracle database running while the kernel is being updated to a new version:

$ who -b
         system boot  2011-09-23 18:52
$ uname -r
2.6.32-200.16.1.el5uek
$ pgrep -lf smon
6037 ora_smon_v1120
 

The above output shows that my system is running a 2.6.32-200.16.1.el5uek kernel. The “-uek” indicates an Oracle Unbreakable Enterprise Kernel which is a pre-requisite for using Ksplice on Oracle Linux.

And now, lets update the currently running Linux kernel to the latest version using Ksplice:

$ sudo uptrack-upgrade -y
The following steps will be taken:
Install [694jrs5f] Clear garbage data on the kernel stack when handling signals.
Install [zfm9vkzx] CVE-2011-2491: Local denial of service in NLM subsystem.
Install [gxqj9ojz] CVE-2011-2492: Information leak in bluetooth implementation.
Install [hojignhn] CVE-2011-2495: Information leak in /proc/PID/io.
Install [fa05bhhk] CVE-2011-2497: Buffer overflow in the Bluetooth subsystem.
Install [04wcg4oc] CVE-2011-2517: Buffer overflow in nl80211 driver.
Install [xjzxf6c1] CVE-2011-2695: Off-by-one errors in the ext4 filesystem.
Install [oqz3q8m2] CVE-2011-1576: Denial of service with VLAN packets and GRO.
Installing [694jrs5f] Clear garbage data on the kernel stack when handling signals.
Installing [zfm9vkzx] CVE-2011-2491: Local denial of service in NLM subsystem.
Installing [gxqj9ojz] CVE-2011-2492: Information leak in bluetooth implementation.
Installing [hojignhn] CVE-2011-2495: Information leak in /proc/PID/io.
Installing [fa05bhhk] CVE-2011-2497: Buffer overflow in the Bluetooth subsystem.
Installing [04wcg4oc] CVE-2011-2517: Buffer overflow in nl80211 driver.
Installing [xjzxf6c1] CVE-2011-2695: Off-by-one errors in the ext4 filesystem.
Installing [oqz3q8m2] CVE-2011-1576: Denial of service with VLAN packets and GRO.
Your kernel is fully up to date.
Effective kernel version is 2.6.32-200.19.1.el5uek

Note: Although the product is called Ksplice, the service it provides is known as uptrack.

The result of running the uptrack-upgrade command is that my system is now running kernel version 2.6.32-200.19.1.el5uek and it happened without a reboot or even stopping the running Oracle database! The output also shows that updating the running kernel occurred by installing small chunks of code corresponding to each patch that was applied to the kernel source code when the new kernel version was put together.
The output below shows that the system was not rebooted nor that the running database was restarted.

$ who -b
         system boot  2011-09-23 18:52
$ pgrep -lf smon
6037 ora_smon_v1120
$ uname -r
2.6.32-200.16.1.el5uek

It may be a bit confusing that uname –r still reports kernel version 2.6.32-200.16.1.el5uek while in reality the kernel version is 2.6.32-200.19.1.el5uek. According to the documentation this is expected behaviour and there is an uptrack-uname command available to report the kernel version that is actually running as shown below:

$ uptrack-uname -r
2.6.32-200.19.1.el5uek

In case you want to know which updates were applied to the running kernel the uptrack-show command is your friend:

$ sudo uptrack-show
Installed updates:
[694jrs5f] Clear garbage data on the kernel stack when handling signals.
[zfm9vkzx] CVE-2011-2491: Local denial of service in NLM subsystem.
[gxqj9ojz] CVE-2011-2492: Information leak in bluetooth implementation.
[hojignhn] CVE-2011-2495: Information leak in /proc/PID/io.
[fa05bhhk] CVE-2011-2497: Buffer overflow in the Bluetooth subsystem.
[04wcg4oc] CVE-2011-2517: Buffer overflow in nl80211 driver.
[xjzxf6c1] CVE-2011-2695: Off-by-one errors in the ext4 filesystem.
[oqz3q8m2] CVE-2011-1576: Denial of service with VLAN packets and GRO.

Effective kernel version is 2.6.32-200.19.1.el5uek

If, for whatever reason, you want to remove the updates that were applied to the running kernel, it is good to know that this can also be performed without a reboot or any other service disruption by running the uptrack-remove command.

$ sudo uptrack-remove -y --all
The following steps will be taken:
Remove [oqz3q8m2] CVE-2011-1576: Denial of service with VLAN packets and GRO.
Remove [xjzxf6c1] CVE-2011-2695: Off-by-one errors in the ext4 filesystem.
Remove [04wcg4oc] CVE-2011-2517: Buffer overflow in nl80211 driver.
Remove [fa05bhhk] CVE-2011-2497: Buffer overflow in the Bluetooth subsystem.
Remove [hojignhn] CVE-2011-2495: Information leak in /proc/PID/io.
Remove [gxqj9ojz] CVE-2011-2492: Information leak in bluetooth implementation.
Remove [zfm9vkzx] CVE-2011-2491: Local denial of service in NLM subsystem.
Remove [694jrs5f] Clear garbage data on the kernel stack when handling signals.
Removing [oqz3q8m2] CVE-2011-1576: Denial of service with VLAN packets and GRO.
Removing [xjzxf6c1] CVE-2011-2695: Off-by-one errors in the ext4 filesystem.
Removing [04wcg4oc] CVE-2011-2517: Buffer overflow in nl80211 driver.
Removing [fa05bhhk] CVE-2011-2497: Buffer overflow in the Bluetooth subsystem.
Removing [hojignhn] CVE-2011-2495: Information leak in /proc/PID/io.
Removing [gxqj9ojz] CVE-2011-2492: Information leak in bluetooth implementation.
Removing [zfm9vkzx] CVE-2011-2491: Local denial of service in NLM subsystem.
Removing [694jrs5f] Clear garbage data on the kernel stack when handling signals.

All the previously applied updates are taken out, in reverse order, which basically reverts the system back to its original state. The output below shows that this indeed happened without a reboot or stopping the running Oracle database:

$ who -b
         system boot  2011-09-23 18:52
$ pgrep -lf smon
6037 ora_smon_v1120
$ uname -r
2.6.32-200.16.1.el5uek
$ uptrack-uname -r
2.6.32-200.16.1.el5uek
$ sudo uptrack-show
Installed updates:
None

Effective kernel version is 2.6.32-200.16.1.el5uek

Cool, isn’t it? I am impressed!

Please read this Ksplice technical paper for some background information on the Ksplice technology.

Please keep in mind that Ksplice will only update the running kernel in memory and does not install a new kernel RPM. It does re-apply the updates automatically after a system reboot and will also check for new updates on a regular basis. Ksplice can download and install new updates automatically whenever they become available ensuring your kernel is always up-to-date!
-Harald

Posted in Linux | Leave a Comment »

How to setup a private DNS for your virtual cluster

Posted by Harald van Breederode on January 26, 2010

One of the challenges I faced recently was building a virtual cluster based on Oracle 11g Release 2 on top of Oracle Enterprise Linux (OEL) running inside VMware server. Although I have an existing virtual Oracle 11g Release 1 cluster, I decided to build a new one in order to be able to teach both versions of the Oracle University RAC courses I teach. Also, my existing virtual cluster runs OEL4 and for 11gR2 I needed OEL5. I’ll save the story about building the actual cluster for another posting, because I need to perform additional research first, but I’d like to share my solution on how to get around the DNS requirements without making changes to the Oracle corporate DNS server.

Background information

Oracle 11gR2 Clusterware has many new features and two of them require adding resource records to DNS. The features in question are:

  • Single Client Access Name (SCAN)
  • Grid Naming Service (GNS)
Adding resource records to DNS is something that is handled by network administrators in most organizations. However, I consider this a DBA2.0 skill and I think you should gain knowledge in this area if you don’t already have it. A good start would be reading DNS and BIND published by O’Reilly.

Before continuing I will introduce some basic DNS vocabulary. The Domain Name System (DNS) is a client-server architecture which purpose is to translate names into ip-addresses and vice-versa. The server is called the DNS server and the client is called the resolver. The most well-known implementation is the Berkeley Internet Name Domain (BIND) software where the nameserver process is called “named”. As the name suggests DNS is based on domains which are divided into administrative regions which are called zones. A nameserver can be either the master or slave for a zone. The data, called resource records, for a zone is stored in so-called zone files which are maintained on the master nameserver for a particular zone. Slave servers retrieve their zone file from the master server in a process known as a zone transfer. The BIND nameserver is configured by the /etc/named.conf file, and the zone files are usually stored in /var/named.

Making changes to the Oracle corporate DNS server for my virtual cluster is unlikely to happen, nor advisable to do so, and therefore I decided to run a DNS server inside my virtual cluster. This sounds easier than it actually is. The problem is that nameservers are linked to form a tree, thus the nameserver above me links to my nameserver and maybe mine links to a nameserver underneath me. This linking between nameservers is called delegation. Since I am trying to avoid adding resource records to the Oracle corporate nameservers there will be no delegation to my nameserver from the Oracle nameservers. This isn’t a problem as long as clients who wish to connect to my cluster talk to my nameserver and not to the Oracle nameservers. Another problem is that, because of firewalls, my nameserver cannot talk to nameservers outside the corporate Oracle network which means that my nameserver can only resolve names which are within my own zones. Therefore my nameserver should forward any request that it cannot resolve by itself to another nameserver that can.

The implementation

The DNS implementation that I created for my virtual cluster has the following features:

  • I use example.com as my domain.
  • The master DNS server for example.comwill be on my first RAC node.
  • A slave DNS server for example.com will be on my second RAC node.
  • All DNS lookups that fall outside example.com will be forwarded.
  • Each node uses its local nameserver as its primary.
  • Each node uses the other node’s nameserver as its secondary.
  • DNS security will not be implemented.

The first step in implementing the above setup is to install the bind rpm either using the rpm or yum command.

The next step is to configure the DNS server. Without the right tool this can be quite challenging. I found h2n To be the best tool to handle this job for me. Basically h2n generates all DNS configuration and zone files based on /etc/hosts or any other file with a similar structure. Mine is called hosts.dns and is shown below:

192.168.40.141  el5n1.example.com
192.168.40.142  el5n2.example.com
#
192.168.40.101  el5n1-vip.example.com
192.168.40.102  el5n2-vip.example.com
#
192.168.40.110  el5n-cluster-scan.example.com
192.168.40.111  el5n-cluster-scan.example.com
192.168.40.112  el5n-cluster-scan.example.com
#
192.168.180.135 el5n1-priv.example.com
192.168.180.136 el5n2-priv.example.com

My RAC nodes are called el5n1 and el5n2, and the IP addresses for the public network interface are shown on lines 1 and 2. Their Virtual IP addresses are on lines 4 and 5. Lines 7, 8 and 9 have the SCAN IP addresses. Finally lines 11 and 12 show the private interface IP addresses.

I created a ‘dns’ sub-directory in my home-directory and stored the above file in there to act as my dns setup area.

The next step is to let h2n generate the DNS configuration and zone files for me based on the above hosts file. I stored the required h2n command line arguments in a .conf file called h2n.conf, to prevent myself from entering them over and over again.

-H hosts.dns
-d example.com
-n 192.168.40
-n 192.168.180
-u root@example.com
-W /var/named
-M
-y
+O forwarders { 192.168.40.2; };
+O forward only;
-s el5n1
-s el5n2
-z 192.168.40.141

Line 1 tells h2n which host file to use, line 2 specifies the domain name, lines 3 and 4 specify the subnets in use. Line 5 is to specify who to contact if something is wrong with my DNS setup. Line 6 specifies where the working directory for the name server is. Lines 7 and 8 are to suppress the generation of MX records and to select a particular serial number format. Lines 9 and 10 are options that h2n places in the named.conf file. Lines 11, 12 and 13 tells h2n which machines act as nameservers and that I want h2n to generate config files for a slave nameserver that transfers its zones from the master DNS server at the specified IP address.

Now that I have the files to drive h2n, the actual generation of the DNS configuration file is straightforward.

$ h2n
Initializing new database files...
Reading host file `hosts.dns'...
Writing database files...
Generating boot and conf files...
Checking NS, MX, and other RRs for various improprieties...
Done.

There is one file h2n cannot generate and that is db.cache Which I downloaded and stored in my dns setup directory. The generated named.conf file should be copied to /etc and the db.* should be copied to /var/named. The setup is now complete and the next step is to start the nameserver and to check its status.

$ sudo service named start
Starting named: [  OK  ]

$ sudo service named status
number of zones: 4
debug level: 0
xfers running: 0
xfers deferred: 0
soa queries in progress: 0
query logging is OFF
recursive clients: 0/1000
tcp clients: 0/100
server is up and running
named (pid  8656) is running...

The above output shows that my nameserver started successfully and that it runs normally. The next step is to automate the nameserver startup each time the operating system starts using the chkconfig command as shown below:

$ sudo chkconfig named on

$ sudo chkconfig --list named
named           0:off   1:off   2:on    3:on    4:on    5:on    6:off

The DNS server configuration is now complete but before I can test my nameserver I first need to configure the resolver. This step involves editing /etc/resolv.conf and add the IP address of my two nameservers to the nameserver directives. To save typing fully qualified host names I use the search directive to specify my local domain name.

search example.com
nameserver 0.0.0.0
nameserver 192.168.40.142

The final step is to test my master nameserver by resolving several names and IP addresses inside my own domain and by resolving something outside my own domain to test the forwarding part of my setup.

$ host el5n1
el5n1.example.com has address 192.168.40.141

$ host 192.168.40.141
141.40.168.192.in-addr.arpa domain name pointer el5n1.example.com.

$ host el5n-cluster-scan
el5n-cluster-scan.example.com has address 192.168.40.110
el5n-cluster-scan.example.com has address 192.168.40.111
el5n-cluster-scan.example.com has address 192.168.40.112

$ host prutser.wordpress.com
prutser.wordpress.com is an alias for lb.wordpress.com.
lb.wordpress.com has address 72.233.2.58
lb.wordpress.com has address 72.233.2.59
lb.wordpress.com has address 74.200.243.251
lb.wordpress.com has address 74.200.243.253
lb.wordpress.com has address 76.74.254.123
lb.wordpress.com has address 76.74.255.123

According to the above output my master DNS server is working fine and I can now proceed to configure the slave DNS server on my second RAC node. All configuration and zone files for the slave nameserver were already generated by h2n and all I needed to do was to copy the conf.sec.save to /etc/named.conf, and the db.cache and db.127.0.0 to /var/named on my second RAC node. Before starting the slave nameserver I needed to add ENABLE_ZONE_WRITE=yes to the /etc/sysconfig/named file to allow it to save the transferred zone file to disk. The steps to start the slave nameserver manually and to automate the startup after a reboot are the same as for the master nameserver. The /etc/resolv.conf is of course different because it needs the IP address of the master nameserver.

Wrapping up

To add a new host to my DNS configuration all I need to do is to add the hostname and IP address to the hosts.dns file and to let h2n generate the zone files for me. Thereafter I need to copy the new zone files to /var/named on my first RAC node and tell named to re-read the zone files from disk using the rndc command. To ease this process I wrote the following Makefile which will do all the work for me after I edit the hosts.dns file.

named.conf:     hosts.dns h2n.conf
        h2n
        sudo cp named.conf /etc
        sudo cp db.* /var/named
        sudo rndc reload

The above described DNS configuration allows running my own nameservers inside my virtual RAC cluster giving me the possibility to explore the Oracle11gR2 Clusterware. Also, I gained a little bit of experience in setting up and maintaining DNS.
-Harald

Posted in Linux, Oracle | 17 Comments »

Should I switch to VirtualBox?

Posted by Harald van Breederode on March 22, 2009

Over the last two weeks several students advised me to switch from VMware Server to VirtualBox mainly because of better performance. So far I wasn’t unsatisfied by the performance of VMware, but if there is something freely available that is better I would be stupid to not give it a try.

Being a trainer I learned that reading documentation isn’t going to hurt you thus I first downloaded the manual, and after converting the PDF file to a Word document for better accessibility, I read through it and I was quite happy about the features I read about. Thus my first impression was very positive.

So yesterday I decided to give it a try and downloaded the software. After making a backup of my laptop I installed it. This went without any accessibility issues because the usage of a standard Windows installer which is quite accessible. During the installation the network was brought down, which was announced by the installer, but it failed to come up afterwards. Therefore the required registration process failed also. After a reboot things turned back to normal and I managed to finish the registration process although buttons were announced as edit fields. This raised my doubts about the accessibility.

Next I tried to build a new virtual machine using the GUI interface and this is where I gave up because this thing is almost totally inaccessible! Just as in the registration wizard buttons are announced as edit fields and many other GUI elements are not read at all by my screen reader. Also the menus in the menu bar don’t read as they should and multi-page dialogs don’t seem to respond to control+tab. Lucky enough the de-installation process ran smooth without screwing things up ;-)

So the answer to my question “Should I switch to VirtualBox?” is NO! I can’t verify if the performance is indeed better, but I do know that the accessibility of VMware is way better (although not perfect) than VirtualBox and that is the #1 feature I need. I know that VirtualBox can be managed using a command line interface, but overall I consider this less optimal than my current VMware environment. I rather have something slower but accessible than something quicker but inaccessible.
-Harald

Posted in Accessibility, Linux | 15 Comments »

 
Follow

Get every new post delivered to your Inbox.

Join 238 other followers