Saturday, July 20, 2013

Exadata Patching - Part Three...

** If you find these posts helpful, please feel free to link my blog to you website, your blog or wherever it would make sense. Thanks! **

Part Three - Introduction and Disclaimer
So, we have discussed getting ready for patching. Let's talk a little bit about the general mechanics of patching. One note of caution. This blog post was written just as the July patch was released. While I've been through the readme, the directions and so on, I've not yet applied this patch. Also, since I don't have a TARDIS, I've not been privy the the future patches. The bottom line is that the patches might change, instructions for a given patch set might require you follow a different path than the one I'm going to lay out for you here. It's even possible that the method of patching will change. For example, Metalink document 888828.1 says that the patching frequency for Oracle Database 12c is yet to be determined. In fact, the current QFSDP does not patch Oracle Database 12c at all.

As with everything else, things change. I'll try to update my blog on a regular basis if something changes in a major way.

Also, there is a new tool called OPlan. In these posts, I'm going to assume that your not going to use OPlan, but it might be time to start putting it to work in kind of a test phase. OPlan provides a more automated method of helping you figure out what you need to do to patch your system, including dealing with dependencies and the like (note, I've not used it yet, but this is what I've read). However, there are some current restrictions to OPlan that generally precludes me from using it (such as multiple ORACLE_HOME directories). So, for now, I'm going to let OPlan mature  but I'll probably use it with the July patch in kind of a Beta stage and see how well it works. 

About the QFSDP
As I mentioned in my previous post, the QFSDP contains a number of different patches. Patching Exadata is not a fire and forget exercise. There are a lot of steps that you have to follow. Eventually I think that OPlan will simplify those steps, and also I know that the folks that are working on Exadata are always working to simplify the patching process. I have to say, the folks working on Exadata are amazing and as a DBA I am thoroughly impressed with it. Even though I work in a sales organization, I'm a technical guy and I hate selling. I hate marketing hype and hoopla. With respect to Exadata, I've been deeply, technically, involved in several POV's and I can tell you that Exadata is the real deal.

Slightly Off Course for a Moment
The only thing I don't like about it is that it works so well that it can mask really bad SQL and database designs... On the other hand, even good designs have performance issues and in the age of trying to reduce costs and consolidate, Exadata is an amazing machine. Frankly, I hate using the word product, and the word machine seems to black boxish to me. Still, I've seen to do some fantastic things and there is so much opportunity to then go modify code, design and applications beyond that. I'd throw in the rest of the Exa stack, Exalogic, Exalitics and so on, but this is about Exadata patching.

Finding the Current Patch
The July 2013 QFSDP is found by going to Metalink document 16784347. If you are reading this later on, you can find a link to the current patch in Metalink Document 888828.1, as I showed you earlier.

The first thing to notice on the patching document (in this case 16784347), is that  you need to select the correct release and platform as seen here:

More about the QFSDP
Once  you have done that you can click on the Read Me button to read the Readme file associated with the correct patch. If you look at the read me, it gives you a basic low-down on what is contained in the patch bundle. For example, it indicates that the patch bundle consists of three main, high level, components. These are:
  • Infrastructure
  • Database
  • Systems Management
In each of these components are contained sub-components. For example in the database component there is:
  1. Quarterly Database Patch for Exadata (Jul 2013 - - contains updates to Oracle Database and Oracle Clusterware software ( QDPE Jul 2013)

  2. OPatch - contains software to install/deinstall the 'Database component' patches (

  3. OPlan - contains software that facilitates the 'Database component' patch installation process by providing you with step-by-step patching instructions specific to your environment (
The readme also contains the instructions on preparing to install the QFSDP after you have downloaded it.

Along with the ability to access the readme file, you can also download the bundle patch set.

What's in the QFSDP Download?
Contained within the QFSDP download are a number of individual patch sets. For example, for the infrastructure part of the patch set you get the following files all packed in the zip file:

|- README.txt
|- README.html
|- Infrastructure/
|      --ExadataStorageServer/
|          -
|               -README-14522699.html
|               -README-16432033.html
|      --ExadataDBNodeUpdate/
|          -README.txt
|      --InfiniBandSwitchSoftware/
|          -1.3.3-2/
|               -README.txt
|      --SunRackIIPDUMeteringUnitFirmware/
|          -1.04/
|               -README.txt

Note that there are directory structures for each component in the patch. We get the Storage cell update, the DBNode Update, the Infiniband Switch software and the PDUMetering Unit Firmware.

Note that the patches are all cumulative. So, if your behind by a quarter, just apply the most recent patch and you will be good.

Don't Fall For Over Patching
Also, just because a component is included in the patch, does not mean it needs to be applied. For example, the Infiniband switch patch (version 1.3.3-2) has been around for a long time (as you can see - again - in document 888828.1). How do you know what component versions are in your system already? Look at your Exacheck output. In that output, it will tell you all of the current versions installed on your system. Here is an example of that output (modified slightly for the purposes of space savings):

Note in this output (which is for just one node within an Exadata Machine)  that the current versions of each component is listed.  Compare this to the list in the patch readme. For example, if your Infiniband switch software is already at 1.3.3-2, then there is absolutely no need to install the infiniband part of this patch since you can see (near the bottom of the report) that the Infiniband switch is at version already.

Always the Exceptions
There are always exceptions and the like that you have to deal with. After getting a good feeling for what is in the patch set and what is in your Exadata Rack, go back and make sure you read document 888828.1 again, carefully. Traverse EACH relative link in that document (there are a great number) for any information that might supersede the README's contained in the patch bundle.

There have been times, after a patch has been released, that additional notes are included on the Metalink document. These notes might have you use a different version of OPatch, or perhaps they will indicate that you need to  use a different individual patch set that that which is in the QFSPD.

I'll give you an example. In an earlier patch set they missed including the OS patch for the compute nodes. There is a note that indicates that you need to download a specific patch and apply it as a part of the overall QFSDP patch process. Here is an example of that specific note which applied to the January and April patch set:

There are a lot of complex dependencies to patching, as you can imagine. OPlan seeks to remedy some of that and as I said earlier I plan on at least "trying it out" this go around to see how well it manages those dependencies for me.

Battle Stations - The Critical Issues
You will also notice note 1270094.1 listed in ML document 888828.1. This note is critically important as it's lists all Exadata Critical Issues (indeed that is the name of the document). Notice that there is an item on the critical list for the Infinitude switch, along with an associated patch that you need to apply to the current IB version. So, you might think you are all patched up, but until you read this, you would possibly be wrong. In the end, not only do you need to make sure that is installed, but you need to make sure that patch 12373676 has been applied. It can get confusing, especially if you have not been keeping up with your patches.  The fact is that this patch was released a long time ago (way back in 2011) to the odds are that it's been applied. In fact, most of the issues listed here are older ones (the latest is Feb-2013 I believe).

Downloading the Patch
Speaking of things that have changed, this is one of them.
The way the latest patch set has been setup for distribution has changed in this release. I don't know if this will be the way of the future, but it's new. Previously the patch was distributed as a single ZIP file. Now, as of the July patch set (which is a hefty 4.2GB image when fully extracted) is sub-divided into several 1.5GB zip files that you will download and then install according to the install documentation. I'm not sure if this is the plan for distributing the future patches or not.

That's it for now. More coming in part four. I originally thought this would be a three part blog post but  it's seems that it will be more.... I expect to post part four in the next couple of days...

No comments:

Subscribe in a reader