Thursday, December 19, 2013

Torch... Thank you Carrie.... It's a perfect song....

My wife Carrie gave me this song.... She heals me in so many ways.
I'd never heard the song before, but she knows me so well.
Those of you who really know me, know the meaning behind this song for me.
Perfect song for who I am....
Thank you Carrie - I love you always... You are the best wife ... person ... that anyone could ever hope to know. Thanks for marrying me....

Ananis Morissette - Torch....

I miss your smell and your style and your pure abiding way
Miss your approach to life and your body in my bed
Miss your take on anything and the music you would play
Miss cracking up and wrestling our debriefs at end of day
These are things that I miss
These are not times for the weak of heart
These are the days of raw despondence
And I never dreamed I would have
To lay down my torch for you like this
I miss your neck and your gait and your sharing what you write
Miss you walking through the front door, documentaries in your hand
Miss traveling, our traveling and your fun and charming friends
Miss our Big Sur getaways and to watch you love my dogs
These are things that I miss
These are not times for the weak of heart
These are the days of raw despondence
And I never dreamed I would have
To lay down my torch for you like this
One step, one prayer, I soldier on
Simulating, moving on
I miss your warmth and the thought of us bringing up our kids
And the part of you that walks with your stick-tied handkerchief
These are things that I miss
These are not times for the weak of heart
These are the days of raw despondence
And I never dreamed I would have
To lay down my torch for you like this

Monday, December 09, 2013

This has special meaning to me. If you accept a lie, it's easy to just move on.

It's easy to move on if you accept a lie, without critical thinking.
It's easy to toss away love if you accept the hysteria that is around you.
It is easy to destroy that which you loved by accepting the imaginings of those who say that have your best interests at heart. 
It's easy to ignore the obvious, when you are cloaked in the thinking of the group.
It's easy to hurt who you love, if you wrap yourself in a cloak of hate.
If You Start Accepting EverythingIt's easy to find guilt and fault in the innocent, when the innocent is the only thing that you have to throw away.

It's easy to forget the kindness shown you, when others manipulate situations to make even that kindness look evil.

It's easy to ascribe evil to even the most noble of actions when you have wrapped yourself in a lie.

An untruth is still an untruth no mater how many people believe it.

It's easy to forget how much you were loved when your convinced that you were  never really truly loved.

Deception has an amazing power to destroy life and love. If one will but open their eyes to deception, it will often present itself in all it's glory. The question really is about the virtue of the one who finally opens their eyes to see. Once they see, will they close their eyes again to avoid the inevitable?  Will they close their eyes again in the fear and loathing that is now their comforter? Or will they become the better of this race and realize the deception and what it has done - taken away the love that they could have enjoyed forever.

Moving on.... it's a lazy way to end what could have been amazing if only you had really tried.

Saturday, October 19, 2013

Harry potter on betrail...

What that everyone would stand and fight the good fight rather than give in to the pressure of fear brought on by others....

“What was there to be gained by fighting the most evil wizard who has ever existed?" said Black, with a terrible fury in his face. "Only innocent lives, Peter!"
"You don't understand!" whined Pettigrew. "He would have killed me, Sirius!"
J.K. Rowling, Harry Potter and the Prisoner of Azkaban

Monday, October 07, 2013

Really... I think Battleship is a great movie!

Some people listen to music while they work, some people need quiet. I like to listen to movies and TV shows in the background. One movie I'm somewhat fond of is Battleship. Battleship really didn't do well in the US market, but did very well internationally. I have it loaded up in my DVR whenever I want to watch it I just throw it on the TV and work.

I've noticed that there are lots of critics who pan Battleship... I ask you though, with scenes like these, it can't be all that bad really - can it?

I mean, watch those 16-inchers punishing the Alien ships! I also think the tech that is imagined for the aliens is quite interesting.... Look at these little gremlins:

They are called "Shreaders" and they are nasty buggers. They even have a tail, dang!! Then there are the standard weapons that the aliens have on their ships which are so cool as you can see here:

These things look like big trash can's but blast, they are powerful weapons. Finally, there is the music.... Which you can listen to here:

Some awesome music!

So.... Explosions, not terrible acting, aliens, more explosions, all sorts of good stuff.... I'd love to see a Battleship 2!

Just sayin.....

Thursday, October 03, 2013

To my Carrie... a poem for you... :)

“It is only with true love and compassion that we can begin to mend what is broken in the world. It is these two blessed things that can begin to heal all broken hearts.”
Steve Maraboli, Life, the Truth, and Being Free 
To my Carrie ....

You mend my heart a little bit more each day.
When you met me, I bled blood everywhere,
and splattered my broken heart on the floor.
Hemorrhaging my life, grief was my companion.

Tears were my constant friend,
falling to the floor, puddles.
Walking in them, the splash of pain,
splotches of blood floated on the surface.

I found myself, slowly becoming smaller,
as sadness wore away more foundation.
I found that there was nothing that could fix,
that which seemed so forever a ship broken.

My first mate, left the ship,
but the crew stood behind me.
I felt solace in their attempt to keep me afloat,
until another mate came to be.

First mate know, I harbor no ill will.
It was a moment where the storms blew hard.
it was a time when nothing seemed right,
and you feared you might drown.

Then, the good crew had left the ship,
those who sailed with me for so long.
The ship was older it seemed,
and the crew was destined for greater things, as is the way of life.

Then, I was broken from my mooring by a fierce gale,
I'd not been tied there tightly enough, my fault.
The dock slowly collapsed in the storm and I followed,
the path of the gale, almost breaking up on the rocks it left me on.

And the heart of my hull ruptured,
and the blood of my life spilled free.
I sat, on the rocks, unsure of my fate,
for all had changed in that storm, and I was lost.

A broken heart is an interesting thing,
for some understand it's depth,
some wish to ignore it and hope it will go away,
some stand there, in shock and horror, fixated on it.

There are those who console without understanding,
and those who console without compassion.
There are those who console, without consoling,
and then, there are those like you, who just console in all.

My hull rotted and smelled of the garbage of my life,
stinking like trash cans standing in the sun.
Then, from nowhere was the slight wiff,
fresh air that come literally from above (Canada!).

So, I evolved in that fresh air,
from the rotting hull of a ship.
Into the man that I was,
smelling the air that surrounded me.

Swiftly the sweet air moved into my life,
taking the stench out, smelling like the fresh cut grass of spring,
and the flowers blooming on a warm, rainy day.

Sun and warmth returned to my life.

Then, in an instant it seemed, we were married.My life, my happiness, doubled, tippled and I was happy.
The garbage and it's stench were removed from my life.
Though once in a while, I could smell the dump far away.

Your love and compassion though would overshadow all,
and your heart took mine and stopped it's great fall.
Then just when, I thought it was at it's best.
You granted me a wish, that I thought was beyond belief.

Then the light of our love gave us more purpose.
A spring warmth to your great cleansing sun.
Even though demons try to destroy us,
We know what is right, and what is an illusion.
(to borrow from Pink Floyd)

And together our love will meet all challenges.
Though from time to time my hearts wounds still leak.
You are the power that heals each one, every time.
Your might propels my happiness.

We shall stand fast now - and become more as one.
Your love will serve mine, in happiness and fun.
My love will protect you, in all that there is.
We will be perfect together, and protect each other.

For there be rough waters ahead,
as life always has to offer.
Yet you are my lifeboat forever,
and you may seek your refuge in my hearts harbor.

I am always yours my dear Carrie,
you are the one I've always dreamed of.
You are magnificent and the grace that quells my pain,
even when it comes now, at the oddest of moments.

Gales, pain, life be damned.
When you are by my side, I can fight any fight.
I can walk the path to mending my heart,
though the path is long, with you it seems light.

For it took me a long time to walk that path of pain,
and the road back is long and full of traps.
Emotional traps, gremlins from the past,
all wait to capture me in their snares.

They do not know the pain I've known,
and they do not know that I will fight until I die.
They do not know how much I value truth,
how much I value my legacy, and will fight for it.

With you by my side, nothing is ever lost,
for you are the keeper of my soul.
The protecter of that which was once shattered,
in ways I didn't even really realize.

Getting to you was a rocky road,
and the road still has it's bumps.
As you sit by my side, we hold hands,
and drive into the sunset.

And we don't care how poor the road is,
holes and rocks may be waiting down the road.
In love and truth we drive confident,
that the end will always find us arm in arm.

The universe will say, in the end,
"Good job - well done".
and we shall enter into it's rest,
being  the example of love for all else.

And the universe will look at the blood I lost,
and it will weep just a little bit.
for the stories, even from the begining,
that I will tell by the comforting fires.

To those who are the comforters,
come and join me by the fire and let me thank you.
To those who were the tormenters,
come and join me by the fire and I shall forgive you.

Let those who lied, learn the truth.
Let those who besmirched me, learn that I am not that bad.
Let those who loved me, embrace me again.
Let those who I've missed, fill my heart to it's brim.

Children, all of whom I love dearly,
let them sit beside me so I many converse with them.
Grandchildren, who I await...
Let them play amongst those of all that come.

Come my family and friends, all, present and past.
Enjoy the fire and the feast with us.
We open our arms to you all,
and extend our love around you too.

And may my life's companion and sweet love,
the one who has made my heart and life sing sweetly.
Let her sit with me at the feast, surrounding the fire,
Let me sing her praises, and kiss her fondly.

Let me calm all her hurts and woes,
let me hold her close, and give her love.
Let us sit together, warm by the fire,
and embrace in an eternal kiss of true love. 

Life and great and terrible moments....

“I was blind and heart broken and didn't want to do anything and Gus burst into my room and shouted, "I have wonderful news!" And I was like, "I don't really want to hear wonderful news right now," and Gus said, "This is wonderful news you want to hear," and I asked him, "Fine, what is it?" and he said, "You are going to live a good and long life filled with great and terrible moments that you cannot even imagine yet!”
John Green, The Fault in Our Stars


Monday, September 23, 2013

On karma...

Please note - this is not a religious message. It is a statement of my personal beliefs. 

"I'm a great believer in karma, and the vengeance that it serves up to those who are deliberately mean is generally enough for me.
Beth Ditto"

I believe in karma myself. I think it's a natural by-product of our natures and the type of person and life we choose to live. I also believe the more you hurt someone, lie about them, attack someone and engage in or are associated with false claims, rumors or allegations that karma will not be your friend. 

Look inside yourself and you will find truth. For in truth, you can stare Karma in the eye and not be afraid. Steeped in lies and falsehoods, Karma will not smile at you in the least.

May Karma smile on you... :)

Wednesday, August 28, 2013

I'm creating a new blog for Oracle related stuff...

I've decided to spin off my Oracle related blog, separating it from my personal blog. You will find future Oracle related posts at this address:

It's new, and I don't have it completely setup yet  - but in the next few days you will find I will be starting to link to my Oracle posts over here and adding new information on Oracle Database 12c, and other Oracle related materials.

I will continue to post personal stuff, thoughts and commentary here... Just in case you like to read a radical rant or two. :)



Monday, July 29, 2013

12c New Features... A survey..

I'm working on some things right now related to Oracle Database 12c, and I'd like to do a little survey about your experience with upgrading to different versions of the Oracle Database. You can find the survey here. This is for DBA's only I'm afraid, not developers.  12c New Features content is coming as in more stuff on Exadata Patching and other goodies.

Sunday, July 28, 2013

Exadata Patching Part Five.... Patching the Compute Nodes

In the previous blog post we talked about patching the cell servers. Now, let's talk a little bit about updating the Compute nodes. To begin with, when you look closer at the contents of the QFSDP bundle patch you will find this set of files is included:

- Infrastructure/
|      --ExadataStorageServer/
|          -
|               -README-14522699.html
|               -README-16432033.html

Instantly I think this can get confusing, so let me explain. If you goto MOS and look up the first patch p14522699, you will find that this is the Storage Server Software Patch that you will use to update the Storage Servers (which I call the cell servers for some insane reason that I won't try to explain).

So, what is this other patch? What is  p16432033? While you might think this patch has something to do with the storage server (since it sits in a directory called ExadataStorageServer!) in fact it is the ISO image that can be used to update the compute nodes! I know, it's confusing but that is the way things are.

So, what is this patch you ask:

|          -README.txt

As it's under a directory you might think that it's the ISO for the compute node. Well, it is, in fact, the database patch for the GRID_HOME (GH) and ORACLE_HOME (OH) directories. So you will be forgiven for being a little confused.

All of these patches are clearly documented (wink) in MOS note 888828.1 (make this note your friend).

To further confuse things, the preferred way to patch your database server operating systems is not through the application of the ISO that is contained in the patch. The preferred way is to use the Oracle Unbreakable Linux Network (ULN) and YUM to download the most current OS release for the compute nodes.

The manual steps for updating a compute node are pretty involved. In an effort to reduce the overall effort and possibility of mistake, Oracle has created a script called the DB Node Update Utility ( You can find more information on this tool using MOS note 1553103.1.

Most of the Exadata patches that I do, for various reasons, have to be done manually installing the updates from the current ISO image. If this is something you are going to need to do, reference note 1473002.1 for more information on how to setup the YUM repository for local use.

The bottom line is that for updating the Database Nodes, you will use YUM to grab the needed files in a YUM repository (either the ULN repository or one you create yourself) and then update the compute nodes.

Finally, note 1553103.1 nicely summarizes the various steps that are involved in the update of a compute node. To quote from the note:

A shortened list of the functionality provided by the utility as follows:

  • Creates log file to track script execution and changes
  • Creates diag file with 'before patching' situation
  • Provides 'next steps'
  • Includes checks and workarounds for known issues
  • Creates and runs the backup utility 
  • Checks space requirements of /boot filesystem
  • Checks for left-over snapshots
  • Checks for required files and utilities
  • Includes 'check-only' option
  • Includes 'quiet mode'
  • Relinks all Database and Grid Infrastructure (GI) Homes
  • Enables / disables GI to stop/start
  • Checks for nfs/smbfs mounts and unmounts when possible
  • Validates provided media (zip, ISO, http, file://)
  • Validates user
  • Provides rollback option
  • Generates 'yum repo'-file
  • Validates yum.conf-file
  • Logs console history
  • Stops OSwatcher before patching
  • Unpacks media and helper zip-files
  • Stops db console and em agent
  • Checks for failed 'pvmove' actions
  • Checks for left over snapshots and failed backups
  • Supports multiple em agents
  • Checks for 3GB free space in "/"

I vote for automation! More coming soon!

Wednesday, July 24, 2013

Exadata Patching - Part Four...

So... you have downloaded the patch. Now what do you do. Good question.

Installing the Exadata patch sets *usually* involves the following:

1. Patch the cell servers.
2. Patch the Compute note OS.
3. Patch the GRID_HOME and ORACLE_HOME locations.
4. Run the database upgrade script on each database.

I say usually because there could be additional steps to perform, such as firmware updates, in any given patch set. There is also often updates to Enterprise Manager included in the patch sets, so you will want to consider those updates.

So, let's take a look at patching the cell servers in a bit more detail. In later posts I'll address the other steps.

I can't stress how important it is to be installing these patches from a location with reliable networking. While the internal Exadata network is very fast, redundant and reliable, it's an unfortunate truth that the networks outside the Exadata box may not always be the same. What you don't want is a slow or unreliable network connection that disconnects you in the middle of the the application of the patch. If your network is not reliable, you should consider other options such as the KDM, or directly connecting to the switch between the Exadata rack and your network if possible.
Patching the Cell Servers
The first thing we will want to do is patch the cell servers. Applying the cell server patch will update the entire cell server including OS updates, firmware updates and any other updates that are needed on the cell server (such as configuration related updates). This makes patching the cell servers much easier and it also ensures that the server images remain consistent.

Before you start to apply the cell server patch you will want to decide if you want to keep your Exadata rack up and available during the patch - which would require a rolling patch application, or if you want to take the rack down for about two hours to apply the patches.

The downside of doing a rolling patch is that you are looking at anywhere between and hour and two hours, per compute node, to apply the patch. Of course, the rack is up and available during this time, so the users don't notice anything. From an administrative point of view though this can represent a significant investment in time to upgrade the rack. If you are running a half-rack with 7 cell servers, for example, you are looking at anywhere from 7 to 14 hours to upgrade all of the cell servers. That's a long time for someone to sit and monitor the progress of the patch application!

The benefit of the rolling patch is clear though. If the patch fails, it only fails on one node, and that failure should not impact your availability. The loss of one cell server generally isn't something that's going to take the entire rack down. 

On the other hand, a non-rolling upgrade patches all of the cells at once. This has the benefit of shortening the time to upgrade the entire rack of cell servers - say 1 to 2 hours (I'd plan on 2 hours). On the other hand, upgrading all of the cell servers at once can be a bit of a frightening experience. This is because you are sitting there, hopeful, that the patch will apply successfully on all of the cells, that you have not made any mistakes and that unicorns fart rainbows.

If you can take the outage my personal preference is to always run the upgrade on one cell server (which allows the system to remain up) and then take the outage and upgrade the remaining cells all at once. I just like to see the patch apply successfully once so I get warm fuzzies, and generally I like to reduce the overall time it takes to apply patches for various reasons. Of course, if uptime is your principle goal, then you will want to do a rolling patch.

You can find the specific instructions on applying the current (July) patch to the cells on MOS. Note that you need to have access to Oracle MOS to access this note. Note that each patch has it's own install instructions, so you should review the patches to make sure that nothing has changed in the install process.

Cell Server Patching Prerequisites
While there are a number of steps that you will want to perform, there are some that I consider to be critical to applying the patch (at this time)? They are:

1. Make sure you can access the ILOM on each cell server. This is a critical step, so make sure you execute this on each ILOM.

2. I usually like to run the patches from one of the compute nodes. Determine which compute node you want to use to install the patch from and check ssh connectivity from that node to each of the cell servers. Really, if SSH wasn't working right then the Rack would probably be having problems anyway, but it's always a good idea to double check. The following command can be used to check for SSH connectivity (and user equivalence).

dcli -g cell_group -l root 'hostname -i'

dcli is a nice utility that provides the ability to run commands across a set of nodes via SSH. 
You might wonder what the cell_group business is about. This is a text file (usually in /home/oracle) that lists all the cell servers in the rack. Normally this file should have been created when the Exadata rack was installed.

Also, make sure you check the cyphers as indicated in the install documentation. 

3. Check the prerequisites that apply for the type of patch you are applying. For example, if you are doing a rolling patch you will need to adjust the disk_repair_time parameter for ASM. If you are patching all the nodes, you will be shutting down CRS and all associated services at this time.

4. While the patching program (patchmgr) will check for sufficient disk space, I like to manually check the available disk space on each cell server just to be sure.

Of course, this is just a few of the steps that you need to execute before you apply the patch. As I said before, the best thing in my mind to do is create a checklist that clearly calls out all the steps you intend to execute. While the documentation is good, it does contain a mix of instructions and I feel like it's a much easier and clearer process to follow a checklist first, referencing the documentation when required.

Applying the Patch
Once you have completed the prerequisites then it's time to patch the cell. To perform the upgrade you use the patchmgr utility. Usually you will do this from a compute node, while logged in as root. Oracle specifically says not to use the ILOM for applying patches, which makes sense if you think about it. :)

Once you are ready to patch the cells you generally use the patchmgr utility. The patchmgr utility provides an automated way of applying the patches to the storage cells.

When using the patchmgr utility you will first clean up any old patching related stuff, then you will run the prerequisite checks and then you will start your patching. Patchmgr will be in root of the directory where you unzipped the patch files too. The basic commands you would run are:

./patchmgr -cells cell_group -cleanup
This will clean up any previoius patches on all nodes in the cell_group group.
./patchmgr -cells cell_group -patch_check_prereq [-rolling]
This executes the prerequisite patch check. This will check the node(s) for space availability, current patch  compatibility and so on. Note that this will apply to all nodes on the cell_group list. If any errors appear when running the prerequisites, these should be corrected before proceeding and then you should run patchmgr with the cleanup and prereq check process, again.

./patchmgr -cells cell_group -patch [-rolling]
This executes the actual patch. Using the -rolling parameter will start a rolling patch upgrade. This will apply to all nodes on the cell_group list.

There have been times that I've had to fully path the cell_group location.

The biggest stress I find during the whole process is the rebooting of the cell servers. You really kind of find yourself praying that they will come back up. It can take time, so be patient.

Bottom LineThe bottom line is that the install of the patches is generally pretty straight forward process, but you should carefully follow the instructions provided by Oracle.

Next post, I'll talk about upgrading the Compute Node OS.

It's your birthday...

Happy birthday you....

Saturday, July 20, 2013

Exadata Patching - Part Three...

** If you find these posts helpful, please feel free to link my blog to you website, your blog or wherever it would make sense. Thanks! **

Part Three - Introduction and Disclaimer
So, we have discussed getting ready for patching. Let's talk a little bit about the general mechanics of patching. One note of caution. This blog post was written just as the July patch was released. While I've been through the readme, the directions and so on, I've not yet applied this patch. Also, since I don't have a TARDIS, I've not been privy the the future patches. The bottom line is that the patches might change, instructions for a given patch set might require you follow a different path than the one I'm going to lay out for you here. It's even possible that the method of patching will change. For example, Metalink document 888828.1 says that the patching frequency for Oracle Database 12c is yet to be determined. In fact, the current QFSDP does not patch Oracle Database 12c at all.

As with everything else, things change. I'll try to update my blog on a regular basis if something changes in a major way.

Also, there is a new tool called OPlan. In these posts, I'm going to assume that your not going to use OPlan, but it might be time to start putting it to work in kind of a test phase. OPlan provides a more automated method of helping you figure out what you need to do to patch your system, including dealing with dependencies and the like (note, I've not used it yet, but this is what I've read). However, there are some current restrictions to OPlan that generally precludes me from using it (such as multiple ORACLE_HOME directories). So, for now, I'm going to let OPlan mature  but I'll probably use it with the July patch in kind of a Beta stage and see how well it works. 

About the QFSDP
As I mentioned in my previous post, the QFSDP contains a number of different patches. Patching Exadata is not a fire and forget exercise. There are a lot of steps that you have to follow. Eventually I think that OPlan will simplify those steps, and also I know that the folks that are working on Exadata are always working to simplify the patching process. I have to say, the folks working on Exadata are amazing and as a DBA I am thoroughly impressed with it. Even though I work in a sales organization, I'm a technical guy and I hate selling. I hate marketing hype and hoopla. With respect to Exadata, I've been deeply, technically, involved in several POV's and I can tell you that Exadata is the real deal.

Slightly Off Course for a Moment
The only thing I don't like about it is that it works so well that it can mask really bad SQL and database designs... On the other hand, even good designs have performance issues and in the age of trying to reduce costs and consolidate, Exadata is an amazing machine. Frankly, I hate using the word product, and the word machine seems to black boxish to me. Still, I've seen to do some fantastic things and there is so much opportunity to then go modify code, design and applications beyond that. I'd throw in the rest of the Exa stack, Exalogic, Exalitics and so on, but this is about Exadata patching.

Finding the Current Patch
The July 2013 QFSDP is found by going to Metalink document 16784347. If you are reading this later on, you can find a link to the current patch in Metalink Document 888828.1, as I showed you earlier.

The first thing to notice on the patching document (in this case 16784347), is that  you need to select the correct release and platform as seen here:

More about the QFSDP
Once  you have done that you can click on the Read Me button to read the Readme file associated with the correct patch. If you look at the read me, it gives you a basic low-down on what is contained in the patch bundle. For example, it indicates that the patch bundle consists of three main, high level, components. These are:
  • Infrastructure
  • Database
  • Systems Management
In each of these components are contained sub-components. For example in the database component there is:
  1. Quarterly Database Patch for Exadata (Jul 2013 - - contains updates to Oracle Database and Oracle Clusterware software ( QDPE Jul 2013)

  2. OPatch - contains software to install/deinstall the 'Database component' patches (

  3. OPlan - contains software that facilitates the 'Database component' patch installation process by providing you with step-by-step patching instructions specific to your environment (
The readme also contains the instructions on preparing to install the QFSDP after you have downloaded it.

Along with the ability to access the readme file, you can also download the bundle patch set.

What's in the QFSDP Download?
Contained within the QFSDP download are a number of individual patch sets. For example, for the infrastructure part of the patch set you get the following files all packed in the zip file:

|- README.txt
|- README.html
|- Infrastructure/
|      --ExadataStorageServer/
|          -
|               -README-14522699.html
|               -README-16432033.html
|      --ExadataDBNodeUpdate/
|          -README.txt
|      --InfiniBandSwitchSoftware/
|          -1.3.3-2/
|               -README.txt
|      --SunRackIIPDUMeteringUnitFirmware/
|          -1.04/
|               -README.txt

Note that there are directory structures for each component in the patch. We get the Storage cell update, the DBNode Update, the Infiniband Switch software and the PDUMetering Unit Firmware.

Note that the patches are all cumulative. So, if your behind by a quarter, just apply the most recent patch and you will be good.

Don't Fall For Over Patching
Also, just because a component is included in the patch, does not mean it needs to be applied. For example, the Infiniband switch patch (version 1.3.3-2) has been around for a long time (as you can see - again - in document 888828.1). How do you know what component versions are in your system already? Look at your Exacheck output. In that output, it will tell you all of the current versions installed on your system. Here is an example of that output (modified slightly for the purposes of space savings):

Note in this output (which is for just one node within an Exadata Machine)  that the current versions of each component is listed.  Compare this to the list in the patch readme. For example, if your Infiniband switch software is already at 1.3.3-2, then there is absolutely no need to install the infiniband part of this patch since you can see (near the bottom of the report) that the Infiniband switch is at version already.

Always the Exceptions
There are always exceptions and the like that you have to deal with. After getting a good feeling for what is in the patch set and what is in your Exadata Rack, go back and make sure you read document 888828.1 again, carefully. Traverse EACH relative link in that document (there are a great number) for any information that might supersede the README's contained in the patch bundle.

There have been times, after a patch has been released, that additional notes are included on the Metalink document. These notes might have you use a different version of OPatch, or perhaps they will indicate that you need to  use a different individual patch set that that which is in the QFSPD.

I'll give you an example. In an earlier patch set they missed including the OS patch for the compute nodes. There is a note that indicates that you need to download a specific patch and apply it as a part of the overall QFSDP patch process. Here is an example of that specific note which applied to the January and April patch set:

There are a lot of complex dependencies to patching, as you can imagine. OPlan seeks to remedy some of that and as I said earlier I plan on at least "trying it out" this go around to see how well it manages those dependencies for me.

Battle Stations - The Critical Issues
You will also notice note 1270094.1 listed in ML document 888828.1. This note is critically important as it's lists all Exadata Critical Issues (indeed that is the name of the document). Notice that there is an item on the critical list for the Infinitude switch, along with an associated patch that you need to apply to the current IB version. So, you might think you are all patched up, but until you read this, you would possibly be wrong. In the end, not only do you need to make sure that is installed, but you need to make sure that patch 12373676 has been applied. It can get confusing, especially if you have not been keeping up with your patches.  The fact is that this patch was released a long time ago (way back in 2011) to the odds are that it's been applied. In fact, most of the issues listed here are older ones (the latest is Feb-2013 I believe).

Downloading the Patch
Speaking of things that have changed, this is one of them.
The way the latest patch set has been setup for distribution has changed in this release. I don't know if this will be the way of the future, but it's new. Previously the patch was distributed as a single ZIP file. Now, as of the July patch set (which is a hefty 4.2GB image when fully extracted) is sub-divided into several 1.5GB zip files that you will download and then install according to the install documentation. I'm not sure if this is the plan for distributing the future patches or not.

That's it for now. More coming in part four. I originally thought this would be a three part blog post but  it's seems that it will be more.... I expect to post part four in the next couple of days...

Thursday, July 18, 2013

Eeeegads... the patch is out and I'm behind - Exadata Patching Part II (edited - display issues fixed)

The Exadata July patch set is out! I feel like I'm behind on my Blog posts!

In my last post I talked about Exacheck, and how you needed to run and review Exacheck.  If you find items you need to correct, then now is the time to do it! Often, you can click on the view link of the report and it will take you to a more detailed view of the issue, and sometimes it will even include information on how to correct the problem.

One piece of advice, after you whittle down the items on the Exacheck that you feel like need to be addressed before you patch. If this is your first few times doing this, go ahead and open an SR with Oracle and send that Exacheck to support.  When opening the SR, list the items you think need immediate attention for patching and those that you think can wait and get them to double check your thinking, and provide resolution information on the critical items.

So, having reviewed your Exacheck, it's time to start looking at the actual patch.

If you check Metalink Note 888828.1 (that is a lot of 8's) you will find that the new QFSDP for Exadata has been released. QFSDP stands for (drum roll) Quarterly Full Stack Download Patch. Within this note you will find a section called the "Latest Releases and Patching News" - here is the current copy of this section:

As you can see, each QFSDP  has a patch number associated with it. That patch number has a link to the actual patch, that you can download. Additional information is also on the page where the link exists, such as which bugs are fixed in the patch set.

So, what should you review in preparing to apply a patch? There are a TON of notes that you should review before you try to do your first Exadata patch:

1461240.1 - Exadata Database Machine Software and Hardware Maintenance Planning Guide
This  note provides basic information on Engineered Systems patching. Strongly recommended first time read.

1270094.1 - Exadata Critical Issues - You should read this document prior to any patching exercise, and really, I'd read it regularly to keep up with critical issues on Exadata.

1473002.1 - Exadata YUM Repository Population and Linux Database Server Updating

You will want to review that last note really carefully, because YUM will come into play when installing your quarterly patch on the Compute nodes. Should you be downloading the update online or applying the update locally, you will need to get YUM setup in one way or another. So you might want to be working on that now.

So, what is in a QFSDP? Well, we will talk about that in my next Blog post.

For now, read the documentation, start to think about patching and we will talk more about patching in part three....

Saturday, July 06, 2013

Exadata - Preparing for the quarterly patch - Part One

If you are using Exadata you know it's coming. If all goes as scheduled, the next Exadata QFSDP (Quarterly Full Stack Download Patch) will be out later this month. If your running on Exadata you should know that it's very important to keep your patches up-to-date. That being the case, I thought I'd spend a few moments discussing what you can do to prepare for these patches.

The first thing you can do, and you should do this regularly, is to run an Exacheck report. Oracle Metalink Note 1070954.1 describes how to download the Exacheck report. The included user guide and best practices documents will describe how to install Exacheck and run it.

It's important to note that new versions of Exacheck are released fairly regurarly, and you want to download, install and run the most current version of Exacheck before you install the QFSDP. It's really a good practice to run Exacheck reports on a regular basis. This is a good method of cross-checking your Exadata system for any failures, errors and to note any configuration changes that might have slipped through.

I've run Exacheck before every patch that I've applied. In many cases we have found problems on the Exacheck that needed to be corrected before we could apply the patch. While many Exadata environments have ASR (phone home) setup for their Exadata machines, many do not for various reasons. If you are working with an Exadata machine and it's not setup with ASR, then you need to be extra vigilant reviewing your systems health. The beauty of Exadata is it's redundancy, but that redundancy can also mask a variety of problems that occur. If you don't catch and correct those problems, then things can and do eventually fail. If you find problems on the day you want to be applying patches, you might find your patching schedule thrown out the window as you scramble to repair a bad disk, replace a bad cable, or any number of other small things that can go wrong.

If you look on the Oracle Exadata Assessment Report, which is one of the reports that Exadata produces, you will see a message similar to this one:

Note! This version of exachk is considered valid for 31 days from today or until a new version is available

So, even Exacheck will warn you how much longer  you have to run the current version of Exacheck. 

So, you have your exacheck report. What are the things to look for before you install the patch sets? There are a number of things on the Exacheck that can get flagged and some are worth paying attention too, and others are less than important.

First, notice the disclaimer on the report that says:

NOTE : exachk is only one part of the MAA Best Practices recommendation methodology. My Oracle Support "Oracle Exadata Best Practices (Doc ID757552.1)" should be reviewed thoroughly as it is the driver for exachk and contains additional operational and diagnostic guidance that is not programmed within exachk.

So, Exacheck is just part of an overall set of best practices that Oracle recommends. Notice that it says you should review the note. It is not the purpose of Exacheck to remove the need for the DBA to think - rather it gives the DBA things to think about. Just because the report flags something does not mean it's bad and just because the report does not flag something does not mean things are good. At the end of the day, you have to use your experience to interpret the report and decide what is important in your environment and what is an exception.

One output from the Exacheck execution will be the Oracle Exadata Assessment Report. At the top of this report there will be a score allocated to your Exacheck system. This score is supposed to represent the overall health of the system. While this score is a good guide, it's a bit like the database hit ratio in that it can be misleading and it can also cause you to get target fixation. I do not recommend that you focus on lowering the score. Rather, I recommend that you review the result, determine which ones are critical and really apply to your environment and then correct those problems. I've seen people pay way to much attention to this score, and craft a goal that this number be as near 100 percent as possible rather than analyze the result and use the score as a general metric that you build reasonable thresholds around. Don't get score fixation.

The Exacheck reports contains various checks on the Exadata stack including:
  1. ASM check
  2. Cluster wide check
  3. Database check
  4. ORACLE_HOME check
  5. OS Check
  6. Patch check
  7. SQL check
  8. SQL parameter check
  9. Storage server check
  10. Infiniband switch check

These checks include a variety of things such as version checks, best practices checks and so on. So, the question is, what is truly important when it comes to Exacheck and making sure everything is ready for the patch to be applied.

Generally I will ignore database setting failures out of the gate. This isn't because these are not important, but generally a database parameter, on a working database, that is not set in alignment with best practices is not going to impact the application of the patches. So, let's say that you saw the following on your Exacheck report:

FAILDatabase CheckDatabase control files are not configured as recommendedAll Databases

Would this failure require resolution before you applied your patches. Probably not. Not that this isn't something to look and and figure out if your databases are secure, but this should not stop you from applying a database patch.

Now, what about this one:

WARNINGOS CheckFree space in root(/) filesystem is less than recommended.

Note that this is a warning and not a failure, which seems a bit odd to me. What would you do if you saw this error message? Would you apply your patches? No, of course not. Since one of the things we are going to patch is the compute node, we want to make sure that there is plenty of space on the root file system. Also, we are going to be backing up the boot partition on these boxes, and we want the boot partition as clean as possible before we start backing it up, or patching it. So, in my mind, in spite of the fact that this is reported as just a warning, it's really something important to look into.

What do you do if you get a result and you don't know how to react to it? Please, please - contact Oracle Support and ask them! Don't assume it's ok, or that it's not important. If you don't understand it, take steps to understand it and then determine if you need to resolve it.

So, what are the issues that I worry the most about?

  1. Network issues. Cabling, configuration, all are important.
  2. ILOM issues. If an ILOM is flaky, you have a real problem.DO NOT PATCH IF YOU HAVE AN ILOM ISSUE. PERIOD!
  3. Storage cell issues. The storage cells should all be consistently without error. If an error on the storage cell layer is flagged, it's something to pay attention to. Me personal rule, don't patch anything unless the storage cells are 100% healthy without question.
  4. Any database server OS checks that fail due to configuration, infiniband or other infrastructure related errors.
  5. Any database server verify-topology checks that fail.
  6. Any database errors that surface that indicate the versions are different, incorrect or other types of errors.
  7. If the Exacheck report has the following included in it: WARNING! The data collection activity appears to be incomplete for this exachk run. Please review the "Killed Processes" and / or "Skipped Checks" section and refer to "Appendix A - Troubleshooting Scenarios" of the "Exachk User Guide" for corrective actions.
All of this being said, I think the Exacheck reports says it best when it states:

FAIL, WARNING, ERROR and INFO finding details should be reviewed in the context of your environment.

Finally, once you have run Exacheck, reviewed it's findings and corrected those that need correction, it's a good idea to document the final Exacheck results for future reference.

So, now you have your Exacheck ready to go and all problems that need attention corrected! We are one step closer to being ready to apply that patch!

Well, I hope this blog entry was helpful. In the next entry I'll discuss some Exadata results I've seen in the past and the impacts of those results. Then, we will talk about other Quarterly bundle patch related topics. Your comments about your experience with Exacheck are most welcome!  Have a great weekend!

Friday, July 05, 2013

12c New Features Contribution - Want to get published?

This is related to my new Oracle Database 12c New Features book which will be out soon! I'm offering you a chance at getting your name in lights (sort of) and to be immortalized in this new book.  Note there is no compensation, award, prize or anything else guaranteed here. Just opportunity.

We are near the end of the actual writing of the 23c New Features book. My friend, Tom Kyte, has offered to add some running commentary which we are calling "Tom says". In the process, Tom is also offering up some technical comments. I had finished up Chapter 6, which includes coverage on a new feature called Pattern Matching. To be honest and embarrassed a bit, Tom replied that he didn't like my example. In fact, he kind of hated it. There were many reasons why, and I agreed with 90% of them.

So, after feeling really depressed I thought... Well, back to the drawing board with my example. Then it struck me that this is actually an opportunity.

So, I'm giving YOU an opportunity to be published. In the comments to this post (and please copy me in email at, respond with the following by midnight 7/15:

  1. Write a query using the new Oracle Database 12c Pattern Matching feature.
  2. Include sample output demonstrating the query.
  3. Provide all DDL required to re-create the query including the creation of the table (and any other objects), and creation of all data in that table.
  4. The query should be SIMPLE for a first time user to follow, but not so simple that it can be replicated with a SQL statement easily not using the pattern matching feature.
All comments with entries will be published on the blog here as long as they are not bogus, breaking copyrights (so don't just cut and paste from the Oracle documentation or just change it so slightly that you think no one will notice) or are otherwise in bad form.

One awesome entry will be selected for publication (assuming a get an awesome entry - I do not guarantee that any entry will be selected).

I'll contact you if you are selected and then we can work out the legal details. Note that if you are selected you will not be compensated for your entry or any additional follow-up work that you might be asked to do. You will have to warrant that it's your own and give Oracle Press rights to publish it under their copyright. Your name will appear in the acknowledgements of Oracle Database 12c New Features for all your jealous co-workers to oooohhh and aaawwwwhhhh at!

So, get cracking and learn about and use Oracle Database 12c Pattern Matching!

P.S. -- Here is a starting place....

Wednesday, July 03, 2013

Oracle GoldenGate Book is out!

Lots of things Oracle Wise are going on.

I just saw that my new book on Oracle GoldenGate has shipped on Amazon. I'll say that this book was one of the toughest I've written for a lot of reasons. It was written during a time of great personal upheaval.... it has seen my life in the greatest ups and downs that I've ever experienced. This is part of the reason that it took so long to get the book printed. However, we worked hard to keep the book current and relevant - we added a new chapter at the end to cover the latest 11g features of GoldenGate. I'm sure that we will see GoldenGate 12c at some point in time in the not to distant future (I have absolutely no internal knowledge of this fact what-so-ever, this is a pure guess based on the fact that Oracle Database 12c has now been released).

I'm working on the last of the writing on my Oracle Database 12c New Features book! I just finished the initial write of the final "chapter" of the book and am now preparing to go back through all the chapters and test them against the final production code. We write the NF book mostly on the Beta code of a new database version, but we ALWAYS go through every chapter, every code example, and check it against the production code. I've noticed that other 12cv books tend to use the beta and that the early ones out don't seem to check their stuff against production. This has lead to examples that don't work, mis-named parameters and a host of other problems with the books that make them less than useful. I think this approach works best, but I'd be interested in your experiences.

More later on all that was promised and more.

edit: Scott made me realize that I probably did not give the people who helped me with the GoldenGate book and the New Features book too! I don't have all the names at hand, and I don't want to miss anyone, so I'll do that on a follow-on post.

Wednesday, June 26, 2013

Oracle Database 12c... Exadata Patching.....

Oracle Database 12c!
It's out and about now! As I posted yesterday, Oracle Database 12c has come out in all it's glory. Some of the new features that you might find of interest (this is a partial list):

1. Multi-Tennant Databases
2. The ability to move databases across most platforms, using transportable tablespaces, regardless of endian byte format.
3. Native outer join enhancements.
4. PL/SQL object "white" lists.
5. New dbms_utility.expand_sql_text function. (replace views with tables in your queries!)
6. Use a PL/SQL function in your WITH clause!
7. Temporal validity - no, a TARDIS is not included with Oracle Database 12c, but you can define the temporal validity of a given set of data based on a set of dates. If your designs include a lot of effective_date, end_date types of columns, then you will be interested in this one!
8. Identity columns. Also the ability to define a sequence as the default value for a column.
9. Default values for columns when there are NULL values inserted.
10. VARCHAR2, NVARCHAR2 and RAW data types can now accommodate up to 32k.
11. Recover a table using RMAN! That's right, I said a TABLE!!!!
12. Cascading TRUNCATE statements.
13. Adaptive query optimization and SQL plan management.  Get better execution plans at run-time!

That's just a glimpse, and there are many, many more features. Once you have looked through Oracle Database 12c I'd love to hear what your favorite features are. Many more posts on Oracle Database 12c will be coming! Keep an eye out!

Also,  my book Oracle Database 12c New Features will be coming out soon! It's available on Amazon for pre-orders now!

Exadata Patching
It's coming! The next Exadata Quarterly Full Stack Download Patch (QFSDP) is due in July! I've been to a number of customers installing the April QFSDP and one common thing I've found is lots of questions and confusion on what to patch, how to patch and so on. In the next few weeks I'll be providing some thoughts on applying the QFSDP's, what to do before, during and after the patches to get you ready for the patching season.

Right now, let me suggest that you run Exacheck on your system. In almost every client I've been too in the last quarter I've found at least one Exacheck item that needed to be addressed before patches could be applied. If you are on Platinum support, then this might be less of an issue. If you are not on Platinum support or if you are not setup for ASR, this is a critical must-do before you patch your Engineered System.

What are your big Exadata Patching headaches? Leave me a comment and let's talk about it!

If you shut up truth and bury it under the ground, it will but grow, and gather to itself such explosive power that the day it bursts through it will blow up everything in its way.

Oracle 12c and other things!

Well, there it is! Oracle 12c is out and about! You can find it at

Exciting! In the next few weeks this blog will start talking about 12c and we will also start talking about Oracle Exadata. I've been away for a while for a number of reasons, but the time has come to resurface here.

My new book, Oracle Database 12c New Features is currently under way and will be available as quickly as we can get it out. I've also got a new book on Oracle GoldenGate coming out in the next few days!

I hope you will enjoy my newly re-invigorated blog!


For someone special. From Dr. Who:

...You know this is inevitable. And you are forgiven. Always and completely forgiven....
Subscribe in a reader