What does a Great White flopping in a boat have to do with databases?

Greetings! I am the proud expecting father of several kidney stones in August. Yes, in August they are going to clean me out of a number of stones all over the Freeman internal acreage. I can hardly wait.

I saw this picture:


and believe it or not, it made me think of databases. This boat was trolling around with scientists doing shark study when Jaws here leaped out of the water, and settled right where you see it. You can see the story here.

So, does this make you think of databases? No....? Your thinking this is making me think about how fast I could get to the front of the boat. I don't blame you.... I'm sure someone said, "We are going to need a bigger boat"

But what if I suggested this was an Oracle database example. How about the idea of elastic scalability? Freeman, are you hopped up on your pain meds? No! I think this is a perfect idea of what elastic scaling is all about.

So, your database server (the boat) is humming along doing what it was *designed* to do... chumming water, chasing sharks, tagging them, catching an occasional red snapper, whatever. Your ship is built for this mission, it's designed for the load.

Then, in one moment there is this BIG FRIGGING FISH laying on your boat. This is the sudden rush of users. Maybe your a state agency and it's election day, maybe you are some gossip rag that just got sued or you are some database vendor reporting incredible earnings... whatever the story, all of a sudden this big fish just plopped into your neatly tuned system and you realize, it's not going to scale.

In this case, the big shark caused all sorts of damage, causing the boat to need to be towed into port. Do you want your system to have to be towed into port just because it could not scale, and rapidly. This is where true architecture, configuration, *thinking* and planning come into play. Sure, maybe today you are dealing with small, 2 foot long sharks which, even if they did flop up on your boat, won't cause much of a problem (but the teeth do still hurt). But what about the future? What about the unknown? How prepared are you for that? How likely is it?

The is the beauty of RAC I think. Even if your using RAC One Node, or just clustering a single machine for the moment, the ability to easily scale out is powerful. Yes, it might be a little expensive and not everyone is going to need that power, but it is something to consider for mission critical applications/databases.

Anyway.... that's my fish tale and I'm sticking to it.

P.S. Amazingly - the shark LIVED......

The Holistic DBA -- Number Five - Communication - Part Four

** Note: As I have posted this my poor Doctor has informed me I have three kidney stones…. Ugh! So…. It might be a while before I post Part five… Sorry…. I'm afraid surgery awaits me. Gads....**

If you will recall in my last post I provided you with a list of what I consider to be the priorities of a holistic DBA. These were:

1. Be able to reliably, consistently and efficiently backup and recover the databases and all database in accordance with SLA, RPO and RTO’s.
2. Ensure database uptime with respect to all SLA, RPO and RTO’s.
3. Ensure all databases are secure.
4. Monitor all databases in a consistent and reliable manner.
5. Communicate in an effective manner.
6. Help users to help themselves.
7. Work to understand and correct bad behaviors with respect to the organizations data policies.
8. Work to understand and correct bad behaviors with respect to the organizations data designs.
9. Work to understand and correct bad behaviors with respect to the organizations application designs.
10. As befits the organization, your abilities, and time keep up with current data related technologies.

I have already discussed numbers 1-4 previously and now I want to move onto number 5…

Communication. When we went to college or high school, more than a few of us laughed at the notion that we needed a class in communication. I mean, we knew how to talk, didn’t we? We took it for the “easy A” and sighed our way through the instructor as he dolled on. You may have skipped class often, relying on your memory and textbook to get you through, and it did…. Oh how wrong you were my friends. While the academic side of communication may be a lot for us to muster (and just like 10046 tracing, most of us really don’t need to master the academic side) our ability to communicate is critical. It is, in the end, what leads to our failure or our success.

Case in point….

I once knew a DBA who shall remain anonymous. A critical system went down, boom, crash with all the accompanying bells and whistles. Now, because of the date and time and that some month end processing had just occurred, the business needed the database restored to July, 1, 2011 (all names, times and places are changed to protect the innocent). The business communicated the date, the DBA communicated the date back and started the restore. All seemingly went well, the database was up and restored to 6/1/2011. However the users began to report inconsistencies in the data. Yes, most certainly some data was not restored and was missing. Panic ensued, some reason or other was conjured up for the failure and another restore commenced. Again the restore was successful and we though happy day, who let the dogs out, woot! Until the call came that the data was again, completely wrong. Now, I’ve given you just enough information to figure out why we were wrong. Can you see it? I was called in because it was becoming a panic situation. I looked at the restore code, called the DBA doing the restore and pointed out the mistake.

So, what was the communication failure here? Was there one? The customer communicated the right date to restore, and the DBA communicated the date back to the customer. It was all seemingly well communicated. However, there was a failure in communication here (as opposed to a DBA failure in my mind) and a failure of process. What were those?

First the failure to communicate was in not cross checking the date that he read back to the customer (in one format) with the date he used in the actual restore and recover command (in a different format). Note that the customer said July, 1 and our poor DBA (using scripts, again the bane of existence) entered 6/1/2011 instead of 7/1/2011. A simple mistake to make, a hard mistake to catch sometimes when the pressure is on and the customer is like “HEY, GET THIS THING BACK UP!”.

The communication failure here was also a process failure. I’m not going to dive into the dynamics of stress under fire, but our poor DBA was under duress to get this database back up and running. To his credit (he really is a good communicator) he reported the problem down the line. In the end, I think the failure was MY failure as his direct manager (who was technically capable) I did not double check his restore script. I didn’t assign someone to work with him during this crisis. I don’t see this as a failure of the DBA but rather a failure of what should be a mandatory oversight process during a critical outage. The failure to communicate was mine. The failure to get the database restored on time was mine. The buck stops here, with me. I know, it’s cliché, but it’s the way I feel. How do you feel when a mistake is made? Do you try to understand it, do you take ownership of it or in your mind is it just one of those things that happens? In my mind, the latter choice is an utter failure to communicate, with yourself.

Let’s dig deeper into this communication thing. Look at the remaining priorities #6 through #10. Is it not obvious that each of these require that you communicate effectively? Don’t each of those priorities (which I think become critical enterprise wide priorities) require effective communication? What is required to be an effective communicator? There are many websites and books that will walk you through the basics of communication but let me tell you what I think makes for an effective communicator:

One note – this list applies to both public, meeting and private speaking – Some items may have more importance depending on the audience you are talking too.

Your speech - You have got to learn how to speak correctly. This does not mean you need to sound like a snot nosed Harvard law professor. It does mean you need to know how to properly conjugate verbs, how to avoid double negatives and other such things. I don’t claim to be the expert in this area (in fact I’m not) but I think it’s profoundly important to higher levels of success. Don’t use your point-of-birth-origin as an excuse to not speak the tongue correctly. You’re not the Larry the Cable Guy, you are a professional DBA. Be that.

Your vocabulary - If you want to sound like you know what you are talking about, improve your vocabulary. Pepper it with words that add meaning to your thoughts. Understand the current technical jargon (in concert with priority #10) that is related to the topic you are speaking about. Learn what a thesaurus is and use it when you write. Writing is a great way to improve your vocabulary, even technical writing.

Don’t communicate to death by Email – Be an explorer and get out there and meet people! Get your butt up and go find the person and talk to them. Follow-up with an email if you think it’s important. Get updates by email if you like. The point is, in the process of communication, we need to learn to be personable. We need to learn to get along with people. We need to learn that being irritable all the time makes people shy away from us. Learn to go find and meet all the people that make a difference in the organization. Make it a goal, say once a day, to meet someone new, and find out what they do. Learn their name, learn their skills and where they fit in the organization. Knowledge is a powerful thing, and putting a face to a name is important.

Be credible - Understand that if we constantly espouse ideas that are ridiculous that we will lose credibility. However, don’t be afraid to take a chance and let an idea fly. Just be prepared to defend it and to back down if you are found to be in the wrong. Don’t let pride get in your way, rather let your professionalism and a touch of humility and a dusting of persistence be your guide to when it’s time to say “I was wrong”. You will lose very little (if any) credibility if you admit you were wrong. You will lose a lot of credibility if you stomp off and refuse to accept that you were in error.

Pick your moment – Now with respect to #4, you might well be right but the moment was wrong. Perhaps something is going on that is not making the person capable of really listening to your point-of-view. Perhaps it’s your approach. Perhaps it’s any of the items on the list. The thing is, we need to pick the moments to discuss various topics. An idea that you have that is amazing, brilliant but a little risky might fly right after your group has had a great quarter. That same idea might well get summarily dismissed if your organization is struggling to meet its SLA’s. Which in fact, implies you might be skipping the priorities before the communication priority.

Your tone - Your tone is critical and you need to gauge it carefully when you speak. Use your tone to create drama, to create impressions of important with respect to a certain topic or to re-assert everyone’s attention on yourself and not the pizza that just entered the room.

Know what you are talking about - I’ve sat in so many meetings when the pro-ported professional pro-DBA espoused some obnoxious, totally wild, incorrect statement with respect to Oracle and everyone just sat there and nodded their heads. What would you do? Would you correct him? Would you take issue if his plan was wrong? If you don’t know what you are talking about, then you won’t have a clue what to say. If you are not past #1,2,3,4 and moving forward into the later priorities, you are going to have nothing much to add to the conversation. Learn your trade, and learn it well.

I’ll include one key, and I think important, but to that last statement. In fact, you don’t need to know, and cannot know everything. What you do need to know, and must learn to do is figure out how to collect information and understanding quickly, efficiently and understand if that information is reliable. You simply can’t know everything. You can know how to find out about a great many things in a few clicks. You can discover existing standards, tools and technologies quite quickly, if you have the understanding on how to do it.

Know your lines - Know what you’re going to say before you say it. This is particularly hard with respect to one on one meetings or group meetings. However death by power point is the second most common cause of Gout in the United States, only bested by an intense desire to stand-up, pull your hair out of your head and say “I’m mad as hell, and I’m not going to take it anymore”. I’ve also come to understand that the lines “It’s my money, and I need it now” are starting to cause many people to have gout flare-ups.

DO NOT come into a in person presentation with power-points and a dull dry script that you don’t know. It’s disaster waiting to happen.

Similarly don’t go talking to those up in the golden tower without having your facts at hand, without fully understanding your argument and also understanding who your enemy is. Yes, I believe that communication can be a battle area and I come prepared to go to war. You should too. (sometimes surrender is inevitable but you can always re-group for another day).

Practice - This is why priorities #1 through #4 are so important. Yes, they are important to the organization to be sure, but for you to be the best DBA for the organization you need to learn to communication. You need to be willing to communicate, you need to have time to communicate. Getting #1 through #4 in place, automated, repeatable should be the first step to improving on your communication skills. It’s those communications skills that will take you down the yellow brick road to becoming the holistic DBA.

Listen - What does listening have to do with communication? Just about everything! The key to communication is listening. There are incredible orators out there, but when you are in a one on one, or one on group discussion, it’s listening that will be the key to your success. Other people have great ideas. Sometimes those people are quiet, and often go unheard. Help them speak up when they have a good idea. Also, as you listen to the group dynamic, if you allow yourself to “step away” from it for a while and just listen to what’s being said, you might have some insight come to you that you that might not otherwise. You might find yourself being the man-in-the-middle, as it were, the un-polarized DBA with the correct solution that is obvious and defusing. Sometimes listening can lead to a successful meeting where just talking can lead to a disaster.

Does communication really belong at the #5 spot? Part of me really wants to park it in the number one slot for a lot of reasons, and maybe I should. The reason I park it in the #5 slot is that I see it as the gateway to the greater universe and the method by which we hyper-inflate our jobs and become the best damn DBA’s on the planet.

Ok… enough about communication. No, my name is not Covey. I don’t purport to be an expert on communication. I do know what I’ve seen work, and what I’ve seen fail terribly. I’m also not perfect at it, not by far…. I get frustrated way to easily, I find my tolerance levels dip into very low levels way to easily and I lose patience when I get stressed. Just saying, I’m far from perfect.

More about the last priorities on my next post…

DBA 3.0 (The Holistic DBA) – Part Three Afterthoughts

Having posted part 3 a few comments have shown up in my in-box, and a few thoughts have come to mind that I really wish I’d included in this particular article. So I think I’ll share them now.

First, I wish I’d addressed the concern with complexity a bit more directly. In the story of the DBA who took the complex backup spaghetti code and replaced it with equally complex backup spaghetti code, I totally didn’t point out the issue of complexity. Reducing complexity should be one of your key drivers when it comes to “refactoring” your existing DBA code (backup, monitoring, etc…). Reducing complexity reduces likely failures and that gets back to the core of the whole point of what we are shooting for, freeing up your time so that you can pursue more worthwhile ventures than tracing down that annoying bit of code that keeps failing.

Another comment has been made about something I’ll call DBA approachability. I’ll address it in the future, but I’ll say now that I agree one hundred percent with this opinion. I think being approachable fits in with priority #5, which is communication. A lot of our DBA crankiness comes from the fact that we are usually busy trying to deal with the implications of priorities #1 through #4 not really being mature. Not automated. Not self service. We have a list of tasks to do (x number of databases to restore, 5 emails from developers asking for help, my boss wants my status report and now you are asking me what I think of flashback database? Stand in the back of the line buddy). When #1 through #4 are out of wack, so are we and so is our ability to communicate efficiently, and we find we are stressed and over loaded. We really can't do our job effectively.

Then…. There are those folks who just don’t have a nice disposition. I once worked in a place, I’ll call it the Entire Massive Political Insight Realignment Engagement Institute (or the EMPIRE Institute). At the EMPIRE institutes DBA shop everything was a mess. Everyone was running around like a droid with his head cut off. It was pandemonium. In one of the various DBA shops there was myself, Ben and Sidious (who’s mother apparently loved Star Wars)). In spite of the Craziness, Ben was the nicest guy you would ever meet. He could be taking on five storm troopers, three wookies and a sand worm and still give you a warm smile, sit down with you are talk about your problem and help you find a solution. He'd pour you a cup of hot chocolate, offer your a doughnut and listen to your problems. If you had a suggestion to offer, he'd listen to that too.

Then there was Sidious. Just walking by his cubicle you could feel the ice coldness of his being. Somewhere in the background, you could hear the dark voices singing the song of lament and danger. When you poked your head in, he would turn in his chair, and the hideous look on his face made it clear, you were not wanted.

“What is it you want, young Freeman?”, he would ask. Clearly he had no respect for me what-so-ever. The problem is, he had no respect for anyone, he had no time for anyone other than his apprentice somewhere in the shadows lurking (ok…. So I’m making some of this up, but it’s based on a true story, I promise! ). He just sits there, in his throne chair, watching over his empire lest anyone rebel and try to seize power.

“Um…. They asked me to create some tablespaces, schemas and other object like things. I need the DBA keys please”, I responded.

Sidious, on the right arm of the chair, pointed to the piece of paper that contained the passwords I desired. He looked at me, “You want these, don’t you?”

“Um, yeah…. Your manager told me he told you to give them to me.”

Sidious looks at me, “No one manages the dark side of the force but me young one.”

“But your manager….”

“I said NO ONE Manages the dark side of the force but ME young one.”

The point of all this was Sidious thought he owned the world. I was given a directive to do certain things because Sidious didn’t know HOW to do them, was screwing things up and I was there to fix those screw ups. Sidious was hell bent on showing me who was in charge. He was hell bent on making sure my life was miserable. He was hopeful that I would become his minion and bow to his demands. Kindly but firmly I resisted the temptations of the dark side. The temptation to tell him to take a hike down a long nuclear chamber, the temptation to cut him down with my Skywalker 1000 side-arm. However, I had weapons that he could not contend with. I had the force at my back (in the form of management three levels deep), I had persistence and I had a kind but firm, I'm not going to take this blue lightning light crap from you brother, you are messing with the wrong guy.

After the ensuing battle between myself and Sidious, ending at the manager’s office at least three times, I finally pried the passwords from his cold, tired, 600 year old hands. While there were words had (mostly by Sidious and management), hearts broken (mostly the near-retirement age women sitting in the area who were subjected to Sidious and his words and actions) and a few cubicles burned with his evil blue lightning like electrical powers, I managed to keep my cool.

Really, I did.

Really!!

The point of all of this really is that even if we have to say no, there are ways of telling people no that are positive, and there are ways of telling people no that cause mass rebellion, planets to be destroyed, and general hostility throughout the galaxy. How you communicate tells allot about who you are, the kind of person you are, and has some direct impacts on the organization.

I mean it, I did keep my cool! Really!

I’ll put it this way. Why do we have developers using crap (I'm sorry, highly developed software) like hibernate to build schemas? It’s because they don’t want to come talk to us about the problems they are having with provisioning development environments, or we are not responsive to their needs, or we are not providing help and education with schema design and have not had the time or the inclination to teach them, to communicate with them. The same holds true for the desire to have constraints be present in source code rather than the database, etc…etc….etc….

So the bottom line is, a self-assessment is required, and probably on a regular basis. The basic question is this, are you really accessible? Is your countenance that of Ben or Sidious? Based on that answer, how does that impact the organization and how does it make your life easier or harder. Tough questions, tough answers and even tougher solutions sometimes. Or maybe not. Sometimes the solution is just learning to smile, say a kind word or two, remove the F-Bombs from your vocabulary and …. When you have time, you might attend to that face of yours. Sidious really needed some massive plastic surgery… I heard it was an old battle wound during the dark days but wow…. He was so shriveled up, and those eyes were so eerie…. Like they just looked through you.

All joking aside…. We will address communication in the next installment. Being nice, being personable, being trusted are all part of that equation.

And I really did keep my cool...

DBA 3.0 (The Holistic DBA) – Part Three

DBA 3.0 (The Holistic DBA) – Part Three

It’s been a little while since I wrote part two of my DBA 3.0 series. I find it’s 3am in the morning, I can’t sleep, I don’t feel well and I’m cold.

A perfect time to write.

First, I’ve decided that my concept of DBA 3.0, though well intended, is not very descriptive. There is one comment on my previous post where someone mentioned that they used the moniker DBA 3.0 in a work to describe something. Now, DBA 3.0 is generic enough that I don’t feel all that bad about adopting the name, but in my mind that is not descriptive of the concept. So, I’m rechristening the idea as the Holistic DBA (and maybe to make it extra cool we could add something like – The Next Generation to the title…. naaahhhhhhh).

In exploring the idea of the Holistic DBA in my previous posts, I lamented that many are entrenched in their cubicles, that many do not speak the language of business, that many are too busy writing scripts or digging into the next new technology. All of these things, in their proper proportion are good things, no doubt. It’s when the priorities get out of wack, then what we do (or as often what management tells us to do) can get us into trouble. In my world view, the holistic DBA is not a specialized DBA. They don’t spout out 10046 translations without using TKPROF, they don’t dump the contents of datafile headers and use bbed to magically open the database (and if they did I’d question if they were really an expert at all).

Let me say, first and foremost, that not every DBA needs to be a holistic DBA. Also, not every organization needs to have every DBA be a holistic DBA. There is a need in the technical world for specialization if your organization can afford it and if the need can be justified based on cost, meeting customer expectations and principally (in my mind), meeting established Service Level Agreements (SLA), Recovery Time Objectives (RTO) and Restore Point Objectives (RPO). I will also counter that every specialist DBA needs to have the skill sets and countenance to be a holistic DBA, especially that of communication.

There will always be a need for Lewis, Kyte, Millsap and that breed of expert (forgive me if I left your name out, you are all much smarter than me and I freely admit it).

And now, I’ll let the shock wear off your face with the notion of SLA’s, RTO’s and RPO’s. You might be asking, “What, do organizations really have those?”. Indeed, the best run, most successful data organizations do have those, somewhere.... often in a form that is pretty much useless, and not updated since the last great depression.

The organizations that have repeatable success have those. Organizations that fly by the “seat of their pants” typically do not, and that is one reason why those types of data organizations fail, eventually.

Eventually is a key word here too. Just because your ham-strung, bailing wire and spit, technical marvel has not yet failed, be sure that it will someday fail. When that day comes the song will be “Who’s got the job today? Oh Yeah! Oh Yeah!” as opposed to, “Information, I need the number for the unemployment office”.

If you think that it’s just the little boys who fail, and that the big boys and your big boy organization is immune from failure because of its vast size, number of DBA’s and it’s wonderful glowing talk about procedure, process and all the like, you are waiting to stand in the unemployment line too (or you should be already there). No my friends, the evil angel of death is an equal opportunity player when it comes to data center death and subsequent total mis-management of the world. In fact, I would suggest that the bigger you are, the harder you are more likely to fall and fall hard. Just because it hasn’t happened yet does not mean that the grim reaper does not have you on his visiting list… he is, after all, not omni-present. The stories I could tell of the foolish big boys and the key assumptions they made that caused massive failures.

So… What do we do to fix these problems? First of all, we have to realize that we have some principle goals as DBA’s. I’d like you to consider the following items as part of a list of these principle goals:

1. Be able to reliably, consistently and efficiently backup and recover the databases and all database in accordance with SLA, RPO and RTO’s.
2. Ensure database uptime with respect to all SLA, RPO and RTO’s.
3. Ensure all databases are secure.
4. Monitor all databases in a consistent and reliable manner.
5. Communicate in an effective manner.
6. Help users to help themselves.
7. Work to understand and correct bad behaviors with respect to the organizations data policies.
8. Work to understand and correct bad behaviors with respect to the organizations data designs.
9. Work to understand and correct bad behaviors with respect to the organizations application designs.
10. As befits the organization, your abilities, and time keep up with current data related technologies.

Do you notice anything about being the hero of the day on this list? Good, because you should never need to be the hero of the day. Yeah, it feels good when it works, but the unemployment line feels even worse when it does not.

Do you notice irritating details with respect to policies and procedures and doing backup and recovery testing? You didn’t? Really? Look closer at the list my friend. It’s in there.

Now…. I ordered this list in a very specific order for a specific reason and I wonder if you can tell me why? What is it about this list that, when done in the order listed, makes for a holistic DBA?

First, I think they are arguably in some order of importance. You might shift a couple of them around, say 2 and 3 or 8 and 9 (or you might consider them one and the same), but generally they are in order of priority (in my eyes) to any data organization. What else though, is magical about this list?

Automation, replication, consistent execution. Look carefully at 1 – 4. Holy cow, they are all something that can be documented, easily. They can be implemented rather easily one time, fire and kinda-forget within the constraints of an established policy (for example: Occasional maintenance, testing and the like thrown in to the schedule) and automation.

Something else I’ve noticed about the whole automation thing, we really like re-inventing the wheel don’t we? Why do we feel like we have to spend hours of time re-inventing the wheel by writing korn scripts, throwing them into Cron and having them run all over creation. Oh, here’s is a smart guy, he put them all on a shared NFS drive. UGH. Do I really need to mention the problem here? Do I really need to say this is the wrong way to be doing things?

Case in point – again the characters, places and all identifying information have been properly scrubbed. Maxwell Smart was a DBA in a rather large DBA organization. Now Maxwell was tasked with the assignment of getting RMAN “up and running” and to replace the stock of old, spaghetti code of backup scripts in the process. Maxwell is a pretty smart guy, but he really needs guidance and he really didn’t get much on this assignment. He felt the weight of this assignment on his shoulders quite broad and heavy. As a result, Maxwell replaced the old spaghetti code backup scripts and replaced them with a new set of equally spectacular spaghetti code backup RMAN backup scripts. Essentially we changed nothing in the process, except the tool that did the backups. The overall architecture was not considered, the total non-supportability of the scripts in the future was not considered, and the result was a product that was no better than what it replaced.

The point is that the holistic DBA, as we continue to define what and who they are (in my own limited understanding) is that we hope that Maxwell Smart will have moved past steps 1-4 long ago. That Maxwell would have moved onto the steps that offer the greater good, #5 in particular, and then steps 6 through 10. If Maxwell were a holistic DBA he might have come to understand some principles of architecture and design, that he didn’t possess when he re-created hell and he might well have created a backup haven.

Perhaps your retort to using technologies is something like “Well Grid Control didn’t work with backups n number of versions ago and so I gave up on it.”. Really. How very odd. The one product that Oracle gives you to make your life easier and at the first hint of failure, you give up on it. Instead you resort to korn, perl, or heaven knows what scripting language is your favorite and you spit out this cool, really dense code. WOW…. Meet Dr. Frankenstein. You have your creation and its’ a monster. Do I really need to tell you why it’s a monster? The bottom line is the holistic DBA put’s his ego and desire to raise the dead on hold and does it the right way. He/She does it in a way that is automated, centrally manageable and easy to use.

So, your retort, “what if it is broken”? Then my friend, it’s time to remember your priorities above, call Oracle support and get the damn thing fixed.

Then you retort, “They never work on my SR, they never call me back”…..my goodness someone call the wambulance and rush ahead to priority number 5 and get it down fast. The bottom line is that the holistic DBA can not be a passive “Yes man”. You don’t have to be a jerk, you don’t have to be evil incarnate to get what you want but you do have to persevere. You have to pursue excellence, even if those around you seem not to be following the same path. Heavens, this is leading me down the path to calling this the Black Belt DBA but I’ll avoid the temptation.

If you are not getting the support you desire from Oracle (or any other vendor) then who’s responsibility is it to get the level of support you need? YOU! It is your job to make these things work, and to bust dam’s (and perhaps even utter one or two aloud) in your pursuit of a solution. Oracle support can be very responsive if you know how to use it, and if you are persistent. If you don’t know how to use it, beyond opening an SR on Metalink, there are some great training tools available to teach you how to use the support system. Working with any support organization is a bit like working with an old car with an engine that tends to flood way to often. You have to figure out how it works, what knobs to pull and just how to pull them. Once you figure that out, the car will start up pretty well just about every time. Again though, I put the onus back on you to make support work for you. If you are going to sit and wait for support to come looking for you then the fault, in my opinion, lies within yourself.

So, we’ve figured out that 1-4 are pretty easy to take care of (though the implementation may take some time and major initial effort which can frustrate those who love to do things by the seat of their pants). Do you notice something else about #1-4? First, they don’t require you to know how to do 10046 tracing. They don’t require that you know how to do explain plans. They don’t require that you have to be able to do data file header dumps or a triple summersault with a half-gainer pike on the database (is that possible?). You might protest, but I need to know how to do these things? I need to be able to look like I know what I’m doing. I agree, you need to look like you know what you are doing. Remember that this list is a list of priorities. Few people will remember you for your marvelous once every six months 10046 trace that fixed one year old SQL code that was not designed to be scalable. They just won’t. They will remember that your database was restored flawlessly and on time and that the business never even noticed things were down. Better yet, they won’t even remember because they didn’t notice and there was no need to. If you have a good boss, he will remember. If you get off the wambulance and show some persistence in your reviews, he will certainly remember.

Here is another point about the over-arching nature of numbers 1 through 4. They can be costly if not done right. If you find yourself iterating only between these various areas of DBA Ville, you are costing your employer vast sums of money that really does not need to be spent.

Another point about 1 through 4 is about automation and simplification. You need to realize that these are just lines in the page and that there are other needs in this section that you need to “read between the lines”. Automate, automate, simplify, simplify. For example, if you find that you are creating lots of databases or lots of schemas, and that takes a lot of your time, find a way to automate that workflow. Make it user serviceable. The tools are out there to allow you to do basic, simple, automated provisioning. Rather than spending 6 hours to 5 days to provision a database, let’s spend 2 seconds as you click the approval email that is a part of the overall workflow. These are the kinds of things that you need to be doing, now.

Plenty has been said about re-inventing the wheel, about the costs of re-inventing the wheel and the dangers of re-inventing the wheel. Use the Damn wheel why don’t you and quite finding little reasons not to use the wheel and build your own wheel. “But there are bugs in the wheel”, you say. All wheels have bugs. The problem is, the only person who can solve your wheels bug is you, and if you leave, then who is it going to be. Sure, the Oracle wheel might have a bug or “undocumented” feature in there, but can you really cost-justify the time and money spent re-inventing the wheel rather than taking the support organization to task and making them fix the problem? Also, keep in mind that the wheel they created, has been tested many more times and in various different ways than the wheel you are creating.

The real point, with numbers 1 through 4 is that they really can take up an inordinate amount of time if not done correctly. They can also cause an inordinate amount of damage to the organization if not done properly. We need to have these settled, on automatic pilot and flying without aide of an instructor reliably. Then we have freed up time for tasks that are more strategic, and therefore quite important.

That the leads to the that magical mystical number 5? Let’s talk about that and the rest of them in my next post.