Friday, January 27, 2012

Engineered Systems - Complexity vs. Simplicity...

Oracle started the Engineered Systems decade (my word!) with the introduction of Exadata. Now we have Exalogic and other machines that provide this engineered system concept that I generically like to call Exa* (Exa-Splat). A lot has been touted with respect to the feature sets of these systems, and I've seen enough POV's now (both first-hand and word of mouth) that I can tell you that the performance improvements are real and significant.

However, I think that the story of Engineered system is about much more than being able to take a query that ran before in 24 hours and now it runs in 3 minutes. Sure, that's great and very important, but there is a strategic reason for Engineered systems and I'm not sure that is getting as much play as it should, because I think it's important.

Does this sound like you? Your environment is this collection of loosely coupled systems. You have these great, powerful database servers but when you look at CPU it's sitting at some number like 20 percent. You have servers all over the place. Your behind on your patching. Your constantly standing up new systems, and all the related infrastructure (network, disk, etc) and it just takes time. Your time to provision has the customers complaining. Your spending all your time reacting to problems, to configuring new environments, to wondering why this switch does not work with this server. You spend a lot of time wondering how to integrate various pieces and which technologies are best and which are certified. If this sounds like you, I call you tactical man. You are tactical in your thinking. Your poor customers really don't have the ability to derive knowledge from you because you are busy standing up systems for them. They are building database designs that are amazingly awful, and you are unable to help them because you have to work on the next fire, the next problem.

Is this you:

(phone rings)
You: Hello
Fred: Hey this is Fred. We need help with a database design. What's this normalization thing?
You: Sorry Fred, I'm working on installing four copies of Linux right now, can't help.
Fred: Hey, I understand, you guys are over worked down there. We will just use Hibernate to generate the schema, no problem.
You: No... wai...
(click on the phone)...

Are you tactical man? I wrote some time ago about the holistic DBA and really... becoming the holistic DBA is all about transitioning from Tactical Man to Strategic Man. The truth is that engineered systems can help you in this transition. How you ask... First, look at this handy dandy graphic I whipped up:

This image is kind of an exercise in simplicity (and my limited graphics design abilities) but I think it makes a point. As we move from higher complexity to simplicity, we move from being tactical (reactive) in our work to more strategic and proactive. Also, look at the graphic... in moving in this direction we reduce the number of people required and we reduce the costs as well.

Engineered systems is part of this drive towards simplicity. With an engineered system, you let the vendor do the painstaking work of finding the right components, certifying them, and making them all work together like a well oiled machine. With Engineered systems we get the benefit of scale.... If one problem surfaces on a given system, because the system are all similarly engineered we can immediately apply the solution to the problem across the various enterprises running that engineered system. No more wondering about this or that combination of hardware, we know (more properly the vendor knows) and the vendor take responsibility for providing that remediation across the platforms that are deployed.

Other benefits include the reduction of the proliferation of systems and hardware, reducing the overall footprint that has to be managed. This reduces costs and allows you to again move up towards the strategic part of the graph. There are a number of benefits with respect to the implementation of  an engineered solution, be it Exadata or even the Oracle Database Appliance. We should not focus on just one part of the equation (performance) when asking ourselves, what do we do to improve productivity and become more visionary?

As a result... the DBA, the network engineer, the storage administrator, the Unix administrator, are all free to pursue tasks further up the scale, moving in the direction of strategic planning, proactive approaches to business problems and helping the customer do it right the first time. In being visionary about the systems we deploy we can improve our ability to react to strategic needs of the enterprise.

I know... this might sound like a sales pitch for Exa* products.... and I do work for Oracle, but everything I've said here is part of my experience, past and present. Tactical man is alive and the costs of keeping him in business are significant. Strategic man needs to be our goal, and with Strategic man comes a host of benefits including lowered costs and a better Enterprise. Engineered systems can help you migrate to becoming Strategic man...

Sure... really.... we all just love installing Linux and configuring Clusters... but is that really best for the Enterprise? I think not.


Scott said...

In some ways SANs are the worst thing to happen to databases. DBAs stopped thinking about storage layout and what else is on theses spindles. In most shops when a request is made for DB storage it's how much space do you need, 1 TB thats fine it can fit on single or a pair of 2 TB drives. Even when the LUN can perform the plumbing between the SAN and the DB server is often overlooked. Dual 4Gb HBAs are actually quiet slow if you want to move 2GB/sec.

Joel Garry said...

The real Strategic Man looks at what it would cost to buy some black box and convert the current enterprise to it, and figure out whether there is any worthwhile return. So when Larry decides to force Itanium users to go in a different direction, $1M says he screwed up.

Just like Romney saying if someone went up to him and said lets go to the moon, and he would say "you're fired." If he were Strategic Man, he would say "what would it get us?"

Robert Freeman said...

The costs/benefit/reward involves more than just what does it take to buy the box and convert the enterprise. There is much more to it than that. If your replacing 20 boxes that have a 5 year refresh schedule with 2 boxes, what is the return on that. If I'm saving time standing up new boxes and all the required infrastructure and can spend that time working with developers on more efficient data models or getting involved in strategic decisions early.... there are tangible and intangible costs galore that simplicity brings.

I really don't think you "Itanium" argument has a place in this discussion and it sounds more like bitter dregs. I almost opted not to publish it because really it's not about the post in general. The itanium topic is a emotional topic for some, and I have no real desire to go there in this blog.

Robert Freeman said...


I so agree with you and I know that I've been saying this for a long time. One experience I had really shines light on this whole SAN issue. Production system screams Friday evening. Come in Monday morning it's dog slow. There is no apparent difference in usage, nothing new going on, no changes implemented in the application, nothing at all.

We noticed disk response times had gotten terrible. We talked to the SAN guys, no.... no problems here. Everything looks fine, no changes over the weekend... nada. No errors being reported... what the heck was going on.

We spend ALL week working on the problem, but just could not fix it. We tore our hair out... until...

One SAN administrator who had been on vacation came back late that week. He heard about our problem and copped... um, well, he had moved the files, online, to other storage on the SAN. In the end, we were on fewer/slower physical disks that before, so DOH!!

In this case there was no change management, no approvals or controls ... worst of all, there was no visibility into the "black box" that was the SAN for the DBA's to be able to see such changes.

I hate this and it needs to stop! Think about how much money was spent as a result of that one exercise.

4 resources at say, $50/hour * 40 hours (probably more) each... that's some $8000.00 right there. Now add into the equation the customer happiness factor (this was a customer facing application with concurrent customer usage of something on the order of 500 customers an hour with a total customer count of well over 100k... what is the impact there in dollars?

Yeah, the SAN.... and any black box, has the potential to kill.

Subscribe in a reader