Random Thoughts: Blog

My Blog has moved.

Fri, 18 Jul 2008 00:14:12 GMT

Just a quick note that my blog has moved to this location. Thanks so much for your continued readership.

-Mm

The Stars are Bright....

Sat, 31 May 2008 21:13:14 GMT

And so is the progress on our first San Antonio facility. I just recently got back from a Construction review of our first data center coming up in San Antonio. I have to say it is progressing wonderfully. I snapped a few pictures which I posted to my photo albums here. Sorry folks, I did not post any internal shots, but rest assured that it eclipses those on the outside. Enjoy.

Mike

IT Victimization and Lack of Adoption of Energy Efficiency Focus in the Data Center

Fri, 30 May 2008 20:10:04 GMT

I have been talking about Energy Efficiency in the data center for some time. It has been a primary component of my talks at both AFCOM and Uptime. Today I read an article which talks to a lack of interest, desire, or funding to drive efficiencies across the Industry. Its sourced from the Aperature Research Institute It begins with:

A survey by the Aperture Research Institute (ARI) of more than 100 datacenter professionals has shown that organisations are unable or unwilling to meet the expectations set by their adoption of green initiatives for the datacenter. This follows an earlier ARI report published in March 2008, which discovered that 70% of organisations are adopting green initiatives, although 19% of those had omitted the datacenter from that programme.
In the latest ARI study, 74% of those surveyed refused to activate power saving features on devices if it would require a drop in performance. While 37% are concerned that the power/performance ratio doesn’t add up, 15% worry that they have no way to track whether the power saving setting is on or off. Nearly half (48%) of those surveyed blame the business for not using power saving features, saying that users wouldn’t tolerate a drop in performance in the interests of saving power.

Bunk! I am curious if they have ever even tried these features to verify the actual impact. When you take into consideration that the world wide utilization rate of servers is sitting between 6%-15% I am just finding it impossible to wrap my head around this alleged tradeoff. The complaints around the fact that there would be no way to verify the status power saving settings is almost humorous if it wasn't such a terrible thing. The fact is there are ways to check the status and turn it on or off remotely. Even without tools that could do this for you, (System Center Anyone?) I am curious if IT has lost its ability to write and run the most basic of scripts.

It speaks to a phenomenon that I call the "IT Victimization Syndrome" or ITS for short. It really is a terrible affliction. I cannot tell you how many times I go to conferences and the overall tone is how evil everyone is to the IT organization. Everything happens to them and they have no power to combat it. Ill come back to this in a second. The next thing the article talks about is :

When it comes to procurement, energy efficiency and ease of disposal are the lowest priorities, rated as less significant than brand and price. 37% of datacenters have no plans to measure energy efficiency, and 76% do not charge the business for the power used by the IT commissions. One reason for that is a lack of infrastructure for measuring power consumption.

Here is where I think this study got something wrong. Their sample size is too small. In a very unscientific poll performed by me between AFCOM and Uptime Symposium, I asked the collected hosts how many people were measuring for power efficiency. Here amongst perhaps the most sophisticated and significant data center operators in the world, less than 10% raised their hands. Even less than that had chargeback mechanisms that incorporated power.

Im sorry the statement that the lack of infrastructure to measure consumption is just ludicrous. I am fairly certain that most data centers have a UPS system. Here is a free way to calculate total power consumed....its called SneakerNet. Have your facility person stop by this device 3 times a day, write down the power draw on a clipboard, or for the more advanced, even a spreadsheet. This allows you to do many wonderful things like even calculate your PUE or DCiE over the course of the year. But more importantly if you take these numbers, and invest in a power meter that you can loop around individual servers you can actually start collecting power consumption at a pretty granular level. Its a free way with little cost to at least start this effort. As my mother used to say, 'Where there is a will, there is a way'. I personally think there isn't much of a will. Which is an entirely different thing than trying to blame the business or technology.

Ok back to my Victimization rant. The next component talks about the challenges around ghost servers:

The survey also found that decommissioning processes are not strictly followed, and “ghost servers” haunt the datacenters of 19% of organisations. Ghost servers are those servers which the business no longer needs, but which have not been switched off, and which are as a result needlessly consuming electricity, space and other limited resources.

I heard this plenty at the recent conferences. How terrible is it that the property groups / business units/ whatever, are to the poor IT folks. They don't follow the de-commissioning processes. Who owns the Decom process? Who is in the best position to correct this huge injustice? The IT organization! This is a problem that is 100% correctable through better operational practices and attention. This isnt magic folks, this is roll-up your sleeves and make something that works. I walked out of 5 or 6 sessions amongst the various conferences that fell into this Victimization category. They felt more like support group sessions than forward thinking industry sessions.

There is inherent power in the ownership of the infrastructure and its support. Play the victim or define solutions that work. Find the Will!

Mike

Microsoft Environmental Site

Mon, 26 May 2008 19:40:58 GMT

I wanted to highlight the recent re-launch of the Microsoft corporate environmental web-site. There is some great new content and resources available, including a video featuring Rob Bernard discussing Microsoft’s sustainability efforts. Now perhaps I am a bit biased as I am a contributing author and content provider, but I am not sure many of you have ever given the site a visit.

Other recent features include:

Energy Efficiency Best Practices in Microsoft Data Center Operations

-Watch this video (on TechNet Edge) to help make energy-conscious decisions about PC purchases.

How Can Technology Sustain the Environment in the 21st century?

-The software industry is leading the way in helping consumers and businesses increase energy efficiency.

Windows Server 2008: Enabling Energy-Efficient Performance

-Learn how the new power management features in Windows Server 2008 can help reduce your IT electricity bills.

If you get a chance please feel free to visit the new site at http://www.microsoft.com/environment.

Mike

The Phoenix has Landed!

Mon, 26 May 2008 03:43:26 GMT

Way to Go, JPL! Great Job! What an incredible vista of sights! Yet another Notch in the Scout program!

Mike

Fingers crossed for Phoenix Lander

Mon, 26 May 2008 03:37:57 GMT

I have been a huge fan of the NASA's Scout Missions and I am waiting here with great anticipation waiting news on its latest lander, Phoenix. Let's hope the Galactic Ghoul does find a tasty treat tonight inbound to the Northern Martian Icecap. So far, so good. I wish the JPL team the very best and know that I have my fingers crossed. Imagine throwing a baseball from Dodger field in Los Angeles and striking out a Cub in Chicago's Wrigely field. Thats the challenge they are facing.

Mike

Im a Jackass, its official :)

Tue, 20 May 2008 11:14:10 GMT

It turns out Eric Lai is a man after my own heart. I got word that he posted a response to my short dissertation on his article and I took a quick read. Before I started reading it, I envisioned a bunch of kids screaming "FIGHT! FIGHT! FIGHT!". But I have to admit I loved it. I especially loved the "mulish" comment and the basic take away that there was a need, no a desire, for more information. Hard information. He is asking the right questions. I guess at an atomic level I am just happy that there are new questions to be asked in this industry where dogmatic guardians of the standard quo fight change so hard. From the Microsoft angle - I guess my official response is "all in good time." But I can assure you it *is* coming.

Eric - You have an open invitation to cruise around the South Side of Chicago with me anytime and do what we south-siders do best...At least I know you wont run away from a good "go round". We may even stop off and take a gander at a new data center I here is going up nearby :)

/Mike

Stirring Anthills ... A response to the recent Computerworld Article

Mon, 12 May 2008 19:04:28 GMT

When one inserts the stick of challenge and change into the anthill of conventional and dogmatic thinking they are bound to stir up a commotion.

That is exactly what I thought when I read the recent Computerworld article by Eric Lai on containers as a data center technology. The article found here, outlines six reasons why containers won't work and asks if Microsoft is listening. Personally, it was an intensely humorous article, albeit not really unexpected. My first response was "only six"? You only found six reasons why it won't work? Internally we thought of a whole lot more than that when the concept first appeared on our drawing boards.

My Research and Engineering team is challenged with vetting technologies for applicability, efficiency, flexibility, longevity, and perhaps most importantly -- fiscal viability. You see, as a business, we are not into investing in solutions that are going to have a net effect of adding cost for costs sake. Every idea is painstakingly researched, prototyped, and piloted. I can tell you one thing, the internal push-backs on the idea numbered much more than six and the biggest opponent (my team will tell you) was me!

The true value of any engineering organization is to give different ideas a chance to mature and materialize. The Research and Engineering teams were tasked with making sure this solution had solid legs, saved money, gave us the scale, and ultimately was something we felt would add significant value to our program. I can assure you the amount of math, modeling, and research that went into this effort was pretty significant. The article contends we are bringing a programmer’s approach to a mechanical engineer’s problem. I am fairly certain that my team of professional and certified engineers took some offense to that, as would Christian Belady who has conducted extensive research and metrics for the data center industry. Regardless, I think through the various keynote addresses we’ve participated in over the last few months we tried to make the point that containers are not for everyone. They are addressing a very specific requirement for properties that can afford a different operating environment. We are using them for rapid and standard deployment at a level the average IT shop does not need or tools to address.

Those who know me best know that I enjoy a good tussle and it probably has to do with growing up on the south side of Chicago. My team calls me ornery, I prefer "critical thought combatant." So I decided I would try and take on the "experts" and the points in the article myself with a small rebuttal posted here:

Challenge 1: Russian Doll Like Nesting servers on racks in containers and lead to more moreness.

Huh? This challenge has to do with the perceived challenges on the infrastructure side of the house, and complexity of managing such infrastructure in this configuration. The primary technical challenge in this part is harmonics. Harmonics can be solved in a multitude of ways, and as accurately quoted is solvable. Many manufacturers have solutions to fix harmonics issues, and I can assure you this got a pretty heavy degree of technical review. Most of these solutions are not very expensive and in some cases are included at no cost. We have several large facilities, and I would like to think we have built up quite a stable of understanding and knowledge in running these types of facilities. From a ROI perspective, we have that covered as well. The economics of cost and use in containers (depending upon application, size, etc.) can be as high as 20% over conventional data centers. These same metrics and savings have been discovered by others in the industry. The larger question is if containers are a right-fit for you. Some can answer yes, others no. After intensive research and investigation, the answer was yes for Microsoft.

Challenge 2: Containers are not as Plug and Play as they may seem.

The first real challenge in this section is about shipment of gear and that it would be a monumental task for us to determine or provide verification of functionality. We deploy tens of thousands of servers per month. As I have publicly talked about, we moved from individual servers as a base unit, to entire racks as a scale unit, to a container of racks. The process of determining functionality is incredibly simple to do. You can ask any network, Unix, or Microsoft professional on just how easy this is, but let’s just say it’s a very small step in our "container commissioning and startup" process.

The next challenge in this section is truly off base. . The expert is quoted that the "plug and play" aspect of containers is itself put in jeopardy due to the single connection to the wall for power, network, etc. One can envision a container with a long electrical extension cord. I won't disclose some of our "secret sauce" here, but a standard 110V extension cord just won't cut it. You would need a mighty big shoe size to trip over and unplug one of these containers. Bottom line is that connections this large require electricians for installation or uninstall. I am confident we are in no danger of falling prey to this hazard.

However, I can say that regardless of the infrastructure technology the point made about thousands of machines going dark at one time could happen. Although our facilities have been designed around our "Fail Small Design" created by my Research and Engineering group, outages can always happen. As a result, and being a software company, we have been able to build our applications in such a way where the loss of server/compute capacity never takes the application completely offline. It's called application geo-diversity. Our applications live in and across our data center footprint. By putting redundancy in the applications, physical redundancy is not needed. This is an important point, and one that scares many "experts." Today, there is a huge need for experts who understand the interplay of electrical and mechanical systems. Folks who make a good living by driving Business Continuity and Disaster Recovery efforts at the infrastructure level. If your applications could survive whole facility outages would you invest in that kind of redundancy? If your applications were naturally geo-diversified would you need a specific DR/BCP Plan? Now not all of our properties are there yet, but you can rest assured we have achieved that across a majority of our footprint. This kind of thing is bound to make some people nervous. But fear not IT and DC warriors, these challenges are being tested and worked out in the cloud computing space, and it still has some time before it makes its way into the applications present in a traditional enterprise data center.

As a result we don't need to put many of our applications and infrastructure on generator backup. To quote the article :

"Few data centers dare to make that choice, said Jeff Biggs, senior vice president of operations and engineering for data center operator Peak 10 Inc., despite the average North American power uptime of 99.98%. "That works out to be about 17 seconds a day," said Biggs, who oversees 12 data centers in southeastern states. "The problem is that you don't get to pick those 17 seconds."

He is exactly right. I guess two points I would highlight here are: the industry has some interesting technologies called battery and rotary UPS' that can easily ride through 17 seconds if required, and the larger point is, we truly do not care. Look, many industries like the financial and others have some very specific guidelines around redundancy and reliability. This drives tens of millions to hundreds of millions of extra cost per facility. The cloud approach eliminates this requirement and draws it up to the application.

Challenge 3: Containers leave you less, not more, agile.

I have to be honest; this argument is one that threw me for a loop at first. My initial thought upon reading the challenge was, "Sure, building out large raised floor areas to a very specific power density is ultimately more flexible than dropping a container in a building, where density and server performance could be interchanged at a power total consumption level." NOT! I can't tell you how many data centers I have walked through with eight-foot, 12-foot, or greater aisles between rack rows because the power densities per rack were consuming more floor space. The fact is at the end of the day your total power consumption level is what matters. But as I read on, the actual hurdles listed had nothing to do with this aspect of the facility.

The hurdles revolved around people, opportunity cost around lost servers, and some strange notion about server refresh being tied to the price of diesel. A couple of key facts:

· We have invested in huge amounts of automation in how we run and operate. The fact is that even at 35 people across seven days a week, I believe we are still fat and we could drive this down even more. This is running thin, its running smart.

· With the proper maintenance program in place, with professionals running your facility, with a host of tools to automate much of the tasks in the facility itself, with complete ownership of both the IT and the Facilities space you can do wonders. This is not some recent magic that we cooked up in our witches’ brew; this is how we have been running for almost four years!

In my first address internally at Microsoft I put forth my own challenge to the team. In effect, I outlined how data centers were the factories of the 21st century and that like it or not we were all modern day equivalents of those who experienced the industrial revolution. Much like factories (bit factories I called them), our goal was to automate everything we do...in effect bring in the robots to continue the analogy. If the assembled team felt their value was in wrench turning they would have a limited career growth within the group, if they up-leveled themselves and put an eye towards automating the tasks their value would be compounded. In that time some people have left for precisely that reason. Deploying tens of thousands of machines per month is not sustainable to do with humans in the traditional way. Both in the front of the house (servers,network gear, etc) and the back of the house (facilities). It's a tough message but one I won't shy away from. I have one of the finest teams on the planet in running our facilities. It's a fact, automation is key.

Around opportunity cost of failed machines in a container from a power perspective, there are ultimately two scenarios here. One is that the server has failed hard and is dead in the container. In that scenario, the server is not drawing power anyway and while the container itself may be drawing less power than it could, there is not necessarily an "efficiency" hit. The other scenario is that the machine dies in some half-state or loses a drive or similar component. In this scenario you may be drawing energy that is not producing "work". That's a far more serious problem as we think about overall work efficiency in our data centers. We have ways through our tools to mitigate this by either killing the machine remotely, or ensuring that we prune that server's power by killing it at an infrastructure level. I won't go into the details here, but we believe efficiency is the high order bit. Do we potentially strand power in this scenario? Perhaps. But as mentioned in the article, if the failure rate is too high, or the economics of the stranding begin to impact the overall performance of the facility, we can always swap the container out with a new one and instantly regain that power. We can do this significantly more easily than a traditional data center could because I don't have to move servers or racks of equipment around in the data center(i.e. more flexible). One thing to keep in mind is that all of our data center professionals are measured by the overall uptime of their facility, the overall utilization of the facility (as measured by power), and the overall efficiency of their facility (again as measured by power). There is no data center manager in my organization who wants to be viewed as lacking in these areas and they give these areas intense scrutiny. Why? When your annual commitments are tied to these metrics, you tend to pay attention to them.

The last hurdle here revolves around the life expectancy of a server and technology refresh change rates and somehow the price of diesel and green-ness.

"Intel is trying to get more and more power efficient with their chips," Biggs said. "And we'll be switching to solid-state drives for servers in a couple of years. That's going to change the power paradigm altogether." But replacing a container after a year or two when a fraction of the servers are actually broken "doesn't seem to be a real green approach, when diesel costs $3.70 a gallon," Svenkeson said.

Clear as mud to me. I am pretty sure the "price of diesel" in getting the containers to me is included in the price of the containers. I don't see a separate diesel charge. In fact, I would argue that "shipping around 2000 servers individually" would ultimately be less green or (at least in travel costs alone) a push. In fact, if we dwell a moment longer on the "green-ness" factor, there is something to be said for the container in that the box it arrives in is the box I connect to my infrastructure. What happens to all the foam product and cardboard with 2000 individual servers? Regardless, we recycle all of our servers. We don't just "throw them away".On the technology refresh side of the hurdle, I will put on my business hat for a second. Frankly, I don't know too many people who depreciate server equipment less than three years. Those who do, typically depreciate over one year. But having given talks at Uptime and AFCOM in the last month the comment lament across the industry was that people were keeping servers (albeit power inefficient servers) well passed their useful life because they were "free". Technology refresh IS a real factor for us, and if anything this approach allows us to adopt new technologies faster. I get to upgrade a whole container's worth of equipment to the best performance and highest efficiency when I do refresh and best of all there is minimal "labor" to accomplish it. I would also like to point out that containers are not the only technology direction we have. We solve the problems with the best solution. Containers are just one tool in our tool belt. In my personal experience, the Data Center industry often falls prey to the old adage of “if your only tool is a hammer then every problem is a nail syndrome.”

Challenge 4: Containers are temporary, not a long term solution.

Well I still won't talk about who is in the running for our container builds, but I will talk to the challenges put forth here. Please keep in mind that Microsoft is not a traditional "hoster". We are an end user. We control all aspects of construction, server deployments and applications that go into our facilities. Hosting companies do not. This section challenges that while we are in a growth mode now, it won't last forever, therefore making it temporary. The main point that everyone seems to overlook is the container is a scale unit for us. Not a technology solution for incremental capacity, or providing capacity necessarily in remote regions. If I deploy 10 containers in a data center, and each container holds 2000 servers, that's 20,000 servers. When those servers are end of life, I remove 10 containers and replace them with 10 more. Maybe those new models have 3000 servers per container due to continuing energy efficiency gains. What's the alternative? How people intensive do you think un-racking 20000 servers would be followed by racking 20000 more? Bottom line here is that containers are our scale unit, not an end technology solution. It’s a very important distinction that seems lost in multiple conversations. Hosting Companies don't own the gear inside them, users do. It’s unlikely they will ever experience this kind of challenge or need. The rest of my points are accurately reflected in the article.

Challenge 5: Containers don't make a data center Greener

This section has nothing to do with containers. This has to do with facility design. While containers may be able to take advantage of the various cooling mechanisms available in the facility the statement is effectively correct that "containers" don't make a data center greener. There are some minor aspects of "greener" that I mentioned previously around shipping materials, etc, but the real "green data center" is in the overall energy use efficiency of the building.

I was frankly shocked at some of the statements in this section:

An airside economizer, explained Svenkeson, is a fancy term for "cutting a hole in the wall and putting in a big fan to suck in the cold air." Ninety percent more efficient than air conditioning, airside economizers sound like a miracle of Mother Nature, right? Except that they aren't. For one, they don't work — or work well, anyway — during the winter, when air temperature is below freezing. Letting that cold, dry air simply blow in would immediately lead to a huge buildup of static electricity, which is lethal to servers, Svenkeson said.

Say what? Airside economization is a bit more than that. I am fairly certain that they do work and there are working examples across the planet. Do you need to have a facility-level understanding of when to use and when not to use them? Sure. Regardless all the challenges listed here can be easily overcome. Site selection also plays a big role. Our site selection and localization of design decides which packages we deploy. To some degree, I feel this whole argument falls into another one of the religious wars on-going in the data center industry. AC vs. DC, liquid cooled vs. air cooled, etc. Is water-side economization effective? Yes. Is it energy efficient? No. Not at least when compared to air economization in a location tailor made for it. If you can get away with cooling from the outside and you don't have to chill any water (which takes energy) then inherently it’s more efficient in its use of energy. Look, the fact of the matter is we have both horses in the race. It’s about being pragmatic and intelligent about when and where to use which technology.

Some other interesting bits for me to comment on:

Even with cutting-edge cooling systems, it still takes a watt of electricity to a cool a server for every watt spent to power it, estimated Svenkeson. "It's quite astonishing the amount of energy you need," Svenkeson said. Or as Emcor's Baker put it, "With every 19-inch rack, you're running something like 40,000 watts. How hot is that? Go and turn your oven on."

I would strongly suggest a quick research into the data that Green Grid and Uptime have on this subject. Worldwide PUE metrics (or DCiE if you like efficiency numbers better) would show significant variation in the one for one metric. Some facilities reach a PUE of 1.2 or 80% efficient at certain times of the year or in certain locations. Additionally the comment that every 19inch rack draws 40kw is outright wrong. Worldwide averages show that racks are somewhere between 4kw and 6kw. In special circumstances, densities approach this number, but as an average number it is fantastically high.

But with Microsoft building three electrical substations on-site sucking down a total of 198 megawatts, or enough to power almost 200,000 homes, green becomes a relative term, others say. "People talk about making data centers green. There's nothing green about them. They drink electricity and belch heat," Biggs said. "Doing this in pods is not going to turn this into a miracle."

I won't publicly comment on the specific size of the substation, but would kindly point someone interested in the subject to substation design best practices and sizing. How you design and accommodate a substation for things like maintenance, configuration and much more is an interesting topic in itself. I won't argue that the facility isn't large by any standard; I'm just saying there is complexity one needs to look into there. Yes, data centers consume energy, being "green" assumes you are doing everything you can to ensure every last watt is being used for some useful product of work. That's our mission.

Challenge 6: Containers are a programmers approach to a mechanical engineer's problem.

As I mentioned before, a host of professional engineers that work for me just sat up and coughed. I especially liked:

"I think IT guys look at how much faster we can move data and think this can also happen in the real world of electromechanics," Baker said. Another is that techies, unfamiliar with and perhaps even a little afraid of electricity and cooling issues, want something that will make those factors easier to control, or if possible a nonproblem. Containers seem to offer that. "These guys understand computing, of course, as well as communications," Svenkeson said. "But they just don't seem to be able to maintain a staff that is competent in electrical and mechanical infrastructure. They don't know how that stuff works."

I can assure you that outside of my metrics and reporting tool developers, I have absolutely no software developers working for me. I own IT and facilities operations. We understand the problems, we understand the physics, we understand quite a bit. Our staff has expertise with backgrounds as far ranging as running facilities on nuclear submarines to facilities systems for space going systems. We have more than a bit of expertise here. With regards to the comment that we are unable to maintain a staff that is competent, the folks responsible for managing the facility have had a zero percent attrition rate over the last four years. I would easily put my team up against anyone in the industry.

I get quite touchy when people start talking negatively about my team and their skill-sets, especially when they make blind assumptions. The fact of the matter is that due to the increasing visibility around data centers the IT and the Facilities sides of the house better start working together to solve the larger challenges in this space. I see it and hear it at every industry event. The us vs. them between IT and facilities; neither realizing that this approach spells doom for them both. It’s about time somebody challenged something in this industry. We have already seen that left to its own devices technological advancement in data centers has by and large stood still for the last two decades. As Einstein said, "We can't solve problems by using the same kind of thinking we used when we created them."

Ultimately, containers are but the first step in a journey which we intend to shake the industry up with. If the thought process around containers scares you then, the innovations, technology advances and challenges currently in various states of thought, pilot and implementation will be downright terrifying. I guess in short, you should prepare for a vigorous stirring of the anthill.

Struggling with CADE, McKinsey / Uptime Metric

Mon, 05 May 2008 11:34:16 GMT

I guess I should start out this post with the pre-emptive statement that as a key performance indicator I support the use of CADE or metrics that tie both facilities and IT into a single metric. In fact we have used a similar metric internally at Microsoft. But the fact is at the end of the day I believe that any such metrics must be useful and actionable. Maybe its because I have to worry about Operations as well. Maybe its because I don't think you roll the total complexity of running a facility with one metric. In short, I don't think dictating yet another metric, especially one that doesn't lend itself to action, is helpful.

As some of you know I recently gave keynote speeches at both DataCenter World and the 2008 Uptime Symposium. Part of those speeches included a simple query of the audience of how many people are measuring energy efficiency in their facilities. Now please keep in mind that the combined audience of both engagements numbered between 2000-2400 datacenter professionals. Arguably these are the 2400 that really view data centers as a serious business within their organizations. These are folks whose full time jobs are running and supporting data center environments for some of the most important companies around the world. At each conference less than 10% of them raised their hands. The fact that many in the industry including Ken Brill at the Uptime Institute, Green Grid, and others have been preaching about measurement for at least the last three years and less than 10% of the industry has accepted this best practice is troublesome.

Whether you believe in measuring PUE or DCIE, you need to be measuring *something* in order to even get one variable of the CADE metric. Given this lack of instrumentation and\or process within those firms most motivated to do so speaks in large part of the lack of success this metric is going to have over time. It therefore follows, if they are not measuring efficiency, they likely don't understand their total facility utilization (electrically speaking). The IT side may have an easier way of getting variables for system utilization, but how many firms have host level performance agents in place?

I want to point out that I am speaking to the industry in general. Companies like ours who are investing hundreds of millions of dollars get the challenges and requirements in this space. Its not a nice to have, its a requirement. But when you extend this to the rest of the industry, there is a massive gap in this space.

Here are some interesting scenarios that when extended to the industry may break or complicate the CADE metric:

As you cull out dead servers in your environment, your utilization will drop accordingly and as a result the metric will remain unchanged. The components of CADE are not independent. Dead servers are removed so that Average server utilization goes up then Data Center Utilization goes down showing proportionally so there is no change and if anything PUE goes up which means the metric may actually go up. Keep in mind that all results are good when kept in context of one another.
Hosting Providers like Savvis, Equinix, Dupont Fabros, Digital Realty Trust, and the army of others will be exempt from participating. They will need to report back of house numbers to the their customers (effectively PUE). They do not have access to their customers server information It seems to me that CADE reporting in hosted environments will be difficult if not impossible. As the design of their facilities will need to play a large part of the calculation this makes effective tracking difficult. Additionally, overall utilization will be measured at what level?
If hosters exempted, then it gives CADE a very limited application or shelf-life. You have to own the whole problem for it to be effective.
As I mentioned, I think CADE has strong possibilities for those firms who own their entire stack. But most of the datacenters in the world would probably not fall into "all-in" scenario bucket.

I cant help but think we are putting the cart before the horse in this industry. CADE may be a great way to characterize data center utilization but its completely useless if the industry isnt even measuring the basics. I have come to the realization that this industry does a wonderful job in telling its members WHAT to do, but lacks to follow-up with the HOW. CADE is meant for higher level consumption. Specifically those execs who lack the technical skill-sets to make heads or tails of efficiencies and how they relate to overall operations. For them, this metric is perfect. But we have a considerable way to go before the industry at large gets there.

Regardless, I strongly suggest each and everyone adopt the take away at Symposium....Measure, measure, measure.

The Passing of an Icon....

Sun, 03 Feb 2008 06:13:59 GMT

I was informed that Larry Rushing of HP passed on today. Personally I am incredibly saddened by this news. This is truly a sad day, not just for his heartbroken family, but also for the data center industry at large, whether it knows it or not. Simply put, Larry was an icon. Larry was one of those rare individuals who had an immediate impact on those he interacted with. Most recently placed at the spearhead of engineering and constructing HP's data center consolidation efforts, he was the consummate professional. Regardless of where you stood on the individual engineering discussions, you could always count on Larry for a good discussion. His friendly approach and easy style immediately endeared him to you. Those of us in the industry challenged with Data Center Construction and Engineering had a great peer and sounding board in Larry.

To Larry's family, know that his humor and easy going nature enlightened the lives of many around the industry and he will be dearly missed.

To Larry - Our long discussions on power density, redundancy, and design were incredibly fun. As I am sure you are now in the most perfect of places, sharing in the knowledge of the universe, and you finally have the answers the rest of us fools continue to search for.

God Bless you, Larry.

Upcoming Speaking Engagements

Sun, 20 Jan 2008 08:02:36 GMT

For those of you interested I thought I would post some of the upcoming speaking engagements I will be participating in for the next few months. Microsoft is doing a lot of incredible stuff on the data center front these days and I will be sharing some thoughts and insights at these events. If you are attending any of these events or are in the area, please feel free to stop by and say Hi!

03/10/08 - IASA - IT Architect / Regional Conferences 1:45pm - 3pm

Washington, DC

04/01/08 - Data Center World - Keynote Talk 7:45am - 8:45am

Las Vegas, Nevada

04/27/08 - Uptime Institute Symposium / Green Enterprise Computing

Orlando. Florida

2008 Data Center Links of Interest....

Sat, 12 Jan 2008 22:15:32 GMT

Every once in awhile I publish the list of Data Center RSS feeds I read on a regular basis for my internal teams in case they are interested. I thought I should share widely in case any one else might be interested as well. I update this list pretty regularly and will post this link periodically. I use the RSS feature at LIVE.COM which allows me to share my collected feeds as well. The link below will take you to my daily news feed.

http://www.live.com/?addTemplate=51a85671-6e19-40da-ad2b-af5892034eb7

Darwinism on Holiday Vacation

Sat, 22 Dec 2007 08:21:54 GMT

This holiday season my family and I went to Leavenworth, Washington again for a few days. It has become a holiday tradition. For those uninitiated, Leavenworth is a wonderful "Bavarian" village in the Cascade mountains in the American Northwest. During the Christmas season, the town goes all out and you do feel magically transformed as if you were in Bavaria itself. Street signs are in both German and English, the town is lit up with festive holiday spirit complete with a "lighting ceremony" where all of the businesses throw on their Christmas lights all at the same time. There is music pumping in the city streets, crowds wander aimlessly through the shops and bakeries, and its generally impossible to skip the atmosphere it creates. [Pictures forthcoming on my personal website].

During the winter, just off the main street (Hwy 2) is an area adjacent to the major shopping "strip" where the geography of the land is such that it makes an incredible little sled run for kids. You simply cannot pass this area without seeing it packed with little kids and families sledding and having great fun regardless of time of day. As we have come to do every year, we feverishly participate in this event every year as well. My kids love it. This year was particularly fantastic as the conditions made for an incredible mix of snow and heat to make the sledding fast, furious, and fun.

It was here, amidst the enjoyment of seeing the thrill on the faces of my little ones that I started to think about Darwinism, at least on a small personal scale. The process was quite simple really. Get to the top of hill, jump on sled (or in our case plastic discs which were much faster), slide down hill, once at bottom, make sure you do not get demolished by another kid (or parent) on their way down, make it to a safe area to climb back up and repeat the cycle. Simple right?

Apparently not.

My wife has done a wonderful job raising polite and courteous children. Really, no thanks to me on this one. She has done a great job. My kids would patiently wait for the field below to be clear of idiots ..er...people, then slide down. A situation that honestly frustrated me terrifically. My encouragement for my children to just race down and the people would be forced to get out of your way one way or another was looked down upon by my daughter, who, being a better human being, told me that it was just plain rude. She is right of course. But something primal tugged at me. Something told me that we were somehow fundamentally defying the laws of nature. Not to mention the fact I kept running through a new equation in my head that measured maximum enjoyment per minute sledding but had constraints put upon it by these fun-velocity regulators.

Some kids would mill about for minutes standing around smiling like idiots. Others would attempt to climb the very sled run they just went down despite kids above getting ready to careen down the hillside. My kids would patiently wait, and I of course would just sit there in my own head going, "take that idiot out." They would either learn their lesson and start paying attention, or they would be culled from the human herd.

I began to try and attempt a pseudo-scientific study and watch the field beneath me. While my kids were doing the honorable thing, others had no intention and were bloodthirsty in their need for excitement. We had gotten there early, and the sled hill was relatively sparsely populated. So the number of kids/people going down were pretty small. A quick assessment of the participants quickly put them into one of two categories. Lets call it, non-observants versus optimizers. As my kids were trying to do the right thing, I left them out of the study as they would clearly be optimizers given their propensity to watch the hill as they ran to side to ascend the hill and begin again. They would temper this by actually waiting until the field was clear, so they were kind of a control group I guess. Regardless, back to my equation and scientific study. I closely watched the ratio of non-observants versus optimizers throughout the day. At some point, my wife and daughter went off to shop and it was just me and my two boys. Within those two, I have one "control-group" and one clear optimizer. As the day progressed, I expected to see the ratios remain somewhat constant. Much to my surprise, I was proved wrong in a big way.

As the human herd got larger, the amount of "non-observants" by percentage actually increased and at times in great volume, while the optimizers grew at a dramatically slower rate or actually decreased. The non-observant herd was always a larger percentage of the field. I found this fascinating. I would watch the optimizers rocket down the hill and take out many a slow-moving herd member. In many cases, that member would, after a stint on the side licking wounds, would re-appear on the field, slide down, and be taken out yet again. Lesson clearly not learned.

An interesting side observation was the emergence of a sub-group which was those that did learn. Mostly. It was almost more fascinating to watch those that learned to watch the hill as it had an interesting survival of the fittest feel to it. Those children would learn that Kid#1, and Kid#2 were optimizers, and would as a result just watch the hill for those kids. Images of early humans learning that fighting a bear alone might not be the smartest idea came to mind specifically. These were good lessons to learn, that is until optimizer kid #3 arrived and promptly took them out yet again.

Now I must point out that the optimizer herd also culled itself, but in a very different way. They would usually self-select out by sliding down the hill standing up on their sleds crashing in a huge display of flailing limbs. They could also take themselves out after building moguls in the snow which would launch them into the air allowing them to crunch down on the side of their heads. Its important to note that while they took themselves out, they were always smart enough not to be taken out by someone else.

I guess there are two lessons here for all, First - As I think about it, I really am a geek around optimization and should clearly pick up a hobby. Second and perhaps most importantly, if you ever find yourself in Leavenworth make sure it is my polite and courteous children and not me sledding down the hill or I guarantee you I will enforce the laws of nature and take you out.

Microsoft Data Center Services Hiring UPDATED

Thu, 06 Dec 2007 12:21:01 GMT

Are you or Is someone you love bored at work at your dreary Data Center job? Do you know a facilities genius? Do you think you have what it takes to run and operate a large scale Data Center Operation? Are you a wizard of construction management?

If you said 'Yes' to any of these questions, we would like to talk to you!

Ok so its a bit corny, but thought I would use this forum to advertise some current open positions within my organization. If you do have a serious interest please drop me a line!

Current Open Positions

Data Center Manager, Chicago, Illinois

Facilities Program Manager, Chicago, Illinois ** FILLED **

Facilities Program Manager, San Antonio, Texas

Technical Program Manager, Seattle, WA

Technical Program Manager, Dublin, Ireland ** FILLED **

Data Center Manager, San Jose, CA.

Electrical Engineer (PE's only please), Seattle, WA ** FILLED **

Mechanical Engineer (PE's only please), Seattle, WA ** FILLED **

DC Construction Program Manager, Any

Happy Thanksgiving!

Fri, 23 Nov 2007 03:27:01 GMT

I wanted to take a quick moment to wish everyone a Happy Thanksgiving holiday. I hope your day is a special one surrounded by people you love.

Sincerely,

Mike

TechEd 2007: Green Data Center Panel

Thu, 15 Nov 2007 17:18:12 GMT

Barcelona, Spain - Just finished up the Green Data Center Panel and as expected it was a lively debate. The crowd had great questions that were tossed about fast and furious. Additionally there was a good bit of poking and jabbing on the Panel as well. The Panel consisted of representatives of AMD, Intel, HP, Dell, Microsoft and CitiGroup. The panelists fell into one of three categories: Chip Manufacturers, OEMs, and Data Center Users. While I was representing the User perspective you can bet that there were lots of product questions thrown in for good measure to boot. I handled most of those ok, but if they drilled any further down I would have called in Dev support.

Regardless there was a great deal of energy and verbal combat, violent agreement, and I swear that Intel guy threw his water bottle at me :) . There was only about 60 or so attendees, a small fraction of the overall event, but you can tell that this topic is coming up more and more. It was also interesting to speak with my colleague from the Citi Group. They are also building a large scale facility and it was interesting to trade some stories and shared experiences.

I am heading out to dinner this evening and have an early flight home tomorrow morning (6am ouch). I am REALLY looking forward to getting home. This travel schedule is really wearing a bit thin. Luckily I will have the luxury of staying put for some time after this trip.

Mike

New Green Data Center Blog to watch

Thu, 15 Nov 2007 01:31:02 GMT

Barcelona, Spain - I had a chance to talk with Dave O'Hara at TechEd today. Dave is the author of the recent Tech Net article on "Going Green" which is a great read. His TechNet Article can be found here. Dave informed me that he has just launched his own blog site. As Dave has a keen interest in the data center space, you may want to check it out. The link for his site can be found below:

http://greenm3.typepad.com/green_m3_blog/atom.xml

Mike

TechEd 2007 : Special Data Center Presentation

Wed, 14 Nov 2007 23:19:21 GMT

Barcelona, Spain - Today I gave a special presentation at Microsoft's TechEd 2007 conference in Barcelona. The presentation was all about Microsoft's strategy and approach with regards to Data Centers and scale. While a lot of the stuff we do is embargoed for competitive reasons, its always interesting to see just how much interest there is in the industry for even a small glimpse at solving these kinds of issues.

Its also a little daunting to be standing in an auditorium that holds 3000 people to give a talk. The session went really well and I had a ton of folks come up to me after the talk. I love interfacing with our customers in this way. All in all a great day. The after event today was held at a location closer to the Mediterranean and I spent some great time speaking with folks at CERN about operations at scale and the challenges of running complex operations.

Tomorrow I participate in the Green Data Center Panel discussion with experts from across Europe, its likely to be a lively debate. More on that tomorrow.

Mike

The Need for Independents

Tue, 13 Nov 2007 20:30:27 GMT

Data Center Knowledge reported the acquisition of EYP by HP this week. You can see the article here. I have always had a tremendous amount of respect for both organizations. Personally I am very happy for Peter Gross. Peter and I have known each other for a a couple of years and I am sure that this move makes his life much easier in many ways. We have had some really great conversations over time. Hell, he has even given me a lift to the airport when I found myself stranded. He is an all around great man and a tremendous resource for the Mission Critical Facility world.

To tell you the truth,, I have mixed emotions about this deal by and large. Architectural and Engineering firms should really be about the design. The best design for the application or challenge you are trying to solve. Those engineers and architects should be able to view the entire field of product choices and select the best fit. In short, they should be independent. Not tied to a particular vendor, or solution set. It creates to many conflicts of interest. While both firms have impeccable backgrounds, the fact that HP is selling its Smart Cooling product sets provides a conflict of interest for the pure design firm. What if that solution doesn't work for someone? What if it does? How do you know your engineering firm just wont become a shill for whatever product set owns them. I want to restate, I don't necessarily believe that this is the case with this deal. It holds true for any other equipment manufacturer that purchases an engineering firm. Its just one more thing to be wary of.

Its an interesting turn of event that will ultimately likely play out over time. Additionally it will be interesting to see if other hardware and equipment makers follow suit.

For now I will sit back and watch how this plays out.

Mike

Now that the Dust is settling....

Fri, 09 Nov 2007 07:00:09 GMT

I know, I know, its been a while since I have posted to my external blog. I have received a lot of email from many of you giving me grief for not being very active on this site. Even the folks who regularly read my internal blog at Microsoft have mentioned to me that I have been remiss for some time on this space. Well I would bet based on all the news this week everyone has a pretty good idea of some of the things that have been consuming all of my time. Two facilities launched in the same week.

Internally we refer to our facilities by their "class". A class is a collection design characteristics: performance requirements, density, energy efficiency, mechanical optimization, time to market, construction optimizations, cost optimizations, specific PUE targets, and other requirements combined with the overall facility layout. Its similar to how the Navy classifies it's ships. Previously the most famous of which was our "Quincy-Class" facilities. So named due to the fact that their initial debut was in Quincy, Washington. This class has proven incredibly versatile and we have made considerable enhancements to this design with each new iteration and placement resulting multiplicative gains in efficiency and the like. They are simply impressive inside and out.

But as I said, this week represents the culmination of months of work highlighting how the teams really outdid themselves. While the Quincy-class still has significant sea legs, this week brought about two brand new classes. The new Dublin and Chicago-classes have each have very specific design characteristics. Which begs the obvious question: Why three different classes? Its a good question but if you think about it, it has an obvious answer. For one, as I mentioned above there are different design characteristics, essentially they solve different problems or challenges or have been optimized for specific functions. In the end -for all classes-Its all about efficiency. Efficiency in every aspect of the facility. Efficiency in cost. Efficiency in construction. Efficiency in energy consumption, Efficiency in workflow. While data centers *could* look the same (physically, electrically, and mechanically) it doesnt mean they should. To balance against this you also don't want to build one of a kind facilities of intricate Rube Goldberg-ian complexity. Hence our class approach.

With the incredible amount of Software + Services in the pipeline for the cloud services vision outlined by Ray Ozzie, we are really poised to do some ground breaking work. Both from a services and and data center perspective. Those key services with incredible features and functionality need the infrastructure to back it up. I can honestly say that I feel that while Moore's law has had a drastic effect on the equipment and technology within a data center, it has unkindly ignored the facility that housed them. In my personal opinion, together these three classes rectify that situation.

Mr. Moore, I would like to introduce you to the Data Center- Courtesy of the long hours and dedication of Microsoft's Global Foundation Services.

Booyah!

-Mike

PS - I used the new Windows Live Writer for this post, have to say its an awesome blog tool and I am not just saying that because I work here.

Mega Data Center Growth

Sat, 01 Sep 2007 00:45:08 GMT

Alot of you out there are wondering how big is the data center growth out there. A friend of mine sent me this image which outlines some of the larger players in the space. I think the data may be a bit stale, but its an interesting look at how much capital is going into Data Center related spend across just a few companies in this space.

/Mm

Evaporative Chillers are not new technologies!

Tue, 28 Aug 2007 23:00:06 GMT

If I read one more article about how innovative this technology is I am going to start selling my hi-fidelity cassette players on Ebay again.

Rare Endangered Species: Data Center Manager

Tue, 28 Aug 2007 22:57:13 GMT

Computer World published an interesting article on Data Center Managers and the increased demand for people with these kinds of skill-sets. The link for the article is " here." As I thought about the points made in the article some things came bubbling up to the top of mind. The article outlines how Data Center Managers really need to be broad thinkers and solutions architects in a range of technologies much broader that just IT. This assessment is dead-on in its direction, but really doesnt go far enough in my mind on the challenges this type of roll actually has to deal with today or will deal with tomorrow. It mentions the basic requirements: Must be process-centric, Have strong program / project management skills, deep understanding of both IT (servers, network, storage) technologies as well as electrical and mechanical facilities technologies (chillers,CRAC/CRAH, Generators, Rotary or Battery systems, etc) and Security. This is a good rough idea of the skill-sets required, but the job is getting much more complicated than that in my opinion.

While the topic of "green" is briefly mentioned, I dont really see that as being a specific requirement for the job. Most of the sustainable decisions are made during site selection, facility design, and the creation of standard operations processes - which you do once. The Data Center manager wont have much to do with that space outside of reporting (though its still another task on an already full plate). However, another almost minute mention is around regulatory requirements. This is extremely big. The modern data center manager probably goes through numerous audits throughout the course of the year. Whether its just Sarbannes-Oxley in a small localized facility or the or you are pursuing an ISO certification, the Data Center Manager has a huge role in that process. Process credit card information? We have an audit for that! Store Personally Identifiable Information? We have an audit for that! Some of the larger more prolific data centers go through 10-20 audits per year. If you have international facilities compound that problem even further. Sure, the questions are similar, but its more cycles and regulatory stuff they have to deal with.

While not formally announced companies can probably bet on the fact that the EPA and DOE involvement on Data Center Power Consumption will lead to some guidelines and reporting around facility and energy efficiency, carbon footprint, and the like. These tasks will likely fall to the DCM for any given company as they sit in the only place that could accurately collect this information.

Data Centers are becoming critical infrastructural components to many companies whether they know it or not. The progressive companies who understand this will succeed and those that dont will probably learn it the hard way. For example, if you reference back to the article it talks about how the DCM must be the jack of all trades, yet it boggles my mind that the industry has resisted collapsing the Facilities Teams and the Traditional IT into the same function. In my role I get to talk to alot of Data Center Managers at customer companies. I cannot tell you how often in the real world that the disconnect between disparate Facility and IT functions impacts schedule, deployments, and lack of coordinated efforts.

The large players in this space get it. The question is does your company?

In my mind two things are certain - The first is that the demand for quality Data Center Managers is very high. My caution to you is that your organization may not understand the true cost of having a good one until you lose them. The second thing is if you lose them, they wont be on the market long.

/Mm

Chicago Spoken Here...

Mon, 27 Aug 2007 10:16:22 GMT

I was in a meeting this last week with a good mix of some members from the Data Center Services team and some vendors. One of the vendors was from Chicago and coincidently came from the same neighborhood on the south-side as I did. This proved somewhat fatal to the meeting at least from a communications standpoint for everyone else.

As some of you already know my "Chicago" accent at times comes through fairly strong. Having moved to Seattle where the only accent is a decided lack of accent, some might say I stick out a bit. Ahh, but put me in a room with someone else from Chicago and my "sout-side" comes shining "trew." :) It turns out we both knew a guy, who knew a guy, who we both knew. He lives "over bydare". Which is important to someone from Chicago, a city that works on relationships, favors, and neighborhoods. To everyone else its just plain weird. They didnt have anything we wanted to buy but I had some great fun.

For those of you who ever find yourself in a similar situation with me, I found a fun little link off of the Chicago Sun Times called the Chicagopedia. A guide to speaking like a Chicagoan. The link is below, and I hope they keep it up because they are missing quite a few. But at least its a start.

http://www.suntimes.com/chicago/chicagopedia/index.html

/Mm

I might just do it....

Mon, 27 Aug 2007 09:47:35 GMT

My day job involves building and running data centers for one of the largest companies on the planet.

Like most of the folks in my industry we follow the news in this space actively.    There is just a ton of activity going on, and for the first time in a long time there is actually real innovation happening in a space that has lacked significant advances for some time (my opinion).   If you walk into a standard data center today, it would look much the same as one did 10 to 15 years ago.

That is to say if you arent comparing to the latest generation data centers.   See thats the thing, there are a few companies out there today really pushing the envelope in driving efficiency, operations, and data center design.   Alot of these advances are happening inside the facilities, and unless you are well versed in electrical and mechanical engineering you might not see some of these advances.

One of the things that really irks me is the general "black box" view most of the tech industry has around data centers.   They are mystical places where physics and thermodynamics weave an intricate dance of kinetic revolution where the lifesblood of electricity gives animation and life to the living Internet. In the immortal words of Spanky from the Lil' Rascals, "Ah Phooey".

Yet this belief allows for the announcement of just about anything related to them that even sounds technical to be revered with awe-inspiring devotion to the non-initiated.

As an old data center dog, most of these "public" innovations are either extrapolations of things that facilities professionals have been doing for years, or in some cases are themselves things that are already wide spread in the industry.

I think what the industry needs is a Data Center for Dummies book, where the basic tenets of Data Center design, technologies, and the like are discussed. It could also talk about metrics that really matter. I have spoken with lots of data center professionals across a broad spectrum of industries and I know they share my general frustration in this space. Maybe I should write one. Of course it would be my own religion about facilities and operations and the growing complexity for global scale computing.

Most of the books on data centers today talk about operating systems, virtualization, or rack and server layout, and only a very few talk about the facilities themselves. Im not sure anyone would buy it, but given the emerging importance of these complex buildings, I am definitely considering it.

What do you think?

/Mm

Launch of a Public Version of my internal Blog

Mon, 27 Aug 2007 09:34:14 GMT

Well I have finally gone and done it. I am launching a public blog site to talk about technology, life, data centers, software plus services, managed services, and anything else that strikes my fancy. As some of you know, I have been an active blogger inside Microsoft for some time. My internal readership has encouraged me to do something like this for some time. As you would imagine, some of the entries there are for an internal audience only, others may have applicability in a more public forum. Regardless, one thing is for certain, the web will now suffer from yet another blogger who thinks he knows what he is talking about.

Mm

Random Thoughts: Blog

My Blog has moved.

The Stars are Bright....

IT Victimization and Lack of Adoption of Energy Efficiency Focus in the Data Center

Microsoft Environmental Site

The Phoenix has Landed!

Fingers crossed for Phoenix Lander

Im a Jackass, its official :)

Stirring Anthills ... A response to the recent Computerworld Article

Struggling with CADE, McKinsey / Uptime Metric

The Passing of an Icon....

Upcoming Speaking Engagements

2008 Data Center Links of Interest....

Darwinism on Holiday Vacation

Microsoft Data Center Services Hiring **UPDATED**

Happy Thanksgiving!

TechEd 2007: Green Data Center Panel

New Green Data Center Blog to watch

TechEd 2007 : Special Data Center Presentation

The Need for Independents

Now that the Dust is settling....

Mega Data Center Growth

Evaporative Chillers are not new technologies!

Rare Endangered Species: Data Center Manager

Chicago Spoken Here...

I might just do it....

Launch of a Public Version of my internal Blog

Microsoft Data Center Services Hiring UPDATED