My post last week on the open source hypervisor Xen got slashdotted and ended up getting a lot of comments on another site, you can check them out here. The commenters contributed a lot of interesting information about Xen and how it is being used. Here is some of what I learned.

One thing that seems indisputable is that Xen is doing well in ISP/hoster/Internet-facing environments. This is because it's obviously much cheaper than VMware, its performance at least for hosting Linux may be quite a bit better, and the immaturity of the management tools doesn't hurt as much as in the corporate data center. Perhaps the most notable example is Amazon, which is apparently running several thousand servers under Xen at its EC3 "compute capacity in a cloud" site.

But it also seems indisputable at this point that Suse and especially Red Hat have dropped the ball as far as Xen integration is concerned. Maybe things will get better over time, but for now what the commercial Linux distros have delivered for Xen falls far short of the expectations they themselves created not so long ago. Just last week, on the very same day I posted the piece, Red Hat CEO Matthew Szulik confessed to Wall Street that hardly any Red Hat customers have put Xen into production with RHEL5. His comments are pretty amazing:

"...in the last two weeks, I have spoken with five very prominent CIOs, and none of them have moved virtualization near their production environment. Some of the discourse that they have with me is that, although virtualization has been effective in some single server environments, as a way to potentially reduce the cost of adding additional servers, and potentially to add more value to a single-server environment, none of them have described moving virtualization into production environments. They actually have almost uniformly described virtualization as still being a young technology, and awful lot of application dependency and workflow issues needed to be discovered and worked through. The requirement of management services and other discovery technologies are still quite immature to move it into a full enterprise wide scale out."

I guess he's not talking to very many VMware customers, huh? Otherwise he might have noticed that quite a few of them are definitely "in production" and no longer think of virtualization as a "young technology". Another interesting point is that Red Hat has stopped saying the word "Xen" in public, all you get are generic references to "virtualization". There is a very interesting story behind this verbal quirk that I will try to cover in a future post.

But in writing last week's post I probably should have drawn a sharper distinction between Xen and XenSource. The latter of course is the VC funded startup that employs many of the Xen project contributors, including its founder Ian Pratt. The day after I posted the piece, XenSource told Reuters it now has 500 customers, most of them running "just a handful of servers", aside from the big hoster examples like Amazon already mentioned. But having seen its sales boom in the last few quarters (presumably on the strength of XenEnterprise and its ability to host Windows Server as well as Linux), the startup now hopes to double or triple its installed base by the end of the year. That would certainly suggest that they are advancing faster up the adoption curve than VMware was at a similar point in its life cycle.

The big difference in XenSource's approach compared to the distros is that they are positioning their XenEnterprise product (which is a hybrid of open source Xen and their own closed source management tools) as a direct competitor to VMware ESX, something that Red Hat and Novell have not done with their more basic Xen offerings. Given the rapid rise of KVM, maybe Red Hat feels it doesn't have to play along with the Xen team to get a robust open source virtualization solution integrated into Linux. That may be so, but I have to wonder if Red Hat isn't underestimating customer demand for an OS-independent hypervisor that has serious management tools and does a good job with both Microsoft and open source stacks. VMware already has a solution like that, and XenSource is trying to put one together. Microsoft seems to be hoping that Novell and XenSource will help make it a serious contender in the dual stack game too.

In any case, in the run up to Longhorn/Viridian and on the eve of what is likely to be a sensational VMware IPO, XenSource is certainly worth a closer look.

What's going on with Xen, the open source hypervisor that was supposed to give VMware a run for its money? I can't remember how many IT trade press articles, blog posts and vendor white papers I've read about Xen in the last few years. If I had a dollar for every piece ever published about Xen, I'd be... well, not quite as rich as the Google kids, but still very well off. The vast majority of those articles - including a few I've written myself - take it as an article of faith that Xen's paravirtualizing technical approach and open source business model are inherently superior to the closed source alternatives from VMware or Microsoft.

Suffice it to say there has been a lot of excitement and optimism in the pundit community about Xen. As recently as last January, influential blogger-journalist Daivd Berlind flat out predicted that:

"XenSource will eventually overtake VMWare in terms of market share..."

But now that Novell and Red Hat have both been shipping Xen in their commercial Linux distributions for some months, things have grown eerily quiet. Sure, there is still product news coming out of the Xen vendors, and we'll get to that in a moment. But what I'd really like to know is - who's actually using this stuff in production? And I mean actual end-user organizations, not ISPs or hosters. Based on the absence of Xen-related chatter, my guess is that production users of Xen are still few and far between.

The first thing a new technology like Xen needs before it can win over mainstream corporate buyers is a credible, reasonably long list of reference customers. You will look in vain for such a thing on the Red Hat or Novell web sites. Novell has put out exactly two press releases this year about Suse Linux customers who are using Xen, both very skimpy on details. Red Hat published something last year about how thrilled its RHEL4 customer Priceline was about the promised future capabilities of Xen. But that was before they actually got around to shipping Xen with RHEL5 in March. Since then, radio silence.

To be sure, the small VC-funded companies that are selling proprietary management tools for Xen - for example, XenSource and Virtual Iron - do mention a handful of customers on their web sites. But no more than a handful. Taking Novell, Red Hat, XenSource and Virtual Iron together, you'd be hard pressed to come up with a dozen named reference customers for Xen. Pretty thin gruel for a piece of software that has caused so much ink to be spilled, or should I say, page views to be generated.

I don't doubt that hundreds or perhaps even thousands of Linux geeks around the world are kicking the tires of Xen, especially those using the free distributions that include it, like Fedora. But that doesn't make Xen a serious threat to its rivals in actual deployment in real-world data centers, at least not yet. By any measure, VMware is still the 8,000 pound gorilla in this market.

For the sake of comparison, let's remind ourselves just how big VMware's footprint in the market really is. In the first quarter of 2007, the firm - soon to be partially spun off from the EMC mothership - reported license revenue of $170 million, up 84% year-on-year. Assume an average selling price of a little over $3,000 per copy for the flagship ESX hypervisor. That means VMware probably shipped more than 50,000 hypervisors in the first quarter alone, not counting all the free downloads of VMware Server (which, like Microsoft's equally free Virtual Server, is not a true "bare metal" hypervisor). VMware says it has more than 20,000 customers in the world. Based on a back-of-the-envelope extrapolation from the historical revenue numbers published in the company's April SEC S1 filing, there should be somewhere between 250,000 and 300,000 ESX licenses out there. This corresponds to roughly 1% of the world server installed base, an impressively big number. And more copies are being added every day. Looking at VMware's Q1 results, it's a safe bet that its installed base will at least double this year.

For those who believe that Xen will one day overtake VMware, these numbers set the bar pretty high. For the moment, there is nothing to suggest that Xen is achieving anywhere near the growth rate it would need to perform such a feat, even in the distant future. Of course, all great things start out small, and there is still plenty of time for Xen, since we are by all indications only at the beginning of the great virtualization wave.

But still, there is no reason to take Xen's triumph as an inevitable article of faith. If we take off our open source blinders for a moment and look at Xen objectively, we can begin to spot a few of the flaws that appear to be holding it back.

The first problem with Xen is that as a piece of software it is far less mature than VMware ESX. eWeek began its recent lab evaluation of Red Hat Enterprise Linux 5 (RHEL5, the first Red Hat version to incorporate Xen) with the sober warning that:

"... companies in search of an out-of-the-box server virtualization solution should not expect it here..."

The eWeek reviewer goes on to compare the rudimentary open source management tools Red Hat provides for Xen with the fancier proprietary stuff from the VC-funded Xen startups or VMware:

"Compared with VMware's VI3 (VMware Infrastructure 3) and with the Xen-based Virtual Iron and XenEnterprise products we've reviewed, RHEL 5's tools for creating and managing guest machines are pretty Spartan, and our experiences installing and running Windows Server 2003 and RHEL 5 guests contained more troubleshooting and Googling than we would have liked."

Red Hat is continuing to work on its GUI-based Xen installation tool virt-manager, and the just released Fedora 7 has a preview of what the future RHEL 5.1 will look like. But it seems that some assembly is still required to get Xen virtual machines up and running. For example, Fedora reviewers have run into a thicket of configuration issues:

"The Fedora team has followed the XenSource model and begun to store all VM configuration details in a database, referred to as xenstore. This enables the rest of the tools to use one consistent resource for reading configuration details, but also serves to complicate configuration changes. For example, virt-manager now shows all configured virtual machines, whether they are running or not (it used to only show running ones), which enables starting of paused or off virtual machines from within the tool. The bummer is that a complex configuration requires the manipulation of the database through a series of command line tools. Fedora team has eased this a bit through virsh (a command line tool that leverages libvirt)... to get an editable xml representation of the vm configuration... you can still use the standard Xen "xm" commands with a config file to create/start a vm, but these won't be added to the database, so when they stop, they disappear from virt-manager."

Oh my. Editable XML configuration files, obscure command line interfaces, grayed out options in the GUI? Thanks, but no thanks. This thing doesn't sound like it's ready for prime time in Data Center USA.

Of course Red Hat is aware of these maturity issues, and in recent months it has been notably muted in its public hyping of Xen, in marked contrast to last year's big PR run up to the release of RHEL5. It may even be that the Linux distributor's commitment to Xen is not as deep as advertised. To avoid lock-in, Red Hat's engineers have cleverly adopted the strategy of isolating Xen behind an abstraction layer called libvirt. This API can be used to invoke hypervisors other then Xen, such as the open source x86 instruction set emulator Qemu, or the Kernel-based Virtual Machine (KVM), a new open source hypervisor sponsored by a startup called Qumranet. Interestingly enough, Qumranet's co-founder is none other than former XenSource CTO Moshe Bar, and KVM has actually managed to get itself incorporated into the official Linux kernel (with version 2.6.20) ahead of Xen, despite the latter's multi-year head start. So it seems that even on the open source front Xen will have some real competition.

Then there is the question of performance. Xen's promoters have spent the past four years hammering into our heads the notion that paravirtualization is inherently faster than the full virtualization used by VMware. All hypervisors designed to run on x86 chips must cope with the fact that this architecture was simply not designed to make virtualization easy. This is in marked contrast to the competing server chips from the proprietary Unix vendors, like IBM's Power or Sun's UltraSparc. The latter chips carefully segregate instructions available to any random user program from those restricted to the operating system kernel itself. The x86 chips on the other hand blur this distinction, with the result that certain critical instructions produce one result when invoked by the OS kernel and a different result when invoked by a user application. This makes the traditional trap-and-emulate virtualization approach developed by IBM more than thirty years ago impossible. An x86 virtualization scheme must somehow intercept these errant instructions before they wreak havoc, and hypervisor designers have a choice of two ways to do this. Either their hypervisor must translate the offending binary code on the fly while the guest OS is running, or else they must modify the ambiguous instructions in the original source code of the guest OS prior to compilation. VMware pioneered the first approach, while Xen and Microsoft's future Viridian hypervisor adopt the second.

On the face of it, binary translation sure sounds like it would make for slower hypervisor performance than paravirtualization. After all, the latter approach lets time-critical functions like device drivers run as if they were native code. But despite this advantage on paper, the few published benchmarks comparing Xen and VMware are ambiguous. It must be noted here that VMware employs the scandalous policy of wording its user license so as to forbid the publication of benchmark data it doesn't approve of. This arrogant behavior can only be characterized as a self-awarded license to deceive the customer, because it limits competitors like XenSource to using only the performance data that VMware sees fit to allow into the public domain. Nevertheless, XenSource's own white paper on the subject suggests that Xen's paravirtualization approach produces only modest gains over ESX. In benchmark after benchmark the performance differences between Xen, ESX and native applications are usually only a few percentage points. The conclusion a reasonable observer might draw from this is that in most real-world cases these hypervisors are not really CPU-bound at all, but limited chiefly by available memory and disk input/output performance.

Another question hanging over Xen performance concerns the availability of paravirtualized drivers for Windows. As a practical matter, it seems that Xen in RHEL5 is mostly being used as a partitioning solution for hosting multiple virtual copies of RHEL5 or perhaps RHEL4 on a single RHEL5 physical machine. But the vast majority of data centers these days run Microsoft and Linux stacks side-by-side. Even Steve Ballmer has recognized this. One of VMware's strengths is that it works extremely well with both stacks, and observation suggests that a large number of VMware customers take full advantage of this fact.

Since Microsoft's OS kernel source code is not freely available for modification by third parties the way Linux is, Xen can't make the same performance promises for Windows Server that it can for RHEL or SLES. However, a developer with access to Microsoft's proprietary HyperCall APIs can in principle circumvent this problem by leveraging the virtualization-friendly architecture tweaks in recent Intel and AMD chips (known as Intel VT and AMD-V respectively). XenSource has already released its paravirtualized drivers for Windows Server, and Novell will have them when it ships SLES 10 SP1 in July. According to XenSource user Jim Klein (who doesn't cite his sources), Red Hat won't have these drivers until RHEL 5.2 is released sometime next year. Other bloggers have speculated that the proprietary nature of Microsoft's APIs will make it impossible for Red Hat to use them unless it signs some kind of deal with Redmond. However that may be, there is still no solid benchmark data to demonstrate that these drivers can actually move the Xen performance needle in a big way compared to VMware. But certainly Xen fans can always hope.

To be fair, the Xen story is really only just getting started. Given the incredible financial results that VMware has been racking up (on the eve of its IPO, VMware's revenue run rate is well over one billion dollars per year), there can be little doubt that Red Hat, Novell and XenSource have every incentive to make Xen competitive with ESX. But it remains an open question whether they will be able to do so soon enough to matter. It took VMware five years to reach 1,500 paying customers. Xen doesn't have that much time, though, because soon it will have to square off against a new competitor, Microsoft's Viridian hypervisor, which is scheduled to be incorporated into Windows Server 2008 within a few months of the new OS's planned release at the end of this year.

Don't confuse Viridian - aka Windows Server virtualization - with Microsoft's current Windows-hosted (non-bare-metal) offering called Virtual Server. Viridian is a brand new and very ambitious project that has a long list of features (albeit recently pared back) and is explicitly based on the same paravirtualization approach as Xen. Considering that Viridian will be coming to market an amazing full ten years after the first VMware bare metal hypervisor, it's not surprising that Redmond chose the more modern approach. There is already a lot of technical information about Viridian floating around on the web - Microsoft has been using its Windows Server Division Weblog to propagate the latest news. But since only a handful of early beta testers have actually seen it (the public beta has been delayed until later this year), it's pointless to speculate on how Viridian's performance or ease of management will shape up compared to Xen and VMware. If history is any guide, Microsoft will come out of the chute with a solid but not great product, and then deploy its unrivalled block-and-tackle ground game to gradually narrow the gap with the market leader. The examples of Windows Server 2003, SQL Server 2005 and Exchange Server 2007 are testimony to just how far they can get with this strategy.

Between Microsoft, Novell, Red Hat, XenSource, VMware and the scarcity of good benchmark data, it's going to be tricky for users to figure out which operating systems will deliver the best virtualization performance on whose version of which hypervisor. Microsoft has hired XenSource to develop a "shim" (API adapter) that will allow paravirtualized Linux to run on Viridian, and it has licensed its HyperCall API to Novell to allow that firm to develop a similar shim for Windows Server 2008 running on Xen. It isn't clear at this point how Red Hat will work with these shims, or even what its options will be, though it seems unlikely that geeky API issues like these will seriously slow Red Hat down in its march to build a commercial open source franchise covering the full stack from the metal on up.

The confusion about Xen performance and APIs may not matter all that much, because it is increasingly clear that Microsoft's true strategic rival in the virtualization arena is VMware, not Xen. Even if Viridian doesn't blow VMware out of the water - and that's a safe bet - it is very likely to squeeze Xen, and in doing so it will squeeze Linux. A year ago I thought that Xen offered Red Hat a huge opportunity to take some market share from Microsoft before the release of Longhorn (Windows Server 2008). Apparently Red Hat thought so too, because they planned and promoted the release of the all-important RHEL5 around Xen, even delaying the launch date by several months to smooth out the snags encountered in the integration effort.

But today it's apparent that however Xen evolves in the future, it isn't going to be the Longhorn killer Red Hat thought it would be.

I've been a Linux fan for years, but lately I wonder if the drum beating from the big IT vendors in favor of open source hasn't finally slipped over the edge from sincere enthusiasm to meaningless - or in some cases downright hypocritical - sloganeering. The example that brought this gloomy thought to mind was a recent IBM press release touting a "new open client solution" as an "alternative to vendor lock-in". Wow. Imagine that. An alternative to vendor lock-in.

Upon examination, this press release turns out to be unintentionally hilarious.

"Armonk, NY - 12 Feb 2007: IBM (NYSE: IBM) today unveiled a new Open Client Solution for customers of any size or industry so they can help their employees better collaborate, improve productivity, and lower the total cost of information technology ownership... Customers can benefit from the opportunity to make one investment in the single, flexible Open Client Solution, a more efficient alternative to vendor lock-in because only minor changes are typically required to run on different operating system platforms. The solution can include capabilities for desktop management support and application migration and are aimed at helping customers pilot, implement, and gain value from security-rich and reliable Linux and other open standards-based solutions. Operating system services will be provided by Linux distributors Red Hat and Novell."

Just what does this "open client solution" consist of? Why, haven't you guessed? It's Lotus Notes. Yes, folks, IBM in its bold combat against vendor lock-in has decided the time has come to let you choose whether you want to run Notes on Windows, Red Hat Linux, Suse Linux, or even a Macintosh.

I'm not making this up, they're serious. Just in case you didn't get the message, IBM even inserts the following helpful sub-title into its press release:

"Alternative to Vendor Lock-In Includes Lotus Software Running With Linux or Windows and Macintosh"

The grammar in that heading is a little strange (do they really mean "Linux or Windows and Macintosh"?). But never mind. Did I mention that your choice of collaboration software in this "open solution" also includes IBM WebSphere Portal? Hmm, I wonder what happened to Microsoft Outlook, or Microsoft SharePoint, or BEA WebLogic Portal, or the Oracle Collaboration Suite, or SAP Netweaver with Duet? I guess fighting vendor lock-in sometimes requires a little bit of vendor lock-out too.

I don't know about you, but my guess is that IBM's promise to fight vendor lock-in is about as trustworthy as Nabisco's promise to take the trans fat out of Oreos. Either they're not really going to do it, or they're going to put something else in that's just as bad.

What's odd about the open source debate these days is that it's no longer really about "free and open source" vs. "commercial and closed source". In the hands of IBM and other software-freedom-touting vendors like Oracle, open source has become a cudgel used to beat up on other commercial software vendors. IBM bashes Microsoft because Windows isn't open source. But IBM sells more than $15 billion worth of closed source software per year, and is widely known to have spent billions building up the Linux ecosystem to challenge Microsoft. Similarly, IBM bashed Sun for years for not open sourcing Java, but when Sun finally took the plunge last year and put its Java virtual machine and the JDK under the same Gnu Public License (GPL) as Linux, IBM promptly complained that Sun should have used the Apache license instead. Now the Apache license, although nominally open source, has a very convenient feature for commercial software vendors in that it allows them to privatize a piece of software and incorporate it into their own proprietary products, without publishing modifications back to the community as the GPL requires. So it would seem that Sun has made Java too open source for IBM's taste.

IBM isn't the only offender in this game of marketing smoke and mirrors. Oracle charges astronomical fees for its high-end database, but it bashes Red Hat for charging a fraction of those fees for Red Hat Enterprise Linux, and then brazenly proceeds to resell a binary clone of Red Hat's product. Let's do the math here. Suppose you have a fairly big transactional application and you want to run the enterprise edition of Oracle's 10g database in a clustered configuration on four servers, each equipped with a single Intel quad-core processor. Under Oracle's licensing scheme, you'll need to buy two licenses per server (since each Intel core counts as half a processor), or eight in all. With 10g EE at $40,000 per license and Real Application Clusters at $20,000, that will be 8 x $60,000 or $480,000 in one-time license fees, plus an additional $105,600 per year for support and updates. But if as a thrifty buyer you are willing to throw out Red Hat and bring in Oracle instead as your supplier of Red Hat Linux, you will save an additional $500 per server per year, or $2,000 in all - that is, less than 2% of your recurring annual Oracle maintenance contract and less than 0.5% of your Oracle license fee. With deals like this, it's little wonder that most Red Hat shops have decided they have more important things to do than replace an operating system that's working perfectly well with an identical binary clone from Oracle (remember, most Linux shops that don't want to pay Red Hat prices are already using Fedora or a free RHEL clone like CentOS).

The only reason IBM and Oracle get away with this kind of posturing is that the magical words "open source" have come to function as the software equivalent of carbon offsets. In the dreary conventional view of economics shared by Richard Stallman's side of the open source world and some of his pundit fans, profits are akin to global warming: both are evil. By this logic, common corporate strategies for achieving profits - such as building a better mousetrap, making your brand a household name, or trying to keep the recipe to your secret sauce a secret - are inherently wrong. You see, it's just plain wicked for software developers to go out and create really useful solutions to really hard problems, and then have the nerve to keep them secret so they can charge people money.

But some software vendors are cleverer than others, and have learned to buy indulgences for their sinful profit-craving ways by selectively building open source components into their stack. They do this at the lowest possible layers, preferably those that are costly to develop but unlikely to yield a competitive advantage. For example, those pesky server operating systems that are indispensable to the performance and stability of important applications, yet demand years of thankless system programming to do well.

This is all that IBM and Oracle are doing with their open source marketing. They use the worldwide community of Linux contributors as a low-cost R&D operation and they exploit distributors like Red Hat as a communally owned distribution channel - ideally one that shouldn't get too uppity about demanding the right to make its own profits. Their own software remains every bit as proprietary as the Microsoft products they compete with. And if open source projects begin to climb up the stack and threaten IBM WebSphere or Oracle Fusion middleware - watch out! An enraged software elephant can do a lot of damage just by stomping its feet, even if there is no business sense to its actions, as demonstrated by Oracle's venture into the Linux cloning business in order to punish Red Hat's acquisition of JBoss. I shudder to think what will happen if Red Hat ever acquires MySQL.

IBM and Oracle are not the only technology vendors who have leveraged commodity open source plumbing into fabulous profits at the proprietary application layer. Google is another great example, perhaps one that will prove much more influential in the long run. The Mountain View search firm has built what amounts to the world's largest dedicated compute engine using its own customized dialect of Linux and a stupendous quantity of dirt-cheap PC-caliber motherboards and disk drives. By some estimates the firm now runs over half a million of these homegrown servers, which as nearly pure commodities in the economic sense are the hardware equivalents of Linux at the operating system level.

But in one important respect Google is playing straighter than IBM or Oracle. Unlike them, the search giant doesn't try to gloss over its profit-making ways by flaunting its commitment to open source, despite the fact that it is by far the largest user of Linux on the planet. On the contrary, Google makes no bones about the fact that its search algorithms and its data center architecture are proprietary, and that open source is no more than an accidental ingredient in its ultra-secret, money-making sauce. We would all be better off if IBM and Oracle would get off their high horses and do the same.

The server operating system wars never seem to slow down. Last week it was Red Hat's turn with the announcement of Red Hat Enterprise Linux 5 (REHL5), which incorporates the Xen open source hypervisor. Naturally there's also the endless market speculation about the final feature set and likely arrival date of Windows Server 2007 (aka Longhorn). And then there's Solaris, which with its nice value-add features like DTrace and its new status as open source software is making something of a comeback, it seems.

In this industry we all spend a lot of time following the constant tit-for-tat among the vendors. Frankly it can be a lot more exciting than some other spectator sports I know of, like politics. But every time I start to dig into all the white papers and benchmarks and analyst reports to try to figure out which of these fine offerings has finally managed to get a leg up on the competition thanks to its latest bell or whistle, my head starts to spin. Is it really possible to keep up with all this stuff? Is someone ever going to win the operating system wars and put an end to our confusion? Or are we just on an endless treadmill of tweaks and improvements and competitive breakthroughs that will never settle down into a steady state? And so doubt creeps over me, and I begin to wonder if we haven't all fallen into the software industry equivalent of Groundhog Day.

Then a light goes on, I slap myself on the forehead, and I say to myself, "It's not about the operating system, it's the stack, stupid!".

In other words, people don't really sit around wondering which operating system they should use. Instead, they decide which applications they want to run, and the OS usually picks itself. At least, that's the thought that came to me while reading the interview with Rich Green that Sun posted on their web site last month. Now Green, who returned to Sun last year as Exec VP for Software, is one of the smartest people in software, so he's always worth listening to (full disclosure: I did a long Q&A with him a while back when he was still an EVP at Bill Coleman's datacenter automation startup Cassatt).

It will come as no surprise that Green's pitch is all about how Linux users should come back to Solaris. He has a lot of good points to make, but his basic argument is that Solaris is a lot better than it used to be at supporting the web tier applications that have been the driving force behind the rise of Linux in the datacenter.

"Solaris has always been superb on the back end. But to be honest with you, until a couple of years ago, the Solaris Operating System did not fulfill all the requirements of the edge and Web tiers. Today, however, Solaris has been optimized for the one-, two-, and four-way microprocessors used in AMD and Intel chipsets for many of the same servers that run Linux today. This means that the operating system provides an entire solution for both the database and presentation tiers."

I'm only half convinced by this. I mean, I don't doubt that one of the new Sun servers packed with dual or (soon) quad core chips from Intel or AMD - or even one of Sun's own eight core Niagara chips - makes a nifty replacement for the rack of Dell Xeon boxes at work in a typical web tier. But I'm not so sure that masses of Apache, PHP and MySQL users who are today happily chugging along on Linux are going to start lining up at Sun's door any time soon. In fact, Red Hat seems to have seen this threat coming. Their just announced Red Hat Exchange will be an online store for commercial open source software. Although plenty of people will continue to run their open source stacks on free versions of Linux rather than RHEL, the coming of RHX is bound to have a big impact on the Linux commercial ecosystem.

As for the Oracle users who were thinking of moving from Solaris on Sparc to Linux on Intel-AMD (can we just call it "Amtel"?), well that's another story. There's a lot of anecdotal evidence that Oracle on Solaris continues to outperform Oracle on Linux in high-volume transaction processing applications. This has been one of the historical sweet spots for the vendor Unix systems. And it's probably no accident. It's widely known in the industry that database code has to be carefully optimized to a specific OS and multiprocessor server platform if you want to maximize performance. Most modern operating systems contain special APIs to make this optimization possible. Over the years Sun and HP engineers have spent a lot of time in Oracle's lab carefully tweaking the Oracle database kernel to make it scale on their big SMP Unix boxes. But as far as I know, no one is doing this work for Linux. (Want a geeky example? An industry source recently confided to me that when the server hardware vendors run their Oracle validation test suites on Linux they don't even bother to turn on the non-uniform memory [NUMA] API in the Linux kernel.)

But to really take the measure of Green's claim that Solaris is winning back lost ground to Linux, we need to look at their respective installed bases. According to IDC, there were about 1.5 million active Solaris servers installed around the world at the end of 2005. According to Sun's web site the vendor shipped about 350,000 servers last year, some of which must have been replacement machines rather than new installs. So lets say the Solaris installed base has grown to 1.75 million servers by now.

How big is the Linux installed base in comparison? I haven't seen any published IDC or Gartner numbers on this, but there is a fascinating slide on Microsoft's web site that breaks down the worldwide server installed base into 13 segments and gives an estimate of Linux's share in each one. According to Microsoft, there were 24 million servers in use in 2005, and 15% were running Linux. If we put the server installed base at 27 million today, and then bump up the Linux share to 17%, we get a total Linux server installed base of around 4.6 million machines, or roughly three times the size of Solaris. However, one could object that this is an apples-to-oranges comparison in the sense that most Solaris is in the datacenter while a lot of Linux is out on the perimeter. IBM has recently estimated the number of Linux servers actually installed in datacenter environments at 1.6 million worldwide, which puts Linux slightly behind Solaris.

Conclusion: Solaris is holding its own in the datacenter applications like database, but is still far behind Linux as a platform for the broader open source software stack exemplified by apps like Apache, PHP, Tomcat, or Sendmail. If Solaris is going to live up to Rich Green's claims for it, it will have to break out of its traditional high-end comfort zone. But given the vast proliferation of free Linuxes out there, this is quite frankly going to be hard to do, at least as a revenue generating proposition for Sun.

Free Linux is one thing, but what about commercial Linux? The continued strength of Solaris in the datacenter makes it a significant impediment to Red Hat's effort to migrate up the stack from the barebones operating system layer to the rich application layers above it. These layers include things like virtualization, clustering, security and management. You see, Red Hat is no longer really synonymous with everyman's Linux. Rather, it has evolved into a kind of premium "rich man's" variant of Linux whose true target is a relatively small number of large and wealthy IT organizations. Think Wall Street bulge bracket brokerage firms, major banks, tier one telcos, and big Federal agencies. These are the customers who are going to pay $2,499 per server per year for 24x7 support on RHEL5 Advanced Platform, or even more for Red Hat's new Datacenter Solutions offering.

But the problem for Red Hat is that to win over the premium segment it will have to go head-to-head with Solaris (and, to a lesser extent, with IBM's AIX and HP's HP-UX). Sun has at long last figured out that it was pricing itself out of the market, in effect creating a pricing umbrella that allowed interlopers like Red Hat and Microsoft into the datacenter game. Now Sun has slashed its software support prices and revamped its hardware architectures. It still isn't cheap, not if you buy the whole Sun stack and the support that goes with it, which is why I suspect the company will never catch up with Linux in the broader server market. But it is in a much stronger position to hold its own in the premium market segments that really count for its bottom line. And that spells trouble for Red Hat.

While Solaris and RHEL duke it out for control of the premium "operating systems derived from Unix" segment, both players are overlooking the threat from Microsoft. According to IDC, Windows Server commands about two thirds of total server unit shipments and drives almost three times the dollar value in server system sales as Linux ($5.3 billion vs. $1.8 billion in Q5 2006). It's pretty hard to find a datacenter these days that doesn't have some Windows Server boxes lurking somewhere..

Although the open source world scarcely deigns to notice it, with SQL Server and Exchange in particular Microsoft has assembled two of the most widely used enterprise applications in the market. The power of these applications to attract and retain customers for the Microsoft stack is far more important than most Linux fans recognize. It's hard to get exact numbers, but some insiders put the total SQL Server installed base at more than one million. Compare that to Microsoft's estimate (on the same slide cited above) that the total world installed base for database servers is three million machines, or Oracle's statement in its marketing materials that it has around 250,000 customers. Of course Oracle's database business is larger in dollar terms than Microsoft's, because its database is a lot more expensive. But SQL Server 2005 is growing sales at over 30% more than a year after its release, while Oracle's database revenue in its latest quarter was close to flat.

The danger that commercial Linux faces today is that its leading proponent, Red Hat, has forgotten to pursue a volume strategy and has painted itself into a premium-priced corner that it is busy fighting over with Sun. Fedora is the volume strategy you say? Well, yes, Fedora is free-as-in-beer, which makes it a lot cheaper than RHEL. But that doesn't really address the problem. You see, it really is all about the stack. Linux still owns the web tier, and it doesn't look like anyone can take that away in the foreseeable future. But MySQL and Postgres aren't viable replacements for Oracle or SQL Server, and PHP and Ruby aren't viable replacements for enterprise Java or .Net. And nobody seems to be doing the work needed to make Linux scale on the new multiprocessor scale-up server hardware that is coming back into vogue. It really is beginning to look like Linux is stalling in the datacenter while Solaris holds its own and Microsoft races ahead.

Perhaps its time that the Linux community stopped obsessing about the arcana of intellectual property rules and started worrying more about what customers are actually buying. Remember, it's the stack, stupid!

Is virtualization destroying the server hardware business? That's what the Wall Street Journal would have us believe. In a story published March 6 on its subscription only site, the Journal profiles a series of users who have slashed their equipment count and cancelled new purchases thanks to server consolidation enabled by virtualization. Of course the reporter doesn't quite have the audacity to argue his case in his own words. But he marshals an impressive range of stand-ins for his thesis in the form of Gartner and IDC analysts:

"There was a surprising slowdown in unit sales of x86 servers in the fourth quarter, according to two prominent research firms, Gartner Inc. and IDC. For example, IDC found that sales grew just 1.1% to 1.85 million units, compared with 8.8% growth in the third quarter. That is the worst performance for the market segment since the dot-com bust, and IDC analyst Matt Eastwood lays the blame largely on virtualization... 'It's a fact,' agrees Thomas Bittman, a Gartner analyst. Though companies will still buy servers, virtualization 'is absolutely going to have a slowing effect' on server sales."

Now as a matter of fact, the "virtualization is killing server sales" meme is far from new. Tech blogger (and noted IT pessimist) Nick Carr made the case for this view quite eloquently over a year ago. And I've been getting questions about it from worried Wall Street fund managers for about as long. Their concern is not limited to server chip vendors like Intel and AMD or the server system vendors like IBM, HP, Sun and Dell. It also extends to server operating system vendors, particularly Microsoft and Red Hat.

But how much truth is there to the notion that virtualization is causing server sales to shrink? We can start by recognizing that plenty of companies are indeed using software like EMC's hot selling VMware to prune server sprawl. The examples supplied by the Wall Street Journal are revealing. Here is one:

"David Siles, chief technology officer for Kane County in Illinois, in the fall of 2004 was managing roughly 300 servers scattered across 45 departments -- too many for his three-person staff to handle. Things got so bad, he says, that people were running servers in broom closets and under desks.... When it came time to retire those machines, Mr. Siles used VMware's technology to consolidate the county's computing operations on just 35 new physical servers. He estimates the move will save $30,000 to $40,000 a year in electricity bills. The new machines are running at roughly 65% of their capacity, versus no more than 10% before the shift.... 'We hardly ever buy a new server,' Mr. Siles says. 'The last server I bought was almost a fiscal year ago.'"

There's not much doubt that scenes like this are being repeated all over the country these days. So yes, virtualization is eliminating some server sales in certain segments of the market. This is because it permits a natural and healthy rationalization of the pockets of inefficiency that have grown up in some large and midsize data centers.

This hardware pruning in some ways recalls the great software purge of the earlier part of this decade. During the dot com boom and the run-up to Y2K, many companies loaded up on ERP and other enterprise software licenses. They bought far more user "seats" than they needed at huge discounts to the vendors' inflated list prices, on the theory that they would eventually find a use for these licenses. Then, after the crash, they discovered that "eventually" was going to be much farther in the future than they thought. Since they could pull licenses off the shelf whenever they needed to add more users, the companies abruptly stopped buying new software. The result was the great software recession that lasted roughly from 2001 through 2003. An industrious Morgan Stanley software analyst famously coined the term "shelfware" to denote this demand-killing overhang. As an interesting footnote to history, that analyst has since moved up the food chain to become Co-President of Oracle. His name? Charles Phillips.

But there is a fundamental difference between the shelfware phenomenon and the impact of virtualization on server sales. Software vendors opportunistically sold truckloads of licenses at a discount because doing so pumped up their short-term earnings, not because enterprise software was getting intrinsically cheaper. But servers are getting intrinsically cheaper. Indeed, the utilization gains that virtualization brings are only possible because the throughput of the typical microprocessor has increased dramatically in recent years. You couldn't consolidate servers with VMware if you didn't have a lot of servers sitting around that could run a serious enterprise application like Microsoft Exchange or MySAP ERP and still have plenty of cycles left over.

It must be said that the Journal's anecdotes are a little one-sided. Although the reporter doesn't say so, his customer references were most likely provided by the software vendors who are mentioned in the article. Now it is a perfectly legitimate journalistic practice to resort to this kind of legalized insider information. I confess to having done so myself. After all, no one should be in a better position than the vendor itself to know what its customers are doing (or else the vendor is really in trouble!).

But whatever the source of your data, you need to retain a basic level of skepticism and a concern for empirical truth. In this case, the reporter has conspicuously failed to look for facts that contradict his thesis. A glance at the IDC and Gartner press releases published in February that apparently provided the impetus for the article turns up some interesting items not mentioned by the Journal. The first such item is that growth in server shipments in 2006, although slower than in recent years, wasn't all that bad in absolute terms. Gartner says server unit shipments grew nearly 9% to 8.2 million units in the year, while IDC puts the growth at 5.9% for 7.5 million units. If you believe the Gartner number, then worldwide server shipments grew last year at more than double the rate of the world economy as a whole. Not exactly a gloom and doom story.

True, revenue didn't grow as much as shipments. The server industry is locked in a ferocious price war these days, whose underlying cause is the unrelenting improvement in processor chip "bang-for-the-buck", exacerbated lately by Intel's desire to teach AMD a lesson or two about the effects of volume on pricing. Both IDC and Gartner agree that server revenue grew only a modest 2% in 2006, but their estimates of the amount of that revenue are slightly different. Gartner says the server vendors reaped $52.675 billion in sales, IDC says it was only $52.285 billion. Thus the two revenue estimates are within one percent of each other, while the unit shipment estimates differ by more than 9%. This isn't surprising. When you are dealing with public companies like the big server vendors who by law must report their revenue down to the last penny, it is easier to count dollars than boxes.

The difference in reported growth rates is likely an artifact of some underlying difference in the two analyst firms' estimates of the quantity of servers shipped. Most people don't realize that both IDC and Gartner rely on the server hardware vendors themselves as the primary source of their data. The numbers they publish are approximations that attempt to triangulate among the competing vendors' self-serving claims about their sales and the analysts' own knowledge of the market. One problem in particular that has plagued the market estimates is the growing popularity of non-standard servers such as repurposed desktops and machines assembled from raw components by users or integrators. Since it is impossible to quantify the precise extent of these practices, the analysts are obliged to plug their best guesses into their spreadsheets. It's a good bet that Gartner and IDC don't see eye to eye about the size of the non-standard segment of the server market.

In short, the quantitative foundation on which the "virtualization is killing server sales" meme rests is somewhat shaky, because it overlooks the fact that we don't have a really precise, unbiased measure of the market's true size or rate of growth. The best estimates of server sales in any given year may be off by as much as 5% or 10% (this is a truism, given that even before factoring in a margin of error Gartner and IDC disagree by nearly 10% on shipments).

But there are more interesting items in the IDC and Gartner data that the Journal conveniently neglects to mention. For example, in 2006 the x86-x64 market saw a significant shift from cheap boxes made by HP and Dell to rather more expensive boxes made by Sun. For the full year Sun saw its overall server revenue surge 15.4% according to IDC, while HP and Dell sales were essentially flat at 0.2% and 2.2% respectively. I'm willing to bet that the Sun boxes, mostly Opteron-based, had a higher average chip count than the HP and Dell boxes. IDC doesn't break out the proportion of Sun's surge that was due specifically to its x64 servers, but Sun's own web site indicates that the company sold roughly 100,000 such machines in 2006, a dramatic increase from the year before.

Could it be that with the rise of multicore chips and "commodity" multiprocessor servers the number of individual boxes shipped is no longer a good proxy for the amount of customer demand for server capacity? If so, then the virtualization meme would take another hit. It turns out that the IDC analysts endorse this thesis in their press release. They actually argue that virtualization requires more powerful servers. And if you think about the requirements for additional RAM and input/output bandwidth imposed by running multiple virtual machines on a single server, you can see that this is obviously true.

In fact, if you drill down into IDC's Q4 data you can find some powerful confirmation of the trend toward bigger and more powerful servers in the so-called "commodity" segment of the market. Both Linux and Windows Server saw hardware revenues grow faster in the quarter than unit shipments, which means that average selling prices must have increased. After several quarters of slow growth Linux server revenue jumped 15.3% to $1.8 billion, but the number of Linux boxes sold actually declined by a fraction of a percent (it's possible that IDC has somewhat inflated Linux sales by including some mainframe revenue, but even so the pattern is clear). Exhibiting a similar trend, Windows Server hardware revenue grew 9.4% to $5.3 billion, while units shipped grew only 5.1%. (Incidentally, Windows now drives 40% of all non-mainframe server revenue, surpassing Unix for the first time.)

It's striking to note that the behavior of the Windows and Linux segment of the market - precisely the segment most affected by the current virtualization craze - exactly contradicts the Wall Street Journal's hypothesis. If VMware and its ilk were really strangling server sales, you would expect this segment to have been hit the hardest, but it's actually growing faster than the market as a whole.

But the arguments against the virtualization meme I have listed so far are really just nitpicking. Where the meme breaks down fundamentally is in its complete and abject failure to take account of the elasticity of demand. Thanks to Moore's law microprocessors have been offering more bang for the buck every year for as long as anyone can remember. The more efficient utilization of a given amount of processor capacity that virtualization makes possible is just another way of achieving the same result, except that it relies on clever software instead of better manufacturing processes. In other words, for a variety of deep technological reasons, the commodity that we call server capacity has been getting cheaper and cheaper. Now economists have been teaching us for several centuries that so long as demand for a commodity is unsatiated, a decline in price will result in an increase in sales. Is it really so hard for our grim-faced forecasters of IT doom to grasp that this hoary law of economics just possibly applies to demand for servers?

I mean, seriously, do you think the number of cell phone calls that people want to make is going up or down? Do they want to send more or fewer instant messages? What about the number of web pages they want to see, or the number of podcasts and video streams they want to download? The number of web searches and web purchases they want to execute? The number of retail credit card transactions? The number of RFID scans performed in the average working day? Oh, and if you agree that we Americans are going to want to do all of these things more and more frequently in the future, ask yourself this: is the percentage of the rest of the world's population that is willing and able to do these things on a daily (or even hourly) basis going to get larger or smaller over the next ten years?

There is no better illustration of the impact Moore's law on server demand than the rise of the monster search engines such as Google, Yahoo or Microsoft Live. In the past five years, our addiction to these rather self-effacing utilities has fueled the creation of the largest server complexes the world has ever known. Estimates published last year in the New York Times and Baseline Magazine put Google's total server installed base at 450,000 units vs. 200,000 for Microsoft.

But these estimates were probably already out of date when they were published. We know by Google's own admission that the search engine's server count in 2000 was only about 60,000. My guess is that Google's server count today is well over half a million. In other words, it's roughly ten times larger than it was only seven years ago. Considering the likely short life span of these servers and the likelihood that Google's competitors buy servers in proportion to their share of the search market, I'd go even further and estimate that Google, Microsoft and Yahoo together will probably account for 8% to 10% of all x86-x64 family server sales in the world this year. Compare that with the fact that the share of servers sold this year that will have VMware installed on them is probably well under 2%.

So pardon my optimism, but I just don't buy the idea that virtualization is killing the server hardware business. It's just another bump on the accelerating demand curve for server throughput.

The headline of this post is tongue-in-cheek (but of course you knew that already, right?). Reuters, to the best of my knowledge, has not been banned from reporting. Although based on some of their recent work, they should be.

Seriously, you have to wonder what they have been smoking over there, after reading a headline like this:

"Novell could be banned from selling Linux--group"

The article in question was published by Reuters on February 2, and the "group" referred to is the Free Software Foundation. Here is the opening sentence:

"The Free Software Foundation is reviewing Novell Inc.'s right to sell new versions of Linux operating system software after the open-source community criticized Novell for teaming up with Microsoft Corp."

The author of the article would have us believe that there is a serious possibility Novell could be forced to withdraw from the Linux market as punishment for its highly unpopular deal with Microsoft. And furthermore, this amazing turn of events could occur as early as next month if the all-powerful FSF so decrees. Wow! The reporter goes on to imply that even though Linux produced only 5% of Novell's revenue last year, such a change in plans could hurt Novell's stock price. I'll bet it could. I'll also bet it could make Steve Ballmer feel more than a little strange about the $240 million he just forked over to Novell for the right to distribute 350,000 copies of Suse Linux over the next five years.

The Reuters headline is of course is completely and utterly wrong. Novell is not going to be forced to stop selling Linux. Well, never say never. Maybe aliens will descend on Waltham Mass., kidnap the company's staff and fly them away to another planet. But barring that, Novell is going to be a Linux player for many years to come. If not, I'll eat my hat. The Reuters article doesn't merely brush up against the line of journalistic malpractice. No, it jumps right over the line and dives headfirst into a steaming pit of abject nonsense.

But how on earth, you may well ask, did the reporter ever manage to fall into such a delusion in the first place? Well, that is a tale of a confusion wrapped around a nugget of truth piled on top of a misapprehension.

First, the confusion. If you take the headline at face value (and it would be odd if Reuters intended its headlines to be taken figuratively), it appears that the reporter has confused the GPL license under which Linux is licensed with Linux itself. Instead of saying that Novell could be "banned from selling Linux", what he should have said is:

"Novell's recent patent arrangement with Microsoft may make it difficult for the company to adopt the new version of the GPL that has been proposed by the FSF (but which has already been rejected by key members of the open source community and which the FSF has no power to impose unilaterally)"

Not quite as catchy, but miles closer to the truth. What the reporter has done here is akin to confusing your driver's license with your car. One fits in your wallet, the other goes in your garage. The Free Software Foundation has the self-appointed duty of producing the next version of the GPL, true enough. And by controlling the content of that version it can obviously influence which companies and software projects will want to adopt it.

But the FSF simply has no power to ban anyone from selling Linux itself, because it does not own the intellectual property rights to Linux. The copyrights to the Linux kernel source code are owned by the countless programmers who have contributed to it over the years, including but certainly not limited to Linus Torvalds. In fact, not even these copyright holders can ban someone from selling Linux. Because they have chosen to license their work under the 16 year-old GNU General Public License version 2 (GPLv2), the most they can do is sue someone who violates the terms of that license. But they cannot change the license itself, no more than its original author can, FSF's own Richard Stallman.

Now for the nugget of truth. The FSF doesn't like the Novell-Microsoft deal, and would love to kill it if they could. But of course they know they can't, because as Stallman and FSF's Eben Moglen have acknowledged, the indirect "I promise not to sue your customers" patent protection that Microsoft has granted Novell, ugly as it may be, doesn't violate the terms of GPLv2. That's why Stallman and Moglen have hinted they will make last minute changes to the controversial new version 3 of the GPL they have been struggling to promote for over a year. So Reuters is perfectly correct to suggest that software distributed under the future GPLv3 most likely will not allow arrangements such as the one Novell and Microsoft have made.

But here we come to the misapprehension or - to put it more bluntly - the most abjectly wrong part of the Reuters story. What the FSF thinks of the Novell-Microsoft deal, and whatever changes they may or may not make to the GPLv3, are utterly irrelevant to Novell's legal right to sell Linux, now or in the future. The reason is that Linux is not going to migrate from GPLv2 to GPLv3. A group of key kernel contributors, led by Linus himself, have declared that they are irrevocably opposed to the GPLv3 and have no intention of allowing their code to shift to the new license. Since they as the copyright holders are the only ones who have any say in the matter, that's the end of it. The GPLv3 is dead on arrival, and everyone knows it. (For a discussion of why the kernel contributors rejected the GPLv3, see "Is GPLv3 just a load of 'crap, crap, crap'?" and "Can Stallman force a Linux fork?".)

There you have it. One of the leading news-gathering organizations of the world has demonstrated a mind-boggling inability to get its basic facts straight (apparently not for the first time). If the Reuters reporter had been the least bit interested in the truth, he might have first spent 5 or 10 minutes rooting around on Google, which would have been sufficient to uncover the fact that there is no story here.

Instead, he went for the sensational headline, and ended up inventing a story from whole cloth. He mistook the GPL for the software, and then he got the wrong GPL. That isn't just confusing your driver's license with your car. That's confusing your license with my car. Reuters, out of my garage, please!