Microsoft is trying to get under IBM's skin with some benchmarks run in its Redmond labs using Big Blue's own Java-based test, Trade, and a variant of it ported to C#, which Microsoft calls .NET StockTrader. But as Microsoft throws down the benchmarking gauntlet, IBM is ignoring the calls for a WebSphere duel at the Middleware Corral.
The gauntlet was thrown down by Steven Martin, senior director of developer platform marketing at Microsoft, who launched a site called WebSphereLovesWindows to show off the results from some intricate benchmark tests that the nerds at Microsoft's labs have done to show how WebSphere on AIX iron stacks up to WebSphere running on Windows or the same application ported to C# and not using WebSphere at all, but the Windows stack.
(This site requires you to install the Microsoft equivalent to Flash, Silverlight, and I have managed thus far to avoid installing it. Because I'll give you a direct link to the PDF, which is here. Now you don't have to install Silverlight, either).
Martin is also blogging about the benchmark tests, which have been a pet project of Greg Leake, a techie in Microsoft's Connected Systems Division lab.
Let's back up a bit and start from the beginning before getting into Microsoft's claims about WebSphere performance. IBM created the Trade benchmark to do some internal testing on its servers, and as the name suggests, the Trade test simulates the data processing operations of a stock brokerage (just like the TPC-E test does, but they are not the same code). Trade was formerly known as the WebSphere Performance Benchmark, and the code behind it is implemented in Java and runs on Java-based application servers in a two-tier or three-tier environment.
For the past several years, IBM has been using Trade6 implementation of the test internally to benchmark the performance of its i5/OS, AIX, and Linux systems. A number of other vendors have taken the code (which IBM opened up) and used it to run tests, including Java appliance maker Azul Systems and Microsoft, which ported the code to C# and called it .NET StockTrader. This WebSphere-versus-Windows battle is not new. It has been going on for years, and Microsoft has compared WebSphere on Linux to the Windows stack using Trade on one side and StockTrader on the other.
What is new is that Microsoft is creating a variant of the Trade benchmark to support WebSphere 7, IBM's latest iteration of its Web application server, something that Big Blue has not, as far as anyone knows, done itself. (Leake says that as far as he knows, Big Blue has no intention of do so). The other trick this time around is that Microsoft is running its WebSphere 7-compatible version of Trade on IBM's own AIX operating system and Power Systems iron. It's not only comparing that against the C# StockTrader app running on Windows Server 2008, but also Trade running atop WebSphere 7 on Windows.
And to make its case a little stronger, Microsoft is also releasing the Trade and StockTrader application set (you can get the code here), which allows end users to run their own tests not only against WebSphere and Windows middleware, but also on any Java-compatible Web application server. (Leake says that Microsoft has tested its implementation of the Trade benchmark on systems running Oracle's WebLogic and Application Server middleware, but Oracle's license agreements prevent him from discussing this). The download also includes capacity planning tools to help system admins make use of the Trade and StockTrader results to compare and contrast with their own applications and other systems.
So far, IBM's manhood has not been insulted enough by the publication of WebSphere benchmarks by Microsoft to allow itself to be called out.
"We were amused when we read the Microsoft disclaimer that clearly stated 'Microsoft cannot guarantee the accuracy of any information presented' in the report that forms the basis of these claims," explained Ron Favali, an IBM spokesperson, in an email exchange. IBM's WebSphere executives were not interested in talking about this, and when asked to present some Trade benchmark tests that might refute what Microsoft's claims, I got the brush off.
IBM's position is curious, considering that Microsoft's comparisons were not exactly apples-to-apples and therefore subject to criticism. For the Web application portion of the test, Microsoft chose a Power 570 server with eight 4.2 GHz Power6 cores and put 32 GB of main memory on it. This machine was carved up into two WebSphere images, which hooked into two different Trade databases (which actually ran on four-socket BL680c Xeon blade servers from Hewlett-Packard), which it did through four Gigabit Ethernet links.
This IBM iron was compared to an HP BladeSystem equipped with four two-socket BL460c blades. But oddly enough, each blade only had one four-core, 3 GHz Xeon E5450 processor (instead of two) and had 32 GB of memory. With four blades, you have 16 cores and 128 GB of main memory, and with each blade having two Gigabit Ethernet NICs, for a total of eight NICs, it doesn't surprise me at all that four blades with a total of four times the memory did more work on the Trades test, as Microsoft's benchmarks show that it did.
To be fair, that IBM system did have 16 threads and so does the HP blade setup. But a virtual CPU is not the same thing as a real one when it comes to throughput, and Microsoft surely knows that. Also, the IBM machines were configured with AIX 5.3, not the more current AIX 6.1, which is tuned to take advantage of the Power6 generation of iron.
Moreover, IBM has Power6 blades, so a two-socket JS22 blade and a BladeCenter chassis would be a much more fair comparison to the HP blade setup. The Power 570 is an SMP server, and it is designed to scale up to 16 cores in the generation that Microsoft used in the test. And equally importantly, there were faster 4.7 GHz chips available than the 4.2 GHz ones Microsoft used in its test. Last October, IBM goosed the Power 570, doubling up its cores to a maximum of 32 using 4.2 GHz Power6+ chips.
The point is, the Power 570 is in a different weight class than the BL460c. Microsoft's comparison should have pitted the four-socket, 8-core, 16-thread Power 550 rack server, or better still, a Power 560 with twice as many cores (but running at only 3.6 GHz) against a four-socket ProLiant DL580 using Dunnington processors. And with an equivalent amount of memory per core or thread. Or, compare the HP blades against the IBM blades. The SMP boxes are far, far more expensive in an entry configuration than entry servers, which makes the comparison Microsoft did null and void as far as I am concerned.
Leake was not in much of a mood to hear that today in an interview, but that's the truth. But Microsoft is keen on trying to goad IBM into running tests or somehow reacting to Microsoft's benchmark results.
"We stand by our numbers," said Martin. "If IBM will participate and have a third party validate the results of tests, I'll pay."
That sounds like a pretty good deal, and IBM would be wise to step outside into the sun with its guns loaded. Because Leake understands the criticism and will be perfectly happy, I presume, to redo some tests using like-for-like hardware comparisons, perhaps pitting the new Power6+ versions of the Power 550 or Power 560 against the ProLiant DL380 using Intel's "Nehalem" Xeon 5500 processors or HP blades using these chips against the new JS23 blades. (Both of these IBM machines were announced last week.) Getting the core counts and thread counts as close as possible for the hardware should be the goal for any test. When in doubt, make the core counts the same.
It seems moot to go over the numbers in Microsoft's test, but as you might expect, on the StockTrader C# variant of the Trade application, those four HP blades were able to process 12,576 transactions per second (TPS) running atop .NET instead of WebSphere. Shifting the middleware to WebSphere and switching back to the Java version of the code dropped the throughput down to 11,004 TPS. (Leake did not offer any explanations as to why, we are to simply take this as it is).
The HP blade hardware and base systems software supporting the Trade and StockTrader applications cost $50,161, but adding WebSphere boosted the price on the Windows-based blades by another $37,000 using IBM's Processor Value Unit pricing scheme. So, the all-Windows stack looks very good on the price/performance scale.
By comparison, the Power 570 server that Microsoft used in the Trade-on-WebSphere 7 test - and which I contend is definitely memory constrained and possibly underpowered compared to that HP iron with only 8 cores with threads compared to 16 cores with no threads (SMT just isn't the same as having a real core) - cost $215,728 and another $44,400 for WebSphere 7 priced on that PVU pricing scheme. And on the Trade Java-based test, this machine could handle only 8,016 TPS. The Windows-x64 setup obviously stomps it. But again, so what?
Assuming Trade scales more or less with IBM's rPerf (Relative Performance) internal benchmark test, an eight-core Power 550 server using the new 5 GHz Power6+ chips would deliver about 15 per cent more oomph - or about 9,218 TPS on the Trade test. And a 16-core Power 560 running 3.6 GHz Power6+ chips would yield about 47 per cent more oomph - or about 11,757 TPS. That's right smack dab between the two Windows numbers Microsoft is cheering about.
That IBM Power 550 is more expensive than the blades. It costs $118,284 at list price with only 16 GB of memory, and boosting it up to 128 GB of memory would run $512 per MB, which is half what IBM charges on the Power 570, or $57,344. That's still $175,620, and then you have to add WebSphere on top. The 16-core Power 560 with 128 GB of memory is about the same price at $178,679, but as I said above, the Power 560 does nearly 50 per cent more work on applications that like threads. Such as WebSphere and Java.
A JS23 blade using the new 4.2 GHz Power6+ processors, by the way, has about half the performance of the Power 570 that HP tested, and four of these blades would yield performance that is higher than that offered by the HP blades Microsoft tested, probably on the order of 16,000 TPS on the test. And four blades with 32 GB each plus a BladeCenter H chassis and AIX on the blades runs to $98,229 at list price. Even after the HP blades were fully populated with Xeon processors, it might be a pretty tight race, blade for blade, on the Trade test. Until the Nehalem EP blades are tossed into the mix, of course,
Maybe what can be honestly said is that IBM charges twice as much for Power-AIX iron as it and other vendors do for x64 iron (because of the scalability in the Power platform) and IBM charges for WebSphere while Microsoft is giving away its Windows middleware to extend its growing monopoly in the server space. Neither is a surprise. It has been that way for a decade. ®