Two solid state disk SSD suppliers are arguing about NAND flash performance drop-off.
OCZ supplies NAND flash solid state drives (SSDs) and regularly announces high-performance products. DDRdrive has recently exited stealth mode and makes the X1 hybrid DRAM/NAND SSD. It criticises OCZ and other flash suppliers for products exhibiting a dreadful performance decline after a little use fresh out of the box.
Christopher George, founder and chief technology officer of DDRdrive, talks of "dirty tricks used to drastically inflate IOPS results", "the egregious disparity between promised performance and reality", and "the untold truth about OCZ's *sustained* write IOPS".
George gave a presentation at the Open Storage Summit 2010 on flash SSD performance degradation and we have received a copy of the slide deck. The PowerPoint pitch asserts that "Flash SSDs produce one-time, unsustainable, dramatically inflated results when tested 'new' or after a Secure Erase."
Then George wades in on OCZ, claiming: "Both the OCZ Vertex 2 EX and the OCZ Vertex Pro SSD use the SandForce 1500 controller, which implements compression at the drive level. Benchmarking with an Iometer version that defaults to using extremely compressible data will, in these cases, show dramatically inflated write IOPS results."
He used pseudo random I/O transfer data in Iometer v 2006.07.27 instead, which, he says, provides generalisable results, and tested the X1 and OCZ's Vertex 2 Ex and Pro devices. A slide showing IOPS performance sequentially writing 4KB blocks over time shows a pattern that is repeated again and again with other I/O activities, such as random writes.
4KB Sequential writes after Secure Erase and over one hour's sustained testing.
The Vertex 2 EX initially writes at 50,000 IOPS but after six minutes or so the IOPS number falls off a cliff and drops to 13,000 - 15,000 IOPS where it stays for test after test from seven minutes to an hour after the test session started, and continues at a 12,000 - 10,000 IOPS level from 120 minutes to 180 minutes after the test started.
George sums up his findings like this: "Flash based SSDs succumb to dramatic write IOPS degradation in as short as 10 minutes after [the] device is unpackaged or Secure Erased."
OCZ: 'Not an apples to apples comparison'
OCZ says the Vertex EX 2 "uses Single-Level Cell (SLC) memory, which has 10 times the program/erase (P/E) cycles than MLC-based drives, and features superb 4k random writes up to 50,000 IOPS to give a performance edge over the previous generation."
The supplier says: "Rated speeds may vary slightly depending on the benchmark used, drivers, windows version, bios version and file size. We recommend using ATTO, IOMeter, and PC Mark Vantage for benchmarking SSDs and to achieve maximum rated specifications."
OCZ spokeswoman Jessica Luken said: "This really isn’t an 'apples to apples' comparison, and I know exactly why this company would compare its solution to SSDs. My boss put it into perspective with this analogy: 'It is like comparing a drag car designed to accelerate fast in straight lines to a Ferrari which may not be the fastest vehicle but is built to negotiate the track'."
Luken said: "It is no secret that one of the attributes of NAND-based storage devices is a change in performance from new, fresh-out-of-box (FOB) state to 'pre-conditioned' state. All NAND-based drives would exhibit similar results in this comparison ... The OCZ Vertex2 EX series of drives is among the best in the industry at maintaining close to FOB performance," she said.
"There are, of course, corner cases that show a larger drop in performance. These corner cases – when paired with a platform and software environment that is not NAND SSD optimised – can be made to appear more significant than real world scenarios. OCZ's FAE [Field Application Engineering] team works closely to enterprise customers to ensure their environment is optimised to maintain best-in-class performance and endurance."
She added: "I’d also argue that the benchmarks provided are not accurate, as the testing doesn’t even exhibit what all other third party media are reporting, and it’s clear the testing was definitely run on an unTRIMed drive in dirty state."
OCZ Field Applications engineer Mark Hayashida added: "Though it is true that performance of Flash-based drives can change from its 'new' condition (or after a Secure Erase) over time due to certain conditions, this [DDRdrive] presentation is only looking at a particular area that calls out some worst case scenarios for Flash-based drives, such as ZFS and its intent log (ZIL) performance...
"Typically (what I call an I/O Signature of) ZIL is usually spiky in nature, not always sustained, so with TRIM support, when it comes to ZFS, that should help performance on Flash SSDs running ZIL. That would provide time for the drive to do some 'housekeeping' and maintain a higher level of performance over the life of the drive."
'Specifications are knowingly disingenuous'
Regarding TRIM, a means for Windows to tell the SSD controller about deleted data in the SSD, DDRdrive's George said: "I did run the exact same tests on Windows 7 (which does support TRIM) and the OCZ results do marginally increase. But the overall trend line stays constant; *dramatic* IOPS degradation relative to published specifications. Basically, unless one uses a development version of Iometer, which happens to default to a contrived data pattern, the published specifications are knowingly disingenuous."
The OCZ pitch is that we should use the right SSD for the right job. The X1 may well be great at ZIL performance and the Vertex 2 EX less suitable, but that does not mean you can generalise from ZIL to everyday flash SSD application performance. In other words, you can't use an X1 dragster on the open road but you can use a Ferrari Vertex 2 EX. ®