Over the last few days I’ve had the privilege of talking solid state with Tony Leach of OCZ, Olin Coles of Benchmark Reviews and Tom Colarusso from FiringSquad. We’ve all come to the consensus that SSDs are amongst the most inappropriately-tested devices on the market.

Knowing that programs like ATTO and transfer tests give unreliable results, we use them anyhow because there are no tools that can adequately do the job. Combine this with bad Windows support, crappy controllers, skewed expectations and the eccentricities of the SSD’s performance/reliability mechanisms, we have a whole lot of irritated reviewers and confused users.

As an example, wear leveling technology can give read/write performance that can swing up to 30MB/s in either direction with every run of a transfer test. Can we really be relying on these types of programs given their penchant for specious results?

The entire SSD industry (users, vendors and reviewers) have been forced to perpetuate inaccurate metrics for lack of something that better describes an SSD’s performance characteristics. For want of that better metric, even SSD manufacturers themselves gravitated towards mechanical throughput figures (MB/s sustained) to describe a technology that rarely operates in a sustained transfer environment.

We also speak in terms of IOPS, burst and seek, but they are just a piece of a superior and more accurate metric. Whatever comes of this, it should answer these important questions: How can we demonstrate objectively that one SSD is better than another, and both are superior to the HDD?

We’re close to doing it right, but we’re not there yet. Not even manufacturers with millions invested in R&D are doing it right. But for the sake of accurate numbers — the pride of every reviewer — we are working hard to develop a standardized testing methodology. The need to review these complicated and dynamic products, to serve users in a reliable and accurate way, should be the priority of every website. It’s that important.

For more on the eccentricities of the SSD, take a look at these three pieces to read why this issue is so complicated:

Icrontic
Benchmark Reviews
Toms Hardware