Examining value for money regarding the SPEC benchmarks

Some of the comments in my previous post asked about $/IOPS and $/TB.

Since SPEC doesn’t require prices to be listed, I did my own analysis.

The NetApp numbers are simply 4x the existing 6240 result, which is what EMC did with their submission, they used 4x separate VNX systems and aggregated the result.

I used this clarifying analogy over at Nigel’s blog to explain why this makes sense before anyone yells “but this is not published”:

A storage system typically has some kind of bottleneck – cluster interconnect, number of drives, bandwidth to the controller, etc.

When you’re testing a single system, you’re ultimately hitting one of those bottlenecks.

If you’re testing multiple systems independent of each other, they do not share the bottlenecks (since they’re separate), and your performance will scale linearly as you add systems.

For example, if 1 truck can hold 10 tons of stuff, 4 like trucks will hold 40 tons of stuff, 10 trucks 100 tons, etc. There’s no limit.

Once you inject a limiting factor (“the trucks all have to fit on a bridge and the bridge can take this much load and it’s this big”) then you will have a limitation on how many trucks you can load and put on that bridge.

EMC tested 4 separate “trucks”. In that same way, I can add up the result of 4 separate NetApp “trucks”. Here are the results:

EMC NetApp Difference
Cost (approx. USD List) 6,000,000 5,000,000 NetApp is over 16% cheaper in absolute terms
SPEC SFS NFS IOPS 497,623 762,700 NetApp is 53% faster in absolute terms
Average Latency (ORT) 0.96 1.17 EMC offers a mere 18% less latency (with less NFS OPS) despite using only SSDs!
Space (TB) 60 343 NetApp offers 5.7 times more usable space
$/SPEC NFS IOPS 12.06 6.56 Netapp is 45.6% less expensive per SPEC NFS operation
$/TB 100,000 14,577 NetApp is less than 1/6 the price of EMC per TB
RAID RAID5 RAID-DP NetApp is thousands of times more reliable
Boxes needed to accomplish result 15 (4x separate VNX, each with 2 controllers, plus a total of 5x Celerra VG8 heads and 2 Control Stations) 8x unified controllers NetApp is far less complex (the benefit of a truly unified architecture)

Who can spot the better deal? Smile

I added the latency in the chart, thanks to my buddy Mark Twomey for pointing it out.

You see, people needing enterprise NAS with that kind of performance usually need speed, plenty of space and high reliability. Not just one of the three. BTW, here’s a paper on relative RAID reliability.

NetApp provides all three, in spades, plus great value for money, a truly simple, flexible unified system, and efficiency.

Most customers want to see how a real configuration performs. I refer customers to our SPEC and SPC results constantly since quite frequently their desired configuration is very similar.

Which makes benchmarking realistic configurations actually useful – imagine that.

Maybe EMC needs to submit results with VNX the way they sell it to people, for example:

  • A mix of SSD cache, SSD, high-speed SAS and high-capacity SAS
  • Autotiering
  • RAID6
  • A typical amount of space for a configuration that size

Then submit results.

Keep your existing result of course, but also show the people how what you actually sell them really performs.

I still don’t understand why this is such a hard concept.


Technorati Tags: ,,,,

8 Replies to “Examining value for money regarding the SPEC benchmarks”

  1. Totally agree with your analysis. I’ve actually found the FAS3270 SPC result one of the most realistic for customers. It is something that they would buy and within their budget for real production workloads. FAS3270, 120 SAS HDD and Flashcache doing 64k IOps at 8ms latency. Not showing to compare better perf than competitors showing them to point out how we can reach their, for example, 20,000 IOps requirement quite easily with a configuration that is cost effective

  2. So, striping 2 file systems across 105 flash drives (VNX) vs. striping 2 file systems across 120 SAS drives (6240) means latency will only improve 18%? I’d love to see this with the config you suggested (RAID-6, Snapshots, FAST, cloning).

  3. Shouldn’t your first column start with

    “NetApp is merely 16% cheaper in absolute terms. Despite not using expensive SSDs!”

    I’m a NetApp customer by the way before you use this attempt at humour as a spring board for more partisan bickering 🙂

    1. @John, you are pointing out that NetApp is “16% cheaper” while forgetting that the NetApp config provides “5.7 times” the capacity. I think most people would feel comfortable saving “only” 16% while gaining an extra 470% in capacity, since customers are often concerned with capacity.

      I think you are trying to suggest that NetApp is somehow over charging for their SAS and implying that the NetApp configuration should be significantly cheaper. However would you rather pay $100k per TB or ~$15k per TB?

      1. I think I’d rather pay for ops at a target capacity, which right now is very very tough with netapp if you need low capacity high performance. This has pretty much always been the case. I don’t think his comments should be dismissed off hand quite so easily.

        Upgrades for capacity/performance come in $40k chunks and head swaps are brutally expensive. EMC and 3PAR are no better here at all.

        What I think he’s saying that’s glossed over is that getting say 30TB on NetApp pushing 10k write ops sustained is not any cheaper than anyone else using SSD’s. With NetApp you can get 100’s of TB to hit that number but maybe you don’t need all that space……

        1. @Pablo906,

          Actually NetApp also offers SSDs, so high performance with low capacity could easily be achieved the way EMC did it in this benchmark. That would cover the low capacity/high performance need you state.

          That’s easy.

          What we did was run it with normal drives, get a lot more capacity for the money, yet not suffer performance-wise.

          That’s harder 🙂


Leave a comment for posterity...