NetApp posts great Cluster-Mode SPC-1 result

<Edited to add some more information on how SPC-1 works since there was some confusion based on the comments received>

We’ve been busy at NetApp… busy perfecting the industry’s only scale-out unified platform, among other things.

We’ve already released ONTAP 8.1, which, in Cluster-Mode, allows 24 nodes (each with up to 8TB cache) for NAS workloads, and 4 nodes for block workloads (FC and iSCSI).

With ONTAP 8.1.1 (released on June 14th), we increased the node count to 6 for block workloads plus we added some extra optimizations and features. FYI: the node count is just what’s officially supported now, there’s no hard limit.

After our record NFS benchmark results, people have been curious about the block I/O performance of ONTAP Cluster-Mode, so we submitted an SPC-1 benchmark result using part of the same gear left over from the SPEC SFS NFS testing.

To the people that think NetApp is not a fit for block workloads (typically the ones believing competitor FUD): These are among the best SPC-1 results for enterprise disk-based systems given the low latency for the IOPS provided (it’s possible to get higher IOPS with higher latency, as we’ll explain later on in this post).

Here’s the link to the result and another with the page showing all the results.

This blog has covered SPC-1 tests before. A quick recap: The SPC-1 benchmark is an industry-standard, audited, tough, block-based benchmark (on Fiber Channel) that tries to stress-test disk subsystems with a lot of writes, overwrites, hotspots, a mix of random and sequential, write after read, read after write, etc. About 60% of the workload is writes. The I/O sizes are of a large variety – from small to large (so, SPC-1 IOPS are decidedly not the same thing as fully random uniform 4KB IOPS and should not be treated as such).

The benchmark access patterns do have hotspots that are a significant percentage of the total workload. Such hotspots can be either partially cached if the cache is large enough or placed on SSD if the arrays tested have an autotiering system granular and intelligent enough.

If an array can perform well in the SPC-1 workload, it will usually perform extremely well under difficult, latency-sensitive, dynamically changing DB workloads and especially OLTP. The full spec is here for the morbidly curious.

The trick with benchmarks is interpreting the results. A single IOPS number, while useful, doesn’t tell the whole story with respect to the result being useful for real applications. We’ll attempt to assist in the deciphering of the results in this post.

Before we delve into the obligatory competitive analysis, some notes for the ones lacking in faith:

  1. There was no disk short-stroking in the NetApp benchmark (a favorite way for many vendors to get good speeds out of disk systems by using only the outer part of the disk – the combination of higher linear velocity and smaller head movement providing higher performance and reduced seeks). Indeed, we used a tuning parameter that uses the entire disk surface, no matter how full the disks. Look at the full disclosure report here, page 61. For the FUD-mongers out there: This effectively pre-ages WAFL. We also didn’t attempt to optimize the block layout by reallocating blocks.
  2. There was no performance degradation over time.
  3. Average latency (“All ASUs” in the results) was flat and stayed below 5ms during multiple iterations of the test, including the sustainability test (page 28 of the full disclosure report).
  4. No extra cache beyond what comes with the systems was added (512GB comes standard with each 6240 node, 3TB per node is possible on this model, so there’s plenty of headroom for much larger working sets).
  5. It was not a “lab queen” system. We used very few disks to achieve the performance compared to the other vendors, and it’s not even the fastest box we have.


ANALYSIS

When looking at this type of benchmark, one should probably focus on :
  1. High sustained IOPS (inconsistency is frowned upon).
  2. IOPS/drive (a measure of efficiency – 500 IOPS/drive is twice as efficient as 250 IOPS/drive, meaning a lot less drives are needed, which results in lower costs, less physical footprint, etc.)
  3. Low, stable latency over time (big spikes are frowned upon).
  4. IOPS as a function of latency (do you get high IOPS but also very high latency at the top end? Is that a useful system?)
  5. The RAID protection used (RAID6? RAID10? RAID6 can provide both better protection and better space efficiency than mirroring, resulting in lower cost yet more reliable systems).
  6. What kind of drives were used? Ones you are likely to purchase?
  7. Was autotiering used? If not, why not? Isn’t it supposed to help in such difficult scenarios? Some SSDs would be able to handle the hotspots.
  8. The amount of hardware needed to get the stated performance (are way too many drives and controllers needed to do it? Does that mean a more complex and costly system? What about management?)
  9. The cost (some vendors show discounts and others show list price, so be careful there).
  10. The cost/op (which is the more useful metric – assuming you compare list price to list price).
SPC-1 is not a throughput-type benchmark, for sheer GB/s look elsewhere. Most of the systems didn’t do more than 4GB/s in this benchmark since a lot of the operations are random (and 4GB/s is quite a lot of random I/O).

SYSTEMS COMPARED

In this analysis we are comparing disk-based systems. Pure-SSD (or plain old RAM) performance-optimized configs can (predictably) get very high performance and may be a good choice if someone has a very small workload that needs to run very fast.

The results we are focusing on, on the other hand, are highly reliable, general-purpose systems that can provide both high performance, low latency and high capacity at a reasonable cost to many hosts and applications, along with rich functionality (snaps, replication, megacaching, thin provisioning, deduplication, compression, multiple protocols incl. NAS etc. Whoops – none of the other boxes aside from NetApp do all this, but such is the way the cookie crumbles).

Here’s a list of the systems with links to their full SPC-1 disclosure where you can find all the info we’ll be displaying. Those are all systems with high results and relatively flat sustained latency results.

There are some other disk-based systems with decent IOPS results but if you look at their sustained latency (“Sustainability – Average Response Time (ms) Distribution Data” in any full disclosure report) there’s too high a latency overall and too much jitter past the initial startup phase, with spikes over 30ms (which is extremely high), so we ignored them.

Here’s a quick chart of the results sorted according to latency. In addition, the prices shown are the true list prices (which can be found in the disclosures) plus the true $/IO cost based on that list price (a lot of vendors show discounted pricing to make that seem lower):

…BUT THAT CHART SHOWS THAT SOME OF THE OTHER BIG BOXES ARE FASTER THAN NETAPP… RIGHT?

That depends on whether you value and need low latency or not (and whether you take RAID type into account). For the vast majority of DB workloads, very low I/O latencies are vastly preferred to high latencies.

Here’s how you figure out the details:
  1. Choose any of the full disclosure links you are interested in. Let’s say the 3Par one, since it shows both high IOPS and high latency.
  2. Find the section titled “Response Time – Throughput Curve”. Page 13 in the 3Par result.
  3. Check whether latency rises sharply as load is added to the system.

Shown below is the 3Par curve:

3parlatency

Notice how latency rises quite sharply after a certain point.

Now compare this to the NetApp result (page 13):

Netappspclatency

Notice how the NetApp result has in general much lower latency but, more importantly, the latency stays low and rises slowly as load is added to the system.

Which is why the column “SPC-1 IOPS around 3ms” was added to the table. Effectively, what would the IOPS be at around the same latency for all the vendors?

Once you do that, you realize that the 3Par system is actually slower than the NetApp system if a similar amount of low latency is desired. Plus it costs several times more.

You can get the exact latency numbers just below the graphs on page 13, the NetApp table looks like this (under the heading “Response Time – Throughput Data”):

Netappspcdata

Indeed, of all the results compared, only the IBM SVC (with a bunch of V7000 boxes behind it) is faster than NetApp at that low latency point. Which neatly takes us to the next section…

WHAT IS THE 100% LOAD POINT?

I had to add this since it is confusing. The 100% load point does not mean the arrays tested were necessarily maxed out. Indeed, most of the arrays mentioned could sustain bigger workloads given higher latencies. 3Par just decided to show the performance at that much higher latency point. The other vendors decided to show the performance at latencies more palatable to Tier 1 DB workloads.

The SPC-1 load generators are simply told to run at a specific target IOPS and that is chosen to be the load level. The goal being to balance cost, IOPS and latency.

JUST HOW MUCH HARDWARE IS NEEDED TO GET A SYSTEM TO PERFORM?

Almost any engineering problem can be solved given the application of enough hardware. The IBM result is a great example of a very fast system built by adding a lot of hardware together:

  • 8 SVC virtualization engines plus…
  • …16 separate V7000 systems under the SVC controllers…
  • …each consisting of 2 more SVC controllers and 2 RAID controllers
  • 1,920 146GB 15,000 RPM disks (not quite the drive type people buy these days)
  • For a grand total of 40 Linux-based SVC controllers (8 larger and 32 smaller), 32 RAID controllers, and a whole lot of disks.

Putting aside for a moment the task of actually putting together and managing such a system, or the amount of power it draws, or the rack space consumed, that’s quite a bit of gear. I didn’t even attempt to add up all the CPUs working in parallel, I’m sure it’s a lot.

Compare it to the NetApp configuration:
  • 6 controllers in one cluster
  • 432 450GB 15,000 RPM disks (a pretty standard and common drive type as of the time of this writing in June 2012).

SOME QUESTIONS (OTHER VENDORS FEEL FREE TO RESPOND):

  1. What would performance be with RAID6 for the other vendors mentioned? NetApp always tests with our version of RAID6 (RAID-DP). RAID6 is more reliable than mirroring, especially when large pools are in question (not to mention more space-efficient). Most customers won’t buy big systems with all-RAID10 configs these days… (customers, ask your vendor. There is no magic – I bet they have internal results with RAID6, make them show you).
  2. Autotiering is the most talked-about feature it seems, with attributes that make it seem more important than the invention of penicillin or even the wheel, maybe even fire… However, none of the arrays mentioned are using any SSDs for autotiering (IBM published a result once – nothing amazing, draw your own conclusions). One would think that a benchmark that creates hot spots would be an ideal candidate… (and, to re-iterate, there are hotspots and of a percentage small enough to easily fit in SSD). At least IBM’s result proves that (after about 19 hours) autotiering works for the SPC-1 workload – which further solidifies the question: Why is nobody doing this if it’s supposed to be so great?
  3. Why are EMC and Dell unwilling to publish SPC-1 results? (they are both SPC members). They are the only 2 major storage vendors that won’t publish SPC-1 results. EMC said in the past they don’t think SPC-1 is a realistic test – well, only running your applications with your data on the array is ever truly realistic. What SPC-1 is, though, is an industry-standard benchmark for a truly difficult random workload with block I/O, and a great litmus test.
  4. For a box regularly marketed for Tier-1 workloads, the IBM XIV is, once more, suspiciously absent, even in its current Gen3 guise. It’s not like IBM is shy about submitting SPC-1 results :)
  5. Finally – some competitors keep saying NetApp is “not true SAN”, “emulated SAN” etc. Whatever that means – maybe the NetApp approach is better after all… the maximum write latency of the NetApp submission was 1.91ms for a predominantly write workload :)

FINAL THOUGHTS

With this recent SPC-1 result, NetApp showed once more that ONTAP running in Cluster-Mode is highly performing and highly scalable for both SAN and NAS workloads. Summarily, ONTAP Cluster-Mode:
  • Allows for highly performant and dynamically-scalable unified clusters for FC, iSCSI, NFS and CIFS.
  • Exhibits proven low latency while maintaining high performance.
  • Provides excellent price/performance.
  • Allows data on any node to be accessed from any other node.
  • Moves data non-disruptively between nodes (including CIFS, which normally is next to impossible).
  • Maintains the traditional NetApp features (write optimization, application awareness, snapshots, deduplication, compression, replication, thin provisioning, megacaching).
  • Can use the exact same FAS gear as ONTAP running in the legacy 7-mode for investment protection.
  • Can virtualize other arrays behind it.
 Courteous comments always welcome.
D

46 thoughts on “NetApp posts great Cluster-Mode SPC-1 result”

  1. Dimitris:

    Long time, no hear!

    Congrats on the SPC-1 results; they’re better than I would have expected. You’re clearly in the league with the other mid-range block devices such as 3PAR, the IBM combos, even an older HDS config. Nice work by your engineering team.

    There’s a key problem with the SPC-1, though.

    No one can look at the code unless they pay money to the SPC; and then once you do, you can’t disclose what’s in it. So customers can’t decide for themselves whether or not the test is relevant for their particular situation. It is the quintessential “black box” whose contents can’t be openly discussed.

    My opinion regarding the relevancy of the tests is the opposite of yours; the workloads generated by the SPC-1 are way too uniform and simplistic to be of much use. Relatively uniform workloads don’t benefit from autotiering; real-world ones often do.

    And, since neither of us can point to code fragments and/or specific I/O profiles, we can’t advance the discussion much past that for the time being.

    One question from the results with regards to space utilization, though. Even though you made some statements regarding “full disk surface utilization”, the test results show 40%+ of the available capacity unused.

    Since unused capacity would needlessly boost your configured price-as-tested, what the heck is it doing there?

    My first impression (unsubstantiated, mind you) is that — once again — there’s plenty of free space available to help WAFL along. Most customers I know don’t leave 40%+ of available capacity unused simply to help with performance, unless their vendor tells them so.

    The day I see you guys benchmark a near-fully-used system (like most customers have!) is the day I believe that you’ve changed your ways :)

    — Chuck

    1. Hi Chuck.

      Yes, unfortunately my day job keeps me from posting more often, but sometimes I can’t resist.

      We did fill up to 84% of usable space the last time we did the benchmark with the 3270. With an older ONTAP release, the IOPS/disk were 567 vs 579 for the beefier 6240 running 8.1.1.

      Not much of a change.

      So maybe your WAFL information is outdated?

      We used the same gear we had for the SPEC test (6240 with 450GB disks) and added enough disks to get the posted result. Fewer disks would have meant a slower result simply due to less spindles – at the moment, with SPC-1, we can’t quite crack 600 IOPS/disk. But we’re working on it.

      How about you submit a VMAX? Or VNX? Anything? That will put you in a better place to criticize anyone else’s product.

      And the benchmark isn’t that uniform – there are definite hot spots and the percentage small enough to be placed on SSD. Maybe you need to speak to your engineers and they can re-examine the benchmark. It’s absolutely a fit for autotiering.

      IBM’s result using autotiering proves this, the system stabilizes 19 hours into the benchmark. http://bit.ly/NezCt4

      So the question I have is – why does everyone avoid autotiering? Maybe because it only solves a perception problem? What is the ROI?

      Thx

      D

      1. Hi Dmitri:

        Each of the product groups here at EMC decide what they want to do, including submitting to benchmarks. The Isilon guys, for example, like the SPECnfs.

        Since the SPC-1 test hasn’t changed in over a decade, neither has my general opinion: it’s not an especially useful nor relevant test. That’s probably why no one has done it here yet.

        Although there was an interesting fraternity stunt a while ago where NetApp submitted a test on our behalf. Big laughs all around …

        That being said, the Isilon guys have done some great things with I/O latency and block protocols in the their latest Mavericks release. It’d be interesting to see how, say, a 4 node Isilon does in SPC-1. Not for marketing purposes, just to see how the code stands up under that particular relatively uniform block profile.

        A back-of-the-napkin calculation tells me it would do decently in IOPS/disk, IOPS/node and the $/IOPS comparison.

        Along those same lines, the Iomega guys are doing some nice stuff with their NAS boxes which also support iSCSI. They tell me they’re getting some decent performance these days. Another group that I should mention this too.

        But that wouldn’t mean I’d go out and claim that Isilon or Iomega was Tier 1 Enterprise Storage :)

        Still confused as to why you’d burden your as-tested config with 40%+ unused capacity, which of course badly skews your $/IOPS figure.

        All you would have to do is take the disks out of the box.

        Funny, that.

        — Chuck

        1. Given that Mr Hollis is now apparently a technical expert he no doubt understands the issues of IOPS density and that there are a certain number of IOPS you can get out. Most vendors, including EMC address this in benchmarks by using RAID-10 and small drive sizes, or all flash drives, or possibly a combination of all 3. NetApp does not we use 450GB 15K drives which are the smallest and densest mechanical drives commonly available today.

          This is why even the SPC-1 executive summary includes the application utilization number which is the amount of capacity used in the test divided by the physical storage. The HDS VSP result (which is probably the most comparable in terms of performance to our submission), had an application utilisation figure of 29.25%, our submission had one of 36.98%. As far as I’m aware, our record in this area is second to none, across ANY benchmark, mostly because of our consistent use of RAID-DP,

          If you’re not familiar with the concepts, or want to challenge these assertions, check out

          http://storagewithoutborders.com/2011/09/10/how-does-capacity-utilisation-affect-performance/

          Regards
          John

          1. Hi John

            I do have a reasonable understanding of how IO density works in a WAFL world (or using similar approaches), thank you.

            I also know how IO density works in a purpose-built block storage devices, vs one that has been adapted to the purpose.

            And I also know its a tough sell to ask customers to choose between decent capacity utilization and decent performance.

            Ideally, a storage product should be able to offer both. At the same time, that is :)

            Best regards

            – Chuck

          2. Hi Chuck,

            Maybe you didn’t have time to read my previous response.

            http://recoverymonkey.org/2010/11/09/176/

            We got about the same IOPS/disk with 84% full vs usable space in that one (over 550 SPC-1 IOPS/disk). We also turned on real-time block reallocation.

            Maybe you care to comment on that instead. At the moment you’re sounding a bit like a broken record.

            Anyway, I see a lot of theorizing on your part and not enough proving.

            Until EMC posts any numbers, I doubt you have much grounds to claim anything or criticize any participating vendor’s results.

            At the moment, NetApp has proven that ONTAP Cluster-Mode hangs up there in the Tier1 space for both block and NAS workloads.

            With one architecture.

            D

          3. No Chuck, I don’t think you do understand how IOPS density works as it applies to disk drives, either that, or you’re being disingenuous.

            Disk drives got exponentially bigger, while their performance remained the same over the last 10 years. A 73GB 15K drive did pretty much the same number of IOPS as a 450GB, or 600GB, or 900GB 15K drive. That means when you are performance bound and need to throw lots of disks at the solution, you’re going to have a bunch of capacity spare. As the minimum disk size gets bigger the more space you’re going to have for the same amount of IOPS.

            This is why it’s so important to get the most out of your disks by effective combination with SSD/Flash, and why the IOPS per disk shown in this benchmark is an important metric to measure. 500+ is a very good effort.

            Becuase the IOPS density of SSD is so good, this issue of extra unused capacity can be addressed by using all SSD configurations, as EMC does in most of its submisions but that still isn’t cost effective for large data sets.

            Also while Mr Hollis’ pot is busy calling the NetApp kettle black Let’s take an EMC benchmark using mechanical disks accelerated by SSD –

            http://www.spec.org/sfs2008/results/res2011q2/sfs2008-20110527-00193.html

            60K IOPS from 161 x 300GB 10K Drives – 41 TB usable – only 7 of which was actually used in the test. – 17% capacity utilisatoin

            The closest comparable submission from NetApp in terms of performance is

            http://www.spec.org/sfs2008/results/res2009q3/sfs2008-20090727-00126.html

            56 x 300GB 15K drives – 10.9 TB usable of which 7TB was used in the test – 64% capacity utilisation

            Can EMC show better utilisation figures when combining disk and flash than NetApp ?
            I doubt it, but if you can, please prove it by submitting a benchmark that uses Fastcache

    1. Chris,
      you’ll also note that our $/IOP figure is based on our list price, others like HP and HDS apply around a 50% discount to their list price. I’m sure if you check with your usual sources you’ll get a pretty good idea of what the relationship is between our list price and our street price :-)

    2. Our $/IOPS is “high” compared to what other scale-out unified storage system? :)

      As John points out, if you correct for list price (which I did) then it’s extremely competitive.

  2. On a $/IOP the Huawei fares the best of the listed entries, however, the “advantage” in $/IOP evaporates when you have to cool, power and house, 166% or 720 more drives.

    The Huawei system has a list price of 43% more in order to provide a -13% $/IOP advantage. Most people would not consider this a particularly good tradeoff

  3. OK it took me a while, but I finally figured this out. Chuck is a NetApp mole planted inside EMC by NetApp to make EMC look as foolish as possible.

    Why else would:
    – Chuck claim he no longer cares about NetApp on his blog, yet be the first and most frequent troller on NetApp blogs?
    – Chuck discredit hundreds of man-years of professional performance engineering with “back of the napkin” math for an unreleased product?
    – Chuck fail to grasp basic RAID10 math yielding a ceiling of 50% usable capacity because it’s impossible to get max performance from a production EMC array any other way?

    This is all proof NetApp’s guerrilla marketing campaign is a smashing success. Keep it “UpChuck”! :D

    1. Could not agree more with Radek. I am late to the party on this old post but it is typical EMC. They act like they are leading the pack but in reality they are simply barking louder than anyone else. Keep barking EMC while everyone else actually moves forward.

  4. Would it not be a reasonable compare of latency at equivalent IOPS? For example, these six separate systems clustered together appear to be at 225K IOPS at 90% workload which matches up nicely with the HP 3PAR P10000 at 50% workload. And if you take the latencies at that IOP level then there’s less than a millisecond difference, right?

    Did I also read correctly the config having 6 controller nodes (6 physically separate systems clustered together) each with 48 GB memory/cache and 512 GB FlashCache? If math serves me well, that’s about 3.3TB of cache/Flash that’s fronting 71TB. Or 1TB of cache assisting each 21TB of ASU capacity. Is that correct? The HP 3PAR config proved scale-out by using 768GB of cache in front of 230TB. Would this minimal cache assisted config not test scale-out capability of the architecture versus requiring assistance?

    Speaking of capacity utilization, this statement really got my attention- “As far as I’m aware, our record in this area is second to none, across ANY benchmark, mostly because of our consistent use of RAID-DP” In viewing the results, I see for NetApp-
    Application Utilization – 36.98%, Protected Application Utilization 43.83%, and Unused Storage Ratio 43.2%
    and for 3PAR-
    Application Utilization- 40.23%, Protected Application Utilization 80.46%, and Unused Storage Ratio 14.53%

    Per your statement, how would you compare these numbers and the importance in customer environments? I guess- which is second?

    Thanks so much for sharing and providing an opportunity for “courteous comments.”

    LaMills

    1. Hi LaMills,

      A 1ms difference is huge for certain companies (like some trading firms).

      And do I read this right that you’re complaining about NetApp using the cache that comes free of charge with the 6240 boxes? What would you like us to do, disable it?

      D

      1. No complaints from me, just “courteous comments” to clarify a few items. Points being it’s fairly easy to get a good latency number using 1/3 of the supported spindles sitting behind a full host cache (not a good example of scale-out to me) and despite using nearly 200TB of raw capacity you get less than 90TB of written data. Have a great weekend.

        Go HP 3PAR!!!

        1. LaMills,

          Actually, I’ll disagree – and qualify why:

          We have demonstrated systems in the past running at much closer to full as I keep mentioning and the various commenters keep ignoring: http://bit.ly/Mp32Ic

          In those cases, we got similar IOPS/disk efficiency as we did with the 6240.

          The NetApp performance in SPC-1 comes from 3 places:

          1. WAFL being able to optimize the writes
          2. Extremely intelligent cache and prefetch algorithms
          3. Flash Cache (to a lesser extent – much of the SPC-1 I/O is writes and, as competitors keep saying, Flash Cache doesn’t cache writes :) )

          I contend that 3Par wouldn’t be able to hit 250,000 IOPS at similar low latency as NetApp with a similar number of disks, RAID6 and same amount of data. The IOPS/disk are simply not there to do so.

          Nor could 3Par resort to using more cache since there isn’t a way to add more.

          Maybe 3Par could use autotiering – why not use that?

          Don’t get me wrong, your super-high latency notwithstanding, 3Par (like all the listed vendors) got a great number. At best, though, you can use half your total capacity since RAID10 is needed to get the result. Like all the listed vendors.

          D

  5. By basing $/IOPS on list price you are, I’d argue, invalidating the point of the $/IOPS measure as a comparison point between different systems. Yes, you supply list prices for the other systems you cite and $/IOPS numbers based on these but no customer pays list prices for systems like these. If, as you say, other suppliers use a 50% list price metric then, giving your list price 50% cut, your $/IOPS is $3.35. That looks more reasonable.

    1. Chris, list price is an only ‘firm point’, discount may be vary (and sometime is it did). You cannot define ‘standard discount rate’, but a list price is a well-documented (usually) and stable point for almost every vendor.
      In most cases, using a ‘discounts’ in TSC SPC-1 prices is a well-known ‘little dirty trick’ for lowering an $/IOPS rates and everybody knows it.

  6. Hi Dimitris,

    “2. There was no performance degradation over time.”

    That’s a pretty bald, yet indefinite statement.

    How long did the test last? More importantly: what was the % of overall usage of the total capacity and with what type of data (small, big files, db, etc)?

    In my experience, everything fragments over time, especially if it writes randomly and is a almost full capacity.

    Thanks

    Marin

      1. Hi Dimitris,

        I read the full disclosure. There is nothing which suggest that performance will hold in time with a mixed, real-life workload and at almost full capacity.

        Don’t get me wrong, I’m not picking at NetApp. I believe EVERY storage system degrades over time due to fragmentation.

        As I wrote before, I just think its a bold statement to claim the contrary.

        Marin

        1. Clearly, this shows that, during the benchmark, this system exhibited stable, low latency that didn’t degrade over time.

          The workload is mixed. As to real-life – real-life includes many variations. Every company’s “real-life” is different from another’s.

          All this benchmark does is test a very difficult I/O scenario, typically more difficult than found in most “real-life” cases, which a lot of churn.

          I still believe SPC-1 is far more valuable than running straight 4K random. If you do that, then performance would be far higher for all systems listed.

          As an example – here’s 100% 4K random writes to an older NetApp box (3140, not that many disks, not running Cluster-Mode). http://www.youtube.com/watch?v=U5wb3uoeYAI&feature=player_embedded

          D

          1. 9 hrs of tests at less than 50% of capacity should prove that there is no degradation over time?!

            Come on Dimitris, you must be kidding me…! :)

            (aren’t these systems sold with a 3yr warranty?)

          2. All any participating vendor can prove is what’s tested within the parameters of the benchmark. Like I said in my previous response: “during the benchmark” etc.

            Having said that, at least NetApp has a way to prevent fragmentation over time, in real-time. It’s called “reallocate”. Our previous SPC-1 benchmark was run with it turned on.

            http://bit.ly/Mp32Ic

            Look at page 64.

            Using it for DB-type workloads is recommended.

            What do other vendors have? Since, by your admission, everyone suffers from it, especially systems relying on disk pools and thin provisioning (which, naturally, creates “fragmentation”).

            D

    1. Protected application utilization which maps closes to capacity utilization is 43.83%. Maybe that’s good, but about half of what’s done from HP 3PAR.

  7. I think the only way to satisfy the competitive FUD-mongers, would be to crack open all the HDDs and by using a sharpie, preferably a red one, we will need to mark the areas on each platter occupied with data so a clear and undisputed determination can be made as to whether or not the disks were short stroked. :-)

    These are substantially good results and rather than defending them, the burden should be placed upon those who who criticize them to put their “engines” where their mouth is…

  8. “All any participating vendor can prove is what’s tested within the parameters of the benchmark. Like I said in my previous response: “during the benchmark” etc.”

    Hence we can conclude that measuring performance degradation (and making claims thereof) during a 9 hour test is meaningless.

    cheers.

    Marin

    1. No, we can’t conclude it’s meaningless. The SPC-1 benchmark creates a HUGE amount of write churn, showing stability even for 9 hours is meaningful since it could simulate months’ worth of churn in “real-life” environments that do a lot less writes.

      1. As you wish.

        For all we know, during the first 9 hours of life of a system, WAFL is writing onto new unused blocks (especially since the system never reaches not even 50% of capacity).

        This is far from a “real life” intended use of an enterprise storage system.

        (And yes, I DID read the full disclosure. If there is any paragraph which can back up your claims, I would be happy to re-examine it).

        cheers.

        M.

        1. We can go round and round… Look at this again: http://bit.ly/Mp32Ic – almost 84% of usable storage was used there. Similar high IOPS/disk was achieved. Not that much free space left for WAFL to do its magic, yet, again, performance stayed stable for a workload that’s write-heavy. Smaller NetApp system, less disks, older ONTAP OS.

          Addressable storage capacity: 21,659.386 GB, unused storage: Only 4,248.373 GB (so 25,907.759 GB was usable total, and we used 21,659.386 for the test).

          To spare everyone the suspense: ONTAP performance does start to degrade after about 85% full assuming a lot of writes are going on. Before 85% full, it is far faster than RAID10. As you can see from the tables in the main post.

          So let’s do some math:

          Let’s say I have a 6240 system with 240x 600GB drives.

          The real usable space (after spares, overheads, and dedicated OS drives) is 99TiB (real, binary TB – in Base10 TB it would be 108.86).

          Let’s assume you want your guaranteed super-high IOPS/disk and don’t want to fill it past 85%. That leaves you with about 84 TB usable.

          So, 84TB usable to get far higher performance than on a system with RAID10.

          How much capacity would the RAID10 system have with 240 600GB drives incl. spares and OS drives etc?

          About 60TB?

          So, using this very, very basic math, what’s better value?

          With the same number of disks (240):

          1. 84TB at high performance and high reliability, or
          2. 60TB at less performance and less reliability?

          Hmmm… let’s think about that.

          D

  9. I hear what you are saying, but I have nothing to add to what I already said before.

    M.

    Full disclosure: I am not working for any company neither do I represent anyone but myself.

  10. If you observe NetApp’s progression over the years, the conclusion is inevitable. They started out as a workgroup NAS vendor, then continued adding more and more enterprise-grade functionality, reliability and finally now, enterprise-class performance. Make no mistake, this is a Tier1 result from what is now a Tier1 SAN *and* NAS vendor.

    See JJM’s latest article for further affirmation:

    http://www.storagenewsletter.com/news/marketreport/western-european-external-disk-idc-1q12

    For me, this benchmark result is nothing more than irrefutable proof that after many years of development, C-Mode represents a new tranche of scalability for Data ONTAP. I can see why those being disrupted by NetApp are upset and try to discredit the results via scattered nit-picking, but from an objective perspective, at the moment no one else can match NetApp’s combination of storage functionality, efficiency and performance at scale.

    Which is what makes it my favorite array to work with.

  11. Hi There,

    Just got back from 5 weeks vacation, and i almost missed all this passionate discussion and interest! So i want to make a comment on two customer centric aspects to this NetApp result: – Efficiency and Complexity.

    Efficiency – In my opinion, a performance result on a storage array operating at ~44% application utilisation with 43% Unused (but purchased) capacity doesn’t tell me much about how the array will perform at the most common utilisation range of around 70%, 80%, 90% ++… which is where most customers in Asia tend to run their arrays at. –> NetApp – Can you rerun this SPC-1 test at 85% Protected Application Utilisation and < 15% Unused Storage Ratio?

    Complexity – App C in the FDR documents how to build the TSC. Can you explain why it takes almost 30 pages of Python script to configure a 6 node storage cluster with 12 LUNs and export them to 6 hosts ? The 30 pages of Python are almost too complex for me to calculate our complexity metrics on this config!

    Many Thanks,

    Paul – HP Asia Pac.

    1. Hi Paul,

      This question has been asked a few times and I’ve answered multiple times. Please check the comments. But, once more:

      http://bit.ly/Mp32Ic

      We already ran the SPC-1 tests with a way slower controller and an older ONTAP version at 84% full and got about the same IOPS/disk as with the recent result (with more latency but it was a much weaker box). So, your utilization theory doesn’t hold much water.

      So how about this:

      HP can run the SPC-1 test with RAID6 (since NetApp RAID-DP is similar to RAID6) at 84% full.

      Let’s compare notes then.

      Because I’m not sure you’re calculating things quite correctly – you see, 3Par used RAID10, so, by definition, 50% of the capacity is gone.

      Do your customers in Asia all run RAID10? :)

      D

      1. Dimitris,

        Just want to correct above reply. For the 3270A SPC-1 result, the reported “Protected Application Utilisation” result is 70%, NOT the 84% you are claiming. Since this blog posted is regarding an SPC-1 benchmark result; i think you should keep to referencing data points ratified by the SPC and reported within SPC-1 disclosure reports. Creating your own statistics and metrics dose nothing to make SPC-1 results usable and comparable for our customers.

        Likewise, for the recent 6240 result, the utilisation is just under 44%; with 43% of the physical capacity unusable by the SPC workload application. This isn’t an efficient configuration me thinks!.

        And for the record, the SPC metric on efficiency does include the RAID overhead as well.

        Nice recent posts on short-stroking and IOPs explained; couldn’t agree more.

        Paul [HP]

        1. Paul,

          I have updated the 3270 results post with explanations on the calculations: http://recoverymonkey.org/2010/11/09/176/

          Please read the article.

          A very important metric is “Application Utilization”. That simply means how much of the storage was used vs purchased. Systems with RAID10 will always have under 50% in that metric even if 100% of the usable space is filled up (since you lose half your space with RAID10 to begin with).

          The 6240 result has an “Application Utilization” figure similar to the 3Par result (36.98% vs 40.23%). Most of the other results are far worse (the HDS VSP one is 29.25%, meaning the benchmark data occupied 29.25% of the total raw space sold).

          The 3270 result had an unheard-of Application Utilization of 60.51% – far, far higher than any other SPC-1 result to date.

          Meaning that customers would be able to fill up the box to 60.51% of the raw capacity sold and get high performance.

          That’s a good measure of efficiency.

          D

Leave a comment for posterity...