NetApp posts world-record SPEC SFS2008 NFS benchmark result

Just as NetApp dominated the older version of the SPEC SFS97_R1 NFS benchmark back in May of 2006 (and was unsurpassed in that benchmark with 1 million SFS operations per second), the time has come to once again dominate the current version, SPEC SFS2008 NFS.

Recently we have been focusing on benchmarking realistic configurations that people might actually put in their datacenters, instead of lab queens with unusable configs focused on achieving the highest result regardless of cost.

However, it seems the press doesn’t care about realistic configs (or to even understand the configs) but instead likes headline-grabbing big numbers.

So we decided to go for the best of both worlds – a headline-grabbing “big number” but also a config that would make more financial sense than the utterly crazy setups being submitted by competitors.

Without further ado, NetApp achieved over 1.5 million SPEC SFS2008 NFS operations per second with a 24-node cluster based on FAS6240 boxes running ONTAP 8 in Cluster Mode. Click here for the specific result. There are other results in the page showing different size clusters so you can get some idea of the scaling possible.

See below table for a high-level analysis (including the list pricing I could find for these specific performance-optimized configs for whatever that’s worth). The comparison is between NetApp and the nearest scale-out competitor result (one of many EMC’s recent acquisitions –  Isilon, the niche, dedicated NAS box – nothing else is close enough to bother including in the comparison).

BTW – the EMC price list is publicly available from here (and other places I’m sure): http://www.emc.com/collateral/emcwsca/master-price-list.pdf

From page 422:

S200-6.9TB & 200GB SSD, 48GB RAMS200-6.9TB & 200GB SSD, 48GB RAM, 2x10GE SFP+ & 2x1G $84,061. Times 140…

Before we dive into the comparison, an important note since it seems the competition doesn’t understand how to read SPEC SFS results:

Out of 1728 450GB disks (the number includes spares and OS drives, otherwise it was 1632 disks), the usable capacity was 574TB (73% of all raw space – even more if one considers a 450GB disk never actually provides 450 real GB in base2). The exported capacity was 288TB. This doesn’t mean we tried to short-stroke or that there is a performance benefit exporting a smaller filesystem – the way NetApp writes to disk, the size of the volume you export has nothing to do with performance. Since SPEC SFS doesn’t use all the available disk space, the person doing the setup thought like a real storage admin and didn’t give it all the available space. 

Lest we be accused of tuning this config or manually making sure client accesses were load-balanced and going to the optimal nodes, please understand this:  23 out of 24 client accesses were not going to the nodes owning the data and were instead happening over the cluster interconnect (which, for any scale-out architecture, is worst-case-scenario performance). Look under the “Uniform Access Rules Compliance” in the full disclosure details of the result in the SPEC website here. This means that, compared to the 2-node ONTAP 7-mode results, there is a degradation due to the cluster operating (intentionally) through non-optimal paths.

EMC NetApp Difference
Cost (approx. USD List) 11,800,000 6,280,000 NetApp is almost half the cost while offering much higher performance
SPEC SFS2008 NFS operations per second 1,112,705 1,512,784 NetApp is over 35% faster, while using potentially better RAID protection
Average Latency (ORT) 2.54 1.53 NetApp offers almost 40% better average latency without using costly SSDs, and is usable for challenging random workloads like DBs, VMs etc.
Space (TB) 864 (out of which 128889GB was used in the test) 574 (out of which 176176GB was used in the test) Isilon offers about 50% more usable space (coming from a lot more drives, 28% more raw space and potentially less RAID protection – N+2 results from Isilon would be different)
$/SPEC SFS2008 NFS operation 10.6 4.15 Netapp is less than half the cost per SPEC SFS2008 NFS operation
$/TB 13,657 10,940 NetApp is about 20% less expensive than EMC per usable TB
RAID Per-file protection. Files < 128K are at least mirrored. Files over 128K are at a 13+1 level protection in this specific test. RAID-DP Ask EMC what 13+1 protection means in an Isilon cluster (I believe 1 node can be completely gone but what about simultaneous drive failures that contain the sameprotected file?)NetApp RAID-DP is mathematically analogous to RAID6 and has a parity drive penalty of 2 drives every 16-20 drives.
Boxes needed to accomplish result 140 nodes, 3,360 drives (incl. 25TB of SSDs for cache), 1,120 CPU cores, 6.7TB RAM. 24 unified controllers, 1,728 drives, 12.2TB Flash Cache, 192 CPU cores, 1.2TB RAM. NetApp is far more powerful per node, and achieves higher performance with a lotless drives, CPUs, RAM and cache.In addition, NetApp can be used for all protocols (FC, iSCSI, NFS, CIFS) and all connectivity methods (FC 4/8Gb, Ethernet 1/10Gb, FCoE).

 

Notice the response time charts:

IsilonVs6240response

NetApp exhibits traditional storage system behavior – latency is very low initially and gradually gets higher the more the box is pushed, as one would expect. Isilon on the other hand starts out slow and gets faster as more metadata gets cached, until the controllers run out of steam (SPEC SFS is very heavy in NAS metadata ops, and should not be compared to heavy-duty block benchmarks like SPC-1).

This is one of the reasons an Isilon cluster is not really applicable for low-latency DB-type apps, or low-latency VMs. It is a great architecture designed to provide high sequential speeds for large files over NAS protocols, and is not a general-purpose storage system. Kudos to the Isilon guys for even getting the great SPEC result in the first place, given that this isn’t what the box is designed to do (the extreme Isilon configuration needed to run the benchmark is testament to that). The better application for Isilon would be capacity-optimized configs (which is what the system is designed for to begin with).

 

Some important points:

  1. First and foremost, the cluster-mode ONTAP architecture now supports all protocols, it is the only unified scale-out architecture available. Any competitors playing in that space only have NAS or SAN offerings but not both in a single architecture.
  2. We didn’t even test with the even faster 6280 box and extra cache (that one can take 8TB cache per node). The result is not the fastest a NetApp cluster can go :) With 6280s it would be a healthy percentage faster, but we had a bunch of the 6240s in the lab so it was easier to test them, plus they’re a more common and less expensive box, making for a more realistic result.
  3. ONTAP in cluster-mode is a general-purpose storage OS, and can be used to run Exchange, SQL, Oracle, DB2, VMs, etc. etc. Most other scale-out architectures are simply not suitable for low-latency workloads like DBs and VMs and are instead geared towards high NAS throughput for large files (IBRIX, SONAS, Isilon to name a few – all great at what they do best).
  4. ONTAP in cluster mode is, indeed, a single scale-out cluster and administered as such. It should not be compared to block boxes with NAS gateways in front of them like VNX, HDS + Bluearc, etc.
  5. In ONTAP cluster mode, workloads and virtual interfaces can move around the cluster non-disruptively, regardless of protocol (FC, iSCSI, NFS and yes, even CIFS can move around non-disruptively assuming you have clients that can talk SMB 2.1 and above).
  6. In ONTAP cluster mode, any data can be accessed from any node in the cluster – again, impossible with non-unified gateway solutions like VNX that have individual NAS servers in front of block storage, with zero awareness between the NAS heads aside from failover.
  7. ONTAP cluster mode can allow certain cool things like upgrading storage controllers from one model to another completely non-disruptively, most other storage systems need some kind of outage to do this. All we do is add the new boxes to the existing cluster :)
  8. ONTAP cluster mode supports all the traditional NetApp storage efficiency and protection features: RAID-DP, replication, deduplication, compression, snaps, clones, thin provisioning. Again, the goal is to provide a scale-out general-purpose storage system, not a niche box for only a specific market segment. It even supports virtualizing your existing storage.
  9. There was a single namespace for the NFS data. Granted, not the same architecture as a single filesystem from some competitors.
  10. Last but not least – no “special” NetApp boxes are needed to run Cluster Mode. In contrast to other vendors selling a completely separate scale-out architecture (different hardware and software and management), normal NetApp systems can enter a scale-out cluster as long as they have enough connectivity for the cluster network and can run ONTAP 8. This ensures investment protection for the customer plus it’s easier for NetApp since we don’t have umpteen hardware and software architectures to develop for and support :)
  11. Since people have been asking: The SFS benchmark generates about 120MB per operation. The slower you go, the less space you will use on the disks, regardless of how many disks you have. This creates some imbalance in large configs (for example, only about 128TB of the 864TB available was used on Isilon).

Just remember – in order to do what ONTAP in Cluster Mode does, how many different architectures would other vendors be proposing?

  • Scale-out SAN
  • Scale-out NAS
  • Replication appliances
  • Dedupe appliances
  • All kinds of management software

How many people would it take to keep it all running? And patched? And how many firmware inter-dependencies would there be?

And what if you didn’t need, say, scale-out SAN to begin with, but some time after buying traditional SAN realized you needed scale-out? Would your current storage vendor tell you you needed, in addition to your existing SAN platform, that other one that can do scale-out? That’s completely different than the one you bought? And that you can’t re-use any of your existing stuff as part of the scale-out box, regardless of how high-end your existing SAN is?

How would that make you feel?

Always plan for the future…

Comments welcome.

D

PS: Made some small edits in the RAID parts and also added the official EMC pricelist link.

 

Technorati Tags: , , , , , , , , ,

 

55 thoughts on “NetApp posts world-record SPEC SFS2008 NFS benchmark result”

  1. Hi — Chuck from EMC here!

    I think industry standard benchmarks (like the SPEC) are useful, since they (a) do a reasonable job of emulating real-world workloads, (b) aren’t subject to some of the games we’ve seen elsewhere, and (c) bring out the best in us competitive vendors.

    So, congratulations on your result!

    Now, a bit of “clarification”

    1 — To the best of my knowledge, the SPEC test doesn’t specify the cost of products. I think that’s your own calculation. In the spirit of “fair play” from the SPEC folks, I think you should be identifying that as such.

    2 — I also think it’s important to share the exported (usable) capacity for each configuration, as do the SPEC folks. Looks like 228TB for the NetApp config, and 864TB for the EMC/Isilon config — am I correct?

    3 — I noticed that the NetApp submission said “single namespace” and not “single filesystem”. There’s a material difference, as you know. How many individual and separate filesystems were used to deliver that 224TB usable?

    Either way, you’ve got a SPEC NFS record to be proud of, so take your victory lap. That being said, we both know there’s more to *real* scale-out NAS than simple clustering. And your benchmark results sort of prove our case.

    Best regards

    — Chuck

  2. Hi Chuck, that was quick!

    If you read my post carefully, regarding cost I say: “including the list pricing I could find for these specific performance-optimized configs for whatever that’s worth”.

    SPEC doesn’t demand cost to be published but doesn’t prohibit it either. If the numbers are not right, correct them pls.

    The rest of your arguments completely invalidate the EMC submissions with Celerra… but you were very quick to post those when they became live, in order to grab headlines.

    So what gives, are the Celerra results useless now?

    The main points of the article are those 10 at the end. ONTAP in Cluster Mode provides some unique differentiators that can’t really be matched by any single EMC product.

    I’m even giving Isilon props and get no thanks for it, sheesh… :)

    D

  3. Lee Johns from HP here. Congratulations on the benchmark. We are also proud of our industry leading SPC1 benchmark on the HP 3PAR platform (link removed – please use your own blog for that)

    Your last comments are I assume aimed at EMC but are more than a little disingenuous. When I last looked NetApp had acquired another architecture it is selling to customers for SANs and it is not scale out.

    1. Hi Lee,

      Sure, NetApp acquired Engenio (it was a no-brainer financial decision) and we sell those boxes in scale-out solutions using either Hadoop or StorNext front ends depending on the use case. And the people getting the solution know exactly why they’re getting it.

      I removed your plug from your post, but I allowed your post – usually when I leave comments at HP’s blogs they are not approved no matter how much sense they make.

      D

  4. D —

    I think the biggest problem with this post is that you’re comparing apples and oranges. Not that either fruit is bad, but they’re just not the same!

    I think most customers would see the substantial operational differences between 24 individual, isolated and separate file systems (NetApp), and a single, scalable, auto-balancing and much more manageable single file system (EMC/Isilon, Panasas, etc.).

    The more realistic comparison (I think) would be against the VG8 clustered submission EMC did a while back, where we used multiple filesystems bound to specific nodes, much as you did here. If I remember correctly, you were highly critical of that configuration at the time. I guess things change ….

    And, no, we’re not going to aggregate 24 Isilon filesystems under a single name space to beat you. I can imagine the headline: 25 MILL-YUN SPEC SFS2008 ops!

    Some questions came up as we were bantering your results around. Keep in mind, you don’t have to answer any of these if you don’t want to. Just curious, that’s all.

    We were curious about the “Noah’s Ark” two-by-two hard pairing of the filer nodes. We had been led to believe that you folks were claiming an N+1 scheme, based on some of the earlier presos that had been sent our way. Is this a temporary or more permanent limitation?

    We were also curious about the fact that only half the available capacity was exported for use — since that doesn’t really jive with how customers use products, we were wondering what was up there?

    Also, the problem with your “estimated price” is that it’s not only incorrect, but you then went and did a bunch of derived calculations, which — of course — are also quite wrong as well. Val B. went down this path a while ago (doing math on made up numbers), it wasn’t pretty then either.

    Anyway, we all now have yet another SPEC result to go chew on.

    Thanks for the dialog.

    — Chuck

  5. Hi Chuck,

    It all depends what you want to use the system for.

    In a complex multitenancy environment, that needs multiple kinds of protocols, role-based access control and advanced functionality and efficiency features, with tight application awareness and the ability to run pretty much any workload, the NetApp system is a good bet. Maybe the best one.

    If someone truly wants a one-trick pony with a single large filesystem and high speed access for large files and doesn’t need anything else (as I’m mentioning in the post) then Isilon is a good bet. As are IBRIX, Panasas, SONAS and a few others.

    I posit that Cluster Mode is a better bet for a vastly higher percentage of workloads than is Isilon. As I mention multiple times – it’s a general-purpose storage system, not a niche box.

    Now on to your other questions:

    Exported capacity – it’s a NetApp system, look at the usable capacity in the benchmark. Exported vs usable makes no difference performance-wise with NetApp since all that’s virtualized (but does cause a performance question with many of the umpteen EMC architectures so I get where that’s coming from).

    Understanding how SPEC SFS works helps. It chews up space the with each operation, doesn’t have a fixed amount of space it needs.

    Regarding the VG8 comparison:

    Correct me if I’m wrong but…

    1. Celerra can’t move workloads on the fly from data mover to data mover
    2. Celerra can’t have data on one data mover accessed from another data mover
    3. To add to #2, you can’t use the other celerra data movers to accelerate content that’s on a single data mover.
    4. Celerra data movers can’t participate in a single namespace (that one I’m not too sure about but I think nested filesystems work only within a data mover)
    5. Celerra is NAS only – and last I checked, you can’t move a FC or iSCSI workload between separate VNX systems with zero downtime or reconfiguration (can you on the VMAX?) Your VG8 submission used 4 separate VNX boxes at the back end with no knowledge of each other.

    Ergo…

    I don’t consider the Celerra architecture with a bunch of unrelated VNX/VMAX boxes behind it a true load sharing cluster. It’s an up to 7+1 failover cluster for the NAS heads and a bunch of unrelated cluster pairs for the block heads (again with no ability to shunt workloads between VNX HA back-end pairs).

    Both the Isilon and NetApp submissions are a SINGLE cluster managed as one, with no separate back-end boxes to deal with.

    And finally, ONTAP Cluster Mode is a collection of HA pairs – no odd numbers, only even. That’s just how it’s built.

    Finally, a question for you: the estimated list price for 140 of the exact isilon nodes using in the benchmark including OneFS is what again?

    If it’s helping you out then let me know and I’ll make the change… :)

    D

  6. Quick question. Given that the Isilon solution achieved this result in around 6-7 racks of floorspace (working on 2U per node and 42U per rack), what did the 24 x FAS units in this test take up?

    Just curious…

    Cheers,

    Hugh

  7. I’m curious if any of the files were accessed across the backend cluster network. SPEC can be manipulated to make sure each node never has to access files across the backend network from the clients running the test.

    When we tested GX, we saw as much as a 40% latency hit to when the file you wanted was not on the node the client was connected. Not a big problem when you are more concerned about concurrency, but a pretty big hit none the less.

  8. Hi Jonah,

    We knew you guys would ask and yes, the files were accessed over the back-end cluster network indeedy.

    This is worst-case-scenario performance.

    Look at the submission text:

    “For UAR compliance, each flexible volume was mapped to a subdirectory of the global namespace under root (/data1,/data2,…,/data24). Each volume was accessed over all data network interfaces (ip addresses ip1…ip24) such that each volume had a unique mount point on each node. This ensured that 1/24th of the total access on each node was done through a local interface and 23/24th of the total access was done through interfaces which were remote to the node the volume resides on. For these remote accesses, data traversed the cluster network.”

    No cool features like load-sharing mirrors were used, BTW… that would be cheating :)

    Hugh: 9 racks says my sizer. But the power drawn is not more than Isilon, it’s actually a bit less (but not enough to matter in this comparison).

      1. correct, this was about as unoptimized as possible. With pNFS (assuming SPEC creates a pNFS SPEC SFS suite) the results would be rather higher.

  9. Here in RTP we had an easel set up outside of the datacenter we ran the tests in and I could watch the results get posted. It was fun, and I wondered when we’d shut off the tap. :-)

    cluster results

    So to sum it up – at least half the configuration and 50% more IOPs than the Isilon cluster? I guess if I were at the short end of that math, I’d resort to making up edges to pick at too.

    1. Congratulations to NTAP who now have the edge for NFS specs (when are the CIFS results being published, out of interest?)

      If you begin your article talking about “realistic configurations”, one of the main things a customer will require aside from a level of performance, is a level of capacity.

      People seem to be ignoring the fact that only 288TB was exported/presented to hosts which sounds to me to be terribly inefficient.

      How much usable was in the isilon config?

      Assuming you would actually WANT to use that config to host some data, it would be good to see how the NTAP and Isilon solutions stacked up comparatively, otherwise it’s pointless talking about how much they both cost, size of the config etc.

      Just leave it at an IOPS chest-beating victory lap and don’t try to draw too many long conclusions.

      1. Hi again Hugh,

        The usable capacity was over 500TB, look at the submission. Exported vs usable is all virtualized in the NetApp world.

        Neither Isilon nor NetApp used the full amount of storage. See point # 11 in the original article.

        D

  10. Hi there, I’m Cláudio Veronezi Mendes from – Brazil,
    the reallity here, and most part of the world, is far way from the benchmark SPECs, Have you ever see a costumer using 24xIsilon or whatever?
    I think that the LARGE companies that need to achieve some some amount of IOPS, don’t worry about the price, foot print and stuf.

    I think that what is really important on a benchmark is the mesures, in therms of the IO type, Block size, etc. I’d really enjoy if the EXACTLY SAME amount/type of I/O was sent to all of the players and them read the graphs.

    We usually make fun of this, like, “Volvo had 3 Truks over 400 KM/h in a benchmark” – Put all of them on a plane, open the doors and let the gravity do the job.

    Best Regards.

    Dimitris – Although I’m a EMC fan, I really enjoy your blog,
    Thanks a lot.
    that kind of discution is better than a official course.

  11. Hi Cláudio,

    SPEC and SPC are exactly that, standardized benchmarks that send the exact same type of I/O to the tested systems. That’s the whole point behind those submissions – they’re one of the few “official” ways to compare storage systems.

    All benchmarks are imperfect since they never model exactly how YOU use your storage, but at least we have a couple of standards we can use…

    D

  12. So with 500TB usable that’s still only ~20TB per filer. Seems very low for a 6000.
    Tell me, what would the config look like to hit the IOPS with a comparable usable capacity to the Isilon? I seem to remember the point of the Isilon spec test was to demonstrate linear scaling of performance with capacity.
    And when are the CIFS results being published?

    Cheers

    1. Not sure what the concern is here.

      All one would need to do to increase the capacity is change the drive type to 600GB. No need to even alter the spindle count. With the same drive count and 600GB drives the usable space would be 715TB according to my sizing tool. And one could always add more drives to get more capacity (as you rightly note this is a small amount of space for this cluster).

      I think it’s important to understand that increasing the size of the drives to 600GB SAS would not change the performance of the config one bit.

      Increasing the number of spindles wouldn’t change it, either. It would stay similar just with more space.

      The point of the test was to demonstrate linear scaling with the addition of processing nodes. This is the fastest 24 6240 nodes go with this benchmark.

      With NetApp, a processing node doesn’t have capacity itself – you just add however many disk shelves you need to it. Whereas with Isilon each node is really a server with drives inside it, so as you add nodes you also add capacity linearly.

      Does this make sense, and am I answering your question?

      1. Howdy D.. Jonas from emc here…
        Congrats on your new 24 separate file system SPEC hero number. What you’ve done with 8.1c mode reminds of F5’s Unified name space appliance. Interesting way to work around not having Coral FS. What happened to Coral btw? Can we expect that come out later or was it laid to rest out back in the crowded NTAP acquisition graveyard?

        About your response to Hugh..
        Is this *really* what you tell customers when they ask you this question? Really? This sort of positioning is just plain deceptive.

        Increasing the file system size (usable) addressed by the SPEC test harness would increase the overall working set size and overall churn in and out of cache and PAM. Increased file system size would reduce the amount of data you can cache in PAM, increasing your overall working set considerably thus producing less predictable and dramatically reduced results. If using 2X the file systems size would produce the exact same performance, you guys would have released it that way to be more comparative with ISLN. Showing us the cache/PAM hits you guys achieved on each head would illuminate exactly what’s going on here. Also, if the goal was to get to exactly the usable number you got to, you’d surely use fewer filer heads too…right?:-) You seem to keep making claims that this is more real-world than other submissions. The idea that someone would buy 24 discrete NTAP head islands (nearly the largest system you guys make) to address only 240TB, relative to what you have on these boxes as raw, seems just plain silly. It’s a benchmark config and that’s fine as long as you don’t pass it off as something a customer would buy and actually use for a traditional (non-niche) use case. Comparing this to ISLN is apples and oranges. I don’t visit your blog too often so forgive me if you’ve done this elsewhere but I would love to see you do up a compare table to the VNX results..at least to the 500K mark.. From back of the napkin calcs it looks like the VNX has the best quasi-scale-out (aggregated file systems) number and ISLN has the best true scale-out number.

        Cheers,
        -J

        1. Hi Jonas,

          Deception? Ha…

          It would help you tremendously if you understood how the SPEC SFS2008 NFS benchmark actually works.

          It really does not matter one bit how much capacity you export.

          It chews up space with each operation, and will stop chewing up space when the box can’t go any faster.

          So the idea is to have enough disk space for performance and to accommodate the benchmark size.

          NOTICE THAT ONLY 128TB OUT OF THE 800+ TB WAS USED ON ISILON!!

          Exactly because of that effect. Please do read your own submission again before making allegations.

          And my point is that this is as fast as this cluster will go in this specific test, whether it has 100TB or 50PB space.

          Customers now know the top-end number, and can add space accordingly.

          Hopefully that wasn’t confusing…

          PS: Please see other comments and the bottom 11 points in the article, the VNX with the gateway NAS is not the same thing.

          D

        2. Jonas –
          Thanks for bringing to light random code words from your time at NetApp 4 years ago. As everyone knows, NetApp ceaed all product development in March of 2007 so that kind of information is still highly relevant.
          Billy

          1. Wait a minute…EMC throwing the first stone regarding aquistions or projects that never saw the light of day? got one word for you…..ATMOS…and I only went back 1 year for that tidbit….not back to 2007. If I went farther back I could bring up BlueSky….answers the question ‘what happens when you write a standard and no one adopts it’.

  13. DK –
    Hope you have been well. Mike Dunn (EMC). Couple of quick points/questions.

    1. In your response to Jonah you state that each flexvol was mapped under a “global namespace.” Perhaps I am missing something in my understanding of C-Mode, but is it truly a global namespace or a single namespace for that cluster? There is a very big difference.
    2. You make plenty of mention of how NTAP can be used for both NAS/SAN (FC, NFS etc.) Curious, could I use the same config that was tested for FC connected hosts?
    3. You point out the use of RAID-DP, nothing wrong with that, in comparison to the Isilon protection scheme. Question I have is, if when using C-Mode, and I have a flexvol residing on an one of the HA pairs in the cluster, if I lose that HA pair, do all other HA pairs in the cluster lose access to the data on that FlexVol?

    Send my regards to the team,

    Mike Dunn

    1. Hey Mike, long time no see!

      1. What does “global” namespace mean to you? What if the cluster is globally distributed? All kinds of possibilities.

      2. FC results coming.

      3. Maybe you can answer this first: With the as-tested Isilon config, if I lose 2 nodes, does that mean my ENTIRE CLUSTER of 140 nodes goes down? :)

      How often does a NetApp HA pair fail as a whole in general?

      When you start with something reliable as the base, it helps.

      Not saying Isilon is unreliable – just illustrating architectural differences.

  14. General purpose gets thrown around A LOT in this article. Yet one benchmark is shown. Interesting.

    Please learn how Isilon, or any competitor for that matter, works before jerking it to one benchmark and passing it off as a legit pro/con review between two products.

    1. Hi Randy,

      “General purpose” indeed gets thrown around a lot since the system is, indeed, general purpose. See point #3 towards the end of the original post.

      Are you trying to say that pure scale-out NAS architectures are able to run enterprise DB or VM workloads at low latencies? :)

      We have one benchmark since that’s the one we ran first. Block benchmark results are coming. But the NetApp architecture in general already runs all kinds of workloads – this just expands the base architecture, that’s all.

      D

  15. John (NetApp) here … Very impressive results we’ve put up. The beauty is that we’ll scale out all of the protocols and bring all of the efficiences and application integration along for the ride. No SSDs, no mirrors … just some wafts of smoke emanating from the competition. The numbers speak for themselves and facts can be difficult (if not painful) to accept, particularly for those on the short end of the results.

    EMC’s Isilon, as D has pointed out, is well suited for large sequential injest. Speaking of benchmarks, and not to distract from the subject at hand,
    http://www.channelregister.co.uk/2011/10/28/netapp_video_benchmarks .

    Kudos to EMC/Isilon for taking a stab at a benchmark/market that simply does not suit them well. Looking at the resources used by NetApp to drive substantially more IOPs at substantially lower latency makes the proferred S200 solution seem insulting to the end-user’s intelligence. Add to that pNFS, CIFS, FC, FCOE, iSCSI, de-dupe, thin provisioning, compression, thin cloning, replication, application integration, etc. and you have a strong value proposition for low-latency business-critical applications and IT.

    Casual observers can sense the Fear and Uncertainty in some of the thinly-veiled posts, but there is little room for Doubt in those numbers. Mr. Hollis knows it is far more than “a SPEC NFS record to be proud of ” … but it is that too. ;-)

    Congrats to NetApp Engineering!

    1. Just to add that that I think the point is well made that this config *could* scale to 50PB (using SATA) or 10PB (using SAS) if needed or any where in between using mixed shelves. It would be interesting to see the performance and latency numbers using load sharing mirrors as well.Whilst the throughput would probably stay the same, the latency should drop even further :)

  16. Great post and discussion D. Even more impressive to me than the top-end number is how it scales damn near linearly from 4-nodes to 24-nodes. I’m assuming you can add nodes non-disruptively to an existing cluster and rebalance on-the-fly?

    thanks for posting.
    paulb

  17. I’m struggling with the “stump the chump” questions. I think D gave props to the Isilon system and even pointed out where it shines. I had two general take aways:

    1) Compared to Isilon, NTAP delivered superior results using half the hardware in a non-optimized config on a mid high-end cluster. In other words, this wasn’t nearly as fast as NTAP could go if they really wanted to build a lab queen config in the same fashion as Isilon did to get their results.

    2) NTAP did it on the first and industry’s only scale out Unified Architecture. What this means to the customer is all of the data mgmt, protection, and efficiency features apply to the broad spectrum of use cases they have in their environments.

    We believe that market is substantially larger than the niche market that Isilon targets. Nor does this scale out Unified Architecture target the niche markets that NTAP E-Series addresses best. We are talking apples and oranges in many respects but you have to tip your hat to NTAP since EMC really doesn’t have anything that does compare here. Hence the scattered references to a number of different EMC architectures to argue against the single NTAP architecture you see here.

    Mike Riley – NTAP

  18. Geert (NetApp) here, D. guess you hit a nerve with some of the guess, but like John said; the numbers speak for themselves….

    Great post!

  19. I think I understand the sensitivity the EMC people have displayed here on your blog. On June 27, 2011, they announced their record Isilon performance numbers.

    4 months and few days later, they get toppled off that perch by a yet_to_be_GA’d Cluster-Mode from NetApp. At half the configuration. And about half the price.

    What would be a lot of fun would be to put out performance and VM solution density numbers – and use dedup. ESX goes even faster when dedup’d.

  20. Just a quick comment on the assertion … “it looks like the VNX has the best quasi-scale-out (aggregated file systems)”

    The VNX configuration achieved approximately 500,000 IOPS with 4 X-Blades (I wont count the standby to be fair), and 8 5700 Service Processors, with a total of 240GB RAM over 12 discrete processing units. That works out to around 41,000 IOPS per processing unit, where there was zero cross cluster traffic (ie very highly optimised).

    Indeed, with the VNX as tested, as explained here: http://wp.me/p1k3XN-3Q the various back-end VNX processing units are NOT part of the same cluster. They are completely separate, with no data mobility possible between VNX pairs, and no cluster network back-end.

    Which is why EMC saying the VNX is a comparable architecture doesn’t make sense.

    Using the 24 node audited result we end up with around 62500 IOPS per processing unit with 23/24ths of the traffic going over non-optimised paths. If we’d optimised the paths in the same way EMC did with their implementation of the Uniform Access rule, the results would certainly have been even better.

    The closest comparison would be the 8 Node result noted on the flipboard picture above (which I’ll grant was unaudited), which had 8 discrete processing units and 192GB RAM and a similar number of cores (I think, the number of cores in the SP nodes isnt disclosed, however I assume at least 4 cores per node) which achieved around 520,000 IOPS.

    Bottom line – The FAS6240 has fewer processing units, less memory, similar CPU, and better results

    In my opinion, not inlcuding the contributions of the VNX5700 block SP’s CPUs to the result in the SPEC submission, or when calculating “per node” IOPS figures in the blogs and markting seems a little disingenuous as they offload the entirety of RAID processing, and non volatile write cache processing, and a chunk of the read cache work too.

    In any case per node processing efficienty is a little bit a bit of a moot point because nobody really cares how you package up your CPUs and RAM to get your result, as long as its cost effective and cheap to run, and on both of these fronts, I think FAS arrays have some compelling advantages. I’m sure if we’d wanted to we could have put up 4 seperate 7-mode 6240’s or maybe even 3270s and created similar or better numbers, but what, other than superior single node performance would that have proven ? The cluster mode result using a single vserver as point of aggregation is a far more interesting submission, and this, from what I’ve seen is just the beginning of what we can achieve on this new foundation.

    Regards
    John

  21. So tell us how much of the workload was coming out of read only PAM cards? Anyone can produce great IOPS numbers if the workload is coming out of cache. How much of the benchmark was write based versus read based?

    1. Taken from the Specsfs2008 User Documentation:

      LOOKUP 24%
      READ 18%
      WRITE 10%
      GETATTR 26%
      READLINK 1%
      READDIR 1%
      CREATE 1%
      REMOVE 1%
      FSSTAT 1%
      SETATTR 4%
      READDIRPLUS 2%
      ACCESS 11%

      This is standard for the benchmark, otherwise there wouldn’t be much point in having a standard benchmark!

      Are you suggesting that the Isilon result with 6.7TB of RAM (cache) and 25TB of SSD (cache) are invalid? The Netapp result has 1.1TB of RAM (cache) and 12.2TB of FlashCache (cache). Where is the difference other than the fact that we have *less* cache?

      1. Mike,
        having lots of cache or SSD certainly makes getting a good benchmark result earlier, (as evidenced by a few all solid state benchmarks done by EMC), but these benchmarks are all gated by the CPU and/or the back end cluster interconnects and how efficiently they are being used. Adding more cache (or disk for that matter) without also adding more CPU wont give you a better result.

        Regards
        John

  22. Fair enough. Most of the NetApps system in production are based on Ontap 7 or Ontap 8 7g mode. Ontap 8 cluster mode has been out for a number of years. I no longer have access to NOW but the last I looked 6 months ago a very small number of 8 cluster mode systems had been deployed.

    Please let us know how many Ontap 8 cluster mode not 7mode are in production.

    1. I’m not sure how you can make confident assertions that “a very small number” of ONTAP 8 cluster modes systems had been deployed based on access to the release comparison tool on our website, as this only gives broad ranges for each release, with the smallest babd being 0-500. It’s also worth noting that each of those bands applies to each minor point release in the ONTAP release line (there are currently 4 cluster capable point releases supported in the ONTAP 8 code line and a number more in the GX code line)

      I suppose a qalatative statement like “very small” also depends on your definition of “very small”. If you are comparing it to the 100,000+ FAS controllers running Ontap 7, then the number of scale out deployments from any vendor including Isilon/EMC could well be described as “very small”.

      As you will see NetApp will broaden the applicability of scale out well beyond the current niche workloads currently serviced by the likes of Isilon, and as it does so, ONTAP will grow to be the predominant scale out offering in the marketplace. Like I said before, these benchmarks are only the beginning.

    2. The first RC of 8 was released in Sept 2009, with Ga coming in March 2010. Given that you last checked the matrix apparently 6 months ago, it would only have been out for just over 1 year. Your comment of “a number of years” is misleading I believe.

  23. So I see I will not get a answer on the number of systems running Ontap 8 cluster mode. Here is my guess….500 or less. Let me know if I am in the ball park.

    Why is it that NetApp continues to ship their new systems with Ontap 8 7G mode versus cluster mode. If cluster mode has been GA since March 2010, would it not make sense to ship all new systems with cluster mode.

    1. Hi Mike B…

      It’s good etiquette to disclose your affiliation (EMC).

      Anyway… not really in the ballpark.

      It’s an interesting question you ask. Why are not all NAS systems EMC sells scale-out?

      Why are not all SAN systems EMC sells scale-out?

      What’s the percentage of customers for Isilon vs Celerra/VNX?

      And how many storage architectures do EMC, HP, IBM and others have for NAS and SAN and replication and backup? They’re good products, but how many does a customer need? And why?

      You see, the NetApp value prop stays the same as before with Cluster Mode, only expanded in some interesting ways and with some extra features. Those will be part of another article.

      D

  24. Funny to see how all the EMC people trying to figure out how NetApp could publish such a high number. Looks like that their own NetApp systems in-house are wrongly configured ;-) As always: If you don’t understand how a competitive product works, don’t make assumptions or false statements. Concentrate on your own stuff. If you’re envious of NetApp because you only have a scale-up (VNX) or a scale-out (Isilon) product for NAS and not both for one product than go back to engineering and complain about it. Don’t make others responsible for your faults. You have tons of products, which are shelf warmers, so don’t talk about selling numbers. This is a blog about a solid high performance number where people like Dimitris are proud of.

    Amen.

    -Olli

  25. Olli, as DK points out, it’s proper etiquette to disclose your vendor affiliation. I think it’s funny to see that someone who is paid to provide information on competitive products would get on a soapbox and tell others not to talk about a competitors product and to concentrate on their own.

    Mike Dunn (EMC)

  26. Hi, just a comment on this test environment. 24 node cluster is the biggest count you can support right now, and if I’m not wrong, for 8.1, this 24 node count is for NAS only.

    So please do not mislead the audience by saying NetApp can achieve a higher NFS IOPs count with a lower configuration

    1. I’m not sure how the two statements in the previous comment are related to each other … 24 nodes is the largest node count currently qualified and supported for NFS workloads. Using 24 nodes and much less disk and Flash, NetApp produced a bigger number with less hardware, which I think counts as a “lower configuration”.

      I’m not sure how you conclude this is misleading anyone.

  27. Customer here.

    I read this paper when it was first released and i thought it seemed off but dismissed it since i didn’t have the time. I believe everyone above has brought up most of the technical questions i would ask so for now i’ll leave that out. I would like to mention that we currently have both NetApp and EMC in our environment so i’m as unbiased as can be.

    But the question i am going to ask is why are you incorrectly stating the cost of the NetApp configuration. I know the Isilon system is priced correctly since you were kind enough to post the link to the EMC master pricing list, but i found it weird you didn’t post one to yours… So i decided to go and find some prices on the offical NetApp site which i eventually did.

    So when using your items list on the SpecFS paper, the price comes to almost 9,000,000 and thats WITHOUT the 8.1 cluster mode license since its not listed on the July 11 contract i was looking at.

    I’m curious how your comments about being 50% cheaper then are true. Or were you trying to slip that past us, that EMC was list pricing while yours isn’t.

    This is troubling to a purchaser like myself when a company needs to falsify there information to make there company look better.

    1. Hi John,

      Would you mind letting me know what company you work at? You could drop me a private line if you don’t want to share publicly.

      I would very much like to know how you calculated the price. But I think I already know.

      You see, there have been major changes since the July pricing you are looking at, rendering all the cache boards completely free of charge for the SPEC configuration (each 6240 or 6280 node now gets a no-cost 512GB cache added plus one protocol, in this case NFS).

      There have also been other reductions, everything I mentioned combined making the price difference compared to the July pricing over $2.8m. Do the subtraction and compare to the original figure I posted :)

      So our pricing was absolutely list, and nobody’s trying to slip anything past anyone. I didn’t post a link to our pricing since I’m not aware of a current price sheet that’s internet-accessible through our site. But anyone can contact their VAR and have this priced.

      A blog such as this is a very public document indeed – my personal and professional reputation are on the line every time I post, so I go beyond the “measure twice, cut once” rule and frequently measure 4-5 times… :) (which also explains why I don’t post daily like some folks do – solid research takes time plus blogging is not my day job).

      Let me know if you’d like more clarification.

      Finally, nobody is unbiased. Read this: http://wp.me/p1k3XN-28

      D

Leave a comment for posterity...