More EMC VNX caveats

Lately, when competing with VNX, I see EMC using several points to prove they’re superior (or at least not deficient).

I’d already written this article a while back, and today I want to explore a few aspects in more depth since my BS pain threshold is getting pretty low. The topics discussed:

  1. VNX space efficiency
  2. LUNs can be served by either controller for “load balancing”
  3. Claims that autotiering helps most workloads
  4. Claims that storage pools are easier
  5. Thin provisioning performance (this one’s interesting)
  6. The new VNX snapshots

References to actual EMC documentation will be used. Otherwise I’d also be no better than a marketing droid.

VNX space efficiency

EMC likes claiming they don’t suffer from the 10% space “tax” NetApp has. Yet, linked here are the best practices showing that in an autotiered pool, at least 10% free space should be available per tier in order for autotiering to be able to do its thing (makes sense).

Then there’s also a 3GB minimum overhead per LUN, plus metadata overhead, calculated with a formula in the linked article. Plus possibly more metadata overhead if they manage to put dedupe in the code.

My point is: There’s no free lunch. If you want certain pool-related features, there is a price to pay. Otherwise, keep using the old-fashioned RAID groups that don’t offer any of the new features but at least offer predictable performance and capacity utilization.

LUNs can be served by either controller for “load balancing”

This is a fun one. The claim is that LUN ownership can be instantly switched over from one VNX controller to another in order to load balance their utilization. Well – as always, it depends. It’s also important to note that VNX as of this writing does not do any sort of automatic load balancing of LUN ownership based on load.

  1. If it’s using old-fashioned RAID LUNs: Transferring LUN ownership is indeed doable with no issues. It’s been like that forever.
  2. If the LUN is in a pool – different story. There’s no quick way to shift LUN ownership to another controller without significant performance loss.

There’s copious information here. Long story short: You don’t change LUN ownership with pools, but rather need to do a migration of the LUN contents to the other controller (to another LUN, you can’t just move the LUN as-is – this also creates issues), otherwise there will be a performance tax to pay.

Claims that autotiering helps most workloads

Not so FAST. EMC’s own best practice guides are rife with caveats and cautions regarding autotiering. Yet this feature is used as a gigantic differentiator at every sales campaign.

For example, in the very thorough “EMC Scaling performance for Oracle Virtual Machine“, the following graph is shown on page 35:


The arrows were added by me. Notice that most of the performance benefit is provided once cache is appropriately sized. Adding an extra 5 SSDs for VNX tiering provides almost no extra benefit for this database workload.

One wonders how fast it would go if an extra 4 SSDs were added for even more cache instead of going to the tier… 🙂

Perchance the all-cache line with 8 SSDs would be faster than 4 cache SSDs and 5 tier SSDs, but that would make for some pretty poor autotiering marketing.

Claims that storage pools are easier

The typical VNX pitch to a customer is: Use a single, easy, happy, autotiered pool. Despite what marketing slicks show, unfortunately, complexity is not really reduced with VNX pools – simply because single pools are not recommended for all workloads. Consider this typical VNX deployment scenario, modeled after best practice documents:

  1. RecoverPoint journal LUNs in a separate RAID10 RAID group
  2. SQL log LUNs in a separate RAID10 RAID group
  3. Exchange 2010 log LUNs in a separate RAID10 RAID group
  4. Exchange 2010 data LUNs can be in a pool as long as it has a homogeneous disk type, otherwise use multiple RAID groups
  5. SQL data can be in an autotiered pool
  6. VMs might have to go in a separate pool or maybe share the SQL pool
  7. VDI linked clone repository would probably use SSDs in a separate RAID10 RAID group

OK, great. I understand that all the I/O separation above can be beneficial. However, the selling points of pooling and autotiering are that they should reduce complexity, reduce overall cost, improve performance and improve space efficiency. Clearly, that’s not the case at all in real life. What is the reason all the above can’t be in a single pool, maybe two, and have some sort of array QoS to ensure prioritization?

And what happens to your space efficiency if you over-allocate disks to the old-fashioned RAID groups above? How do you get the space back?

What if you under-allocated? How easy would it be to add a bit more space or performance? (not 2-3x – let’s say you need just 20% more). Can you expand an old-fashioned VNX RAID group by a couple of disks?

And what’s the overall space efficiency now that this kind of elaborate split is necessary? Hmm… 😉

For more detail, check these Exchange and SQL design documents.

Thin provisioning performance

This is just great.

VNX thin provisioning performs very poorly relative to thick and even more poorly relative to standard RAID groups. The performance issue makes complete sense due to how space is allocated when writing thin on a VNX, with 8KB blocks assigned as space is being used. A nice explanation of how pool space is allocated is here. A VNX writes to pools using 1GB slices. Thick LUNs pre-allocate as many 1GB slices as necessary, which keeps performance acceptable. Thin LUNs obviously don’t pre-allocate space and currently have no way to optimize writes or reads – the result is fragmentation, in addition to the higher CPU, disk and memory overhead to maintain thin LUNs 🙂

From the Exchange 2010 design document again, page 23:


Again, I added the arrows to point out a couple of important things:

  1. Thin provisioning is not recommended for high performance workloads on VNX
  2. Indeed, it’s so slow that you should run your thin pools with RAID10!!!

But wait – thin provisioning is supposed to help me save space, and now I have to run it with RAID10, which chews up more space?

Kind of an oxymoron.

And what if the customer wants the superior reliability of RAID6 for the whole pool? How fast is thin provisioning then?

Oh, and the VNX has no way to fix the fragmentation that’s rampant in its thin LUNs. Short of a migration to another LUN (kind of a theme it seems).

The new VNX snapshots

The VNX has a way to somewhat lower the traditionally extreme impact of FLARE snapshots by switching from COFW (Copy On First Write) to ROFW (Redirect On First Write).

The problem?

The new VNX snapshots need a pool, and need thin LUNs. It makes sense from an engineering standpoint, but…

Those are exactly the 2 VNX features that lower performance.

There are many other issues with the new VNX snapshots, but that’s a story for another day. It’s no wonder EMC pushes RecoverPoint far more than their snaps…

The takeaway

There’s marketing, and then there’s engineering reality.

Since the VNX is able to run both pools and old-fashioned RAID groups, marketing wisely chooses to not be very specific about what works with what.

The reality though is that all the advanced features only work with pools. But those come with significant caveats.

If you’re looking at a VNX – at least make sure you figure out whether the marketed features will be usable for your workload. Ask for a full LUN layout.

And we didn’t even talk about having uniform RAID6 protection in pools, which is yet another story for another day.


Technorati Tags: , , , , , , , , ,

11 Replies to “More EMC VNX caveats”

  1. Amen, brother. We’ve learned some of the things that are now documented the hard way. It’s exactly as you described: the slides tell you it’s all going to be easier, faster and generally prettier. But when the implementation workshops come along it’s all “well you shouldn’t do that” and “no, it doesn’t quite work that way”. Very frustrating considering we’ve ruled out cheaper solutions due to some of the EMC promises that they now can’t quite deliver as we expected them to. Anyway, our project is well past the point of no return so we’ll just have to make the best of it.

    1. Thought you did a great job with the blog post. As a delivery engineer for an EMC partner I still think a great deal of good comes from some of the features you speak of that in the SMB space that they benefit from greatly from what I’ve seen post install. I view the “battle” between EMC/NetApp as bad polotics, at the end of the day, they are both great arrays that offer similar features but different implementation paths (some good, some maybe not so good)

      1. Thanks. Oh I never said there’s no benefit from the features – what I have a problem with is the way the features are marketed.

        1. It doesn’t matter whether you are talking about Netapp, EMC, IBM etc, they all market features based on scenarios that the Application Vendor doesn’t support which you have pointed out in their specific Application best practice guides. As Duane mentioned above the SMB benefits greatly from the VNX features mentioned above. In fact, that is why have become a big fan of EMC storage and especially with the VNX line which was updated with many features with its Flare 32 release called Inyo.

          Not unlike some other vendors EMC’s offering allows for both a simple approach where you create a large mult-tiered block pool backed by Fast Cache and Fast Tiering or advanced where you may dedicate spindles for some workloads (Eg. Logs on mirrored spindles to isolate the workload and more likely to ensure predictable and/or guarantee performance) while using a pool for others as you mentioned above. I, as a Storage Architect/Implementor and Administrator appreciate the flexibility.

          Here are some comments specific to each of your sections:

          – LUNs can be served by either controller for “load balancing”
          I agree with most of your comments; the exception being “There’s no quick way to shift LUN ownership to another controller without significant performance loss”. I haven’t completed extensive testing but, from my experience, I haven’t seen significant performance loss for any failover event. With that said, if you are experiencing significant failover events you have either configured your multipathing incorrectly or a failover has been forced because of a major hardward failure or you are upgrading your array software. Of course there are exceptions because certain workloads would be impacted by the additional latency involved when the non-owning SP must fulfill requests though, as mentioned above, those should be exceptions to the rule.

          – Claims that auto-tiering helps most workloads
          I’m not sure what you are getting at in this section. Of course auto-tiering helps most workloads and you chose one which it wouldn’t. If all of the 1GB blocks, for a given workload, already exist in the Extreme Performance Tier (SSD), if course Auto-Tiering isn’t going to help performance but it could help with ROI if some of that workload is rarely used and could exist on cheaper disk. As you pointed out, Fast Cache also isn’t going to have any significant impact on the performance for that data because it already exists on SSD which is exactly what Fast Cache is. In fact, the array algorithms won’t even place data in Fast Cache if it already exists on SSD. In addition (Again speaking to flexibility), if you don’t want that workload to ever tier lower you can force it to always exist in the top tier or simply turn off tiering on that LUN.

          – Claims that storage pools are easier
          First you mention best practice documents. As anyone in the industry knows, best practice documents are just that. They are each vendors get out of jail card as they are generally the perfect scenario for performance, redundancy etc but then again, they also cost the most. In my experience, you account for some/all of them in certain instances but just use them as reference in others because following best practices configuration guidelines are simply too expensive for many organizations to afford. You do realize that those best practice examples are no different for any storage whether that is local or presented by an array let alone the VNX which is your focus.
          “However, the selling points of pooling and auto-tiering are that they should reduce complexity, reduce overall cost, improve performance and improve space efficiency. Clearly, that’s not the case at all in real life.” You may want to spend some time actually working with the product because, in my experience, utilizing Block Storage Pools with Auto-Tiering (Usually paired with Fast Cache to take care of bursty data that doesn’t necessarily meet the requirements for tiering to a higher tier) does generally reduce overall cost, improve performance and space efficiency.

          -Thin provisioning performance
          Not much to comment on this section other than it’s misguided because many application best practice documents don’t suggest the use of any type of Thin provisioning whether that be on the array, on VMware, etc. I have seen quite a few instances of VNX Thin provisioning working very well just as I have on VMware, then again I have seen instances where this doesn’t make sense such as quickly growing workloads but that is the case on any implementation of Thin provisioning.

          -The new VNX snapshots
          I really can’t comment as haven’t spend time comparing performance #’s between the two. Then again, like mentioned above, it depends. Snapshots are a bad idea for certain workloads no matter who’s implemention you are using. VNX snapshots do benefit from the move away from COFW to ROFW and is a step in the right direction. Is it up to par with other Vendors; that I can’t say.

          -The Takeaway.
          You could have said it better “There’s marketing, and then there’s engineering reality.” though I my feeling is that you portay that gap as being quite a bit larger than is actual reality. In fact you seem to conveniently point out issues that exist across any vendor array product when it comes to meeting application best practices. Statements such as “The reality though is that all the advanced features only work with pools. But those come with significant caveats.” is misguided (Fast Cache can be enabled on RG created LUNs) and is unfortunate because I enjoy many of your articles and appreciate reading feedback from someone who is willing to spend the time to provide constructive criticism because it keeps all the vendors honest and that makes my job easier.

          1. very well said, Denny!
            you’re absolutely right about these points. [esp the one where you said, FAST Cache can be enabled on RAID Group LUNs]

            rehan, from India.

          2. Great piece.

            I think the early iterations of the VNX (haven’t seen much enough about VNX2 to have an opinion) had a lot of good ideas that didn’t work well. I had been told repeatedly by EMC technical personnel to not use thin provisioning if we wanted good performance. This was a post sales conversation, so it was frustrating to say the least.

            I have had friends that also deployed a largish VNX environment with fast cache that didn’t make any bit of performance difference to their environments, but had huge performance problems when they found out their mirrorview log disks didn’t have enough IO to keep up with the prod disks. It was too complicated for them to figure out ahead of time, and it took months for support to find the issue
            This wasn’t supposed to happen, they were told the VNX was easy to use, and so much faster than the clariion they moved off of.

            We had recoverpoint, which was very cool, but very complicated. We wanted to move off the old destination array to a new one, and once we saw how much that cost, we just scrapped EMC altogether. It wasn’t that cool or necessary.

            This is the EMC customer experience. EMC is a sales driven company, and after that sale is made, it seems there is very initiative in making the customer happy or to fix any issues unless there is the possibility of something else to sell them.

            I was already on my way to not wanting to work with any EMC products at that point, and the above made it really easy to sell upper management on a new platform altogether.

            You can say I am biased against them all you like, but its because I have been an EMC customer for over a decade, and either the sales people and sales engineers have no idea what they are talking about when they sell products to customers, or they are fraudulent. I still and am not sure which is the answer. It could be both.

            I have not had this experience with all the other major storage vendors I have worked with (HP, NetApp, IBM, Quantum). For the most part the products work as advertised, or any little caveats are well presented before purchasing.

            And the constant FUD from EMC about other products is frustrating. Deduplication and thin provisioning saves us 80% on our virtual environment, and in some cases more. We have lots of like machines, but one of the other enormous benefits of thin provisioning is that the VM swap files are not actually used, so they don’t take up any space. One of our VM environments has over 4TB of swap space provisioned, but never used. This space was used up on the clariion, but not on the storage with thin provisioning.

            My favorite FUD I ever heard about thin provisioning and dedupe was that “You still need all that space to begin with.” Also not true. If you start off with 500 GB of VMs on a 1TB NFS share or LUN and dedupe down to 40GB, and then you deploy 500GB more machines, you shrink back down 80GB, you still have never needed more than 580GB, and only for a day.

            Every time a sales person from EMC calls me (I had 4 cold calls from them last month) I tell them we aren’t going to have a productive conversation, but they want to talk anyway, and I go on like this. Even thought I ask them to stop calling me, someone from a different part of the country will call and start the whole conversation all over again.

            Probably more of a rant than I started with, and I could go on, but its Friday.

  2. Re: EMC being sales-driven: I’ve had the total opposite experience. EMC being great with post-sales (even lending us entire SSD-filled arrays to get us out of a hole, FOR FREE)… while NetApp were the guys who sold us a FAS which went obsolete two weeks after we received it, and now it’s 8 months after purchase and we can’t even get disks any more for the bloody thing.

    NetApp in my experience have some of the worst used-car salesmen tactics in the business.

    1. What’s your serial number? I doubt you can’t get more disks for the system, I would like to clear this up. Maybe the same exact size disk isn’t being sold any more. Larger ones for sure are.

Leave a comment for posterity...