Slide1

Buyer beware: is your storage vendor sizing properly for performance, or are they under-sizing technologies like Megacaching and Autotiering?

With the advent of performance-altering technologies (notice the word choice), storage sizing is just not what it used to be.

I’m writing this post because more and more I see some vendors not using scientific methods to size their solution, instead aiming to reach a price point, hoping the technology will work to achieve the requisite performance (and if it doesn’t, it’s sold anyway, either they can give some free gear to make the problem go away, or the customer can always buy more, right?)

Back in the “good old days”, with legacy arrays one could (and still can) get fairly deterministic performance by knowing the workload required and, given a RAID type, know roughly how many disks would be needed to maintain the required performance in a sustained fashion, as long as the controller and buses were not overloaded.

With modern systems, there is now a plethora of options that can be used to get more performance out of the array, or, alternatively, get the same average performance as before, using less hardware (hopefully for less money).

If anything, advanced technologies have made array sizing more complex than before.

For instance, Megacaches can be used to dramatically change the I/O reaching the back-end disks of the array. NetApp FAS systems can have up to 16TB of deduplication-aware, ultra-granular (4K) and intelligent read cache. Truly a gigantic size, bigger than the vast majority of storage users will ever need (and bigger than many customers’ entire storage systems). One could argue that with such an enormous amount of cache, one could dispense with most disk drives and instead save money by using SATA (indeed, several customers are doing exactly that). Other vendors are following NetApp’s lead and starting to implement similar technologies — simply because it makes a lot of sense.

However…

It is crucial that, when relying on caching, extra care is taken to size the solution properly, if a reduction in the number and speed of the back-end disks is desired.

You see, caches only work well if they can cache the majority of what’s called the active working set.

Simply put, the working set is not all your data, but the subset of the data you’re “touching” constantly over a period of time. For a customer that has, say, a 20TB Database, the true working set may only be something as small as 5% — enabling most of the active data to fit in 1TB of cache. So, during daily use, a 1TB cache could satisfy most of the I/O requirements of the DB. The back-end disks could comfortably be just enough SATA to fit the DB.

But what about the times when I/O is not what’s normally expected? Say, during a re-indexing, or a big DB export, or maybe month-end batch processing. Such operations could vastly change the working set and temporarily raise it from 5% to something far larger — at which point, a 1TB cache and a handful of back-end SATA may not be enough.

Which is why, when sizing, multiple measurements need to be taken, and not just average or even worst-case.

Let’s use a database as an example again (simply because the I/O can change so dramatically with DBs).You could easily have the following I/O types:

  1. Normal use – 20,000 IOPS, all random, 8K I/O size, 80% reads
  2. DB exports — high MB/s, mostly sequential write,large I/O size, relatively few IOPS
  3. Sequential read after random write — maybe data is added to the DB randomly, then a big sequential read (or maybe many parallel ones) are launched.

You see, the I/O profile can change dramatically. If you only size for case #1, you may not have enough back-end disk to sustain the DB exports or the parallel sequential table scans. If you size for case 2, you may think you don’t need much cache since the I/O is mostly sequential (and most caches are bypassed for sequential I/O). But that would be totally wrong during normal operation.

If your storage vendor has told you they sized for what generates the most I/O, then the question is, what kind of I/O was it?

The other new trendy technology (and the most likely to be under-sized) is Autotiering.

Autotiering, simply put, allows moving chunks of data around the array depending on their “heat index”. Chunks that are very active may end up on SSD, whereas chunks that are dormant could safely stay on SATA.

Different arrays do different kinds of Autotiering, mostly based on various underlying architectural characteristics and limitations. For example, on an EMC Symmetrix the chunk size is about 7.5MB. On an HDS VSP, the chunk is about 40MB. On an IBM DS8000, SVC or EMC Clariion/VNX, it’s 1GB.

With Autotiering, just like with caching, the smaller the chunk size, the more efficient the end result will ultimately be. For instance, a 7.5MB chunk could need as little as 3-5%% of ultra-fast disk as a tier, whereas a 1GB chunk may need as much as 10-15%, due to the larger size chunk containing not very active data mixed together with the active data.

Since most arrays write data with a geometric locality of reference (in contrast, NetApp uses geometric and temporal), with large-chunk autotiering you end up with pieces of data that are “hot” that always occupy the same chunk as neighboring “cool” pieces of data. This explains why the smaller the chunk, the better off you are.

So, with a large chunk, this can happen:

Slide1

The array will try to cache as much as it can, then migrate chunks if they are consistently busy or not. But the whole chunk has to move, not just the active bits within the chunk… which may be just fine, as long as you have enough of everything.

So what can you do to ensure correct sizing?

There are a few things you can do to make sure you get accurate sizing with modern technologies.

  1. Provide performance statistics to vendors — the more detailed the better. If we don’t know what’s going on, it’s hard to provide an engineered solution.
  2. Provide performance expectations — i.e. “I want Oracle queries to finish in 1/4th the time compared to what I have now” — and tie those expectations to business benefits (makes it easier to justify).
  3. Ask vendors to show you their sizing tools and explain the math behind the sizing — there is no magic!
  4. Ask vendors if they are sizing for all the workloads you have at the moment (not just different apps but different workloads within each app) — and how.
  5. Ask them to show you what your working set is and how much of it will fit in the cache.
  6. Ask them to show you how your data would be laid out in an Autotiered environment and what bits of it would end up on what tier. How is that being calculated? Is the geometry of the layout taken into consideration?
  7. Do you have enough capacity for each tier? On Autotiering architectures with large chunks, do you have 10-15% of total storage being SSD?
  8. Have the controller RAM and CPU overheads due to caching and autotiering been taken into account? Such technologies do need extra CPU and RAM to work. Ask to see the overhead (the smaller the Autotiering chunk size, the more metadata overhead, for example). Nothing is free.
  9. Beware of sizings done verbally or on cocktail napkins, calculators, or even spreadsheets – I’ve yet to see a spreadsheet model storage performance accurately.
  10. Beware of sizings of the type “a 15K disk can do 180 IOPS” — it’s a lot more complicated than that!
  11. Understand the difference between sequential, random, reads, writes and I/O size for each proposed architecture — the differences in how I/O is done depending on the platform are staggering and can result in vastly different disk requirements — making apples-to-apples comparisons challenging.
  12. Understand the extra I/O and capacity impact of certain CDP/Replication devices — it can be as much as 3x, and needs to be factored in.
  13. What RAID type is each vendor using? That can have a gigantic performance impact on write-intensive workloads (in addition to the reliability aspect).
  14. If you are getting unbelievably low pricing — ask for a contract ensuring upgrade pricing will be along the same lines. “The first hit is free” is true in more than one line of business.
  15. And, last but by no means least — ask how busy the proposed solution will be given the expected workload! It surprises me that people will try to sell a box that can do the workload but will be 90% busy doing so. Are you OK with that kind of headroom? Remember – disk arrays are just computers running specialized software and hardware, and as such their CPU can run out of steam just like anything else.

If this all seems hard — it’s because it is. But see it as due diligence — you owe it to your company, plus you probably don’t want to be saddled with an improperly-sized box for the next 3-5 years, just because the offer was too good to refuse…

D

 

Technorati Tags: , , , , , , , , ,

8 thoughts on “Buyer beware: is your storage vendor sizing properly for performance, or are they under-sizing technologies like Megacaching and Autotiering?

  1. Urban

    D,

    I couldn’t agree more with you. I’m a storage architect at a reseller and I architect EMC, IBM and NetApp storage. Each and every one starts with questions and datacollection. It then goes into storage modeling with enough headroom for 3-7 years of growth (typically 5), depending on their needs. There are performance and capacity targets to achieve. Capacity is easy, then rightsizing spindle size and count is next.

    The newer advanced IO technologies do add some complexity to the equation, but not always. Sometimes you can take worst case scenarios into account. Autotiering is more difficult to model, but you’ve already identified the particulars. If I know the working set, it has to fit withing the IOPS for the appropriate tier. That could be 300 SAS drives, or 20 SSD. Autotiering drives us knowing the working set. Caching drives us to knowing the working set. Modeling is a bit more complicated, but still achievable.

    I know there are a lot of storage architects that will low ball a quote to hit capacity and fix performance on the back end — from every vendor out there. I know there are many that don’t do the due diligence of performance modeling.

    I’ve been doing remote replication for over 10 years. Performance modeling needs to occur for each site and the pipes, or you have a disaster on your hands.

    Let me ask you this: have you seen a lot of NetApp customers who have a 100 TB requirement sold 50 TB counting on dedupe achieving it’s goals? Have you seen a lot of NetApp customers with 4+ TB of FlashCache and all SATA to achieve their goals? I’m interested in your real world architectures you’ve designed capitalizing on these technologies.

    Reply
    1. Pablo906

      I was sold ~4TB when I really use 7+ counting on dedupe to do its job. And it did, but performance is really lackluster. I think things have definitely improved with PAM and GX, so new configs I’m seeing a ton more, fewer shelves approaches in the market place.

      Reply
  2. Dimitris Post author

    Hi Urban,

    Yes, modeling megacaching and autotiering is absolutely doable, my beef is that I see people not doing it right the vast majority of the time. Even the people that try to do it, often don’t quite understand exactly how it should be done, especially with certain architectures and the way they slice and move the data.

    Just yesterday I spoke to a customer that has a huge working set, yet a certain three-letter-acronym vendor is sizing 1% SSD for and a ton of SATA, claiming it will be “not just good, but good enough”, just to be under our pricing.

    This will inevitably blow up in the customer’s face, but the competitor rep doesn’t care, since he’ll have made quota.

    I think that reps should pay steep financial penalties if there are customer satisfaction problems due to misconfigurations that could have been avoided. I assure you it won’t happen again :)

    Another way to do it: Not have hyper-aggressive sales management that forces reps to do crazy things just to make the sale happen. Unfortunately, certain storage companies that are definitely NOT in the top 10 places to work, are abusive to their reps, which just perpetuates bad behavior.

    Regarding your last question:

    I’ve sold configs that needed 1/3rd the storage of a competitor, let alone 50%. It’s not just dedupe, but the combo of NetApp technologies (snaps, clones, dedupe, compression, vaulting) that allows this in the right circumstances.

    Most of my deals have to deal with large DBs, and we take extreme pains to make sure sizing is right. With Flash Cache, I’ve configured systems that needed 40 drives instead of 300 drives for a competitor.

    So yes, we are indeed selling more efficient configs in general, but my point is that proper sizing is painstakingly done to ensure everything will work as advertised yet still provide the efficiencies.

    If the efficiencies won’t work, then we are up-front with the customer. If the customer won’t provide performance data, then chances are we’ll over-engineer it just to be safe.

    This can break down (for all vendors) when the VARs are involved of course, since now a third party enters the equation, and they don’t always size properly (most storage vendor engineers don’t even get to see the majority of the smaller deals that VARs do).

    Even within VARs, as you well know there are levels of storage people, with some companies using glorified sales reps to size and sell storage, and only the larger deals making their way to the REAL storage people within the VAR.

    Problem is, customers don’t know all this stuff is going on…

    VAR training is important. However, many VARs try to sell all storage technologies, and it’s pretty tough to be a performance expert in a single platform, let alone multiple platforms from the same vendor, let alone multiple platforms from multiple vendors.

    Here’s another “buyer beware”:

    If your VAR tells you “we sell everything”, chances are, they’re not experts in “everything”. Something has to give.

    D

    Reply
  3. Paul P

    Yes, I’ve seen the same vendor using those tactics – the most recent I saw: a high end system with 300 spinning disks (of the same type) and just 3 x SSD’s – for a tiering solution??? Would like to see how that would work.

    Reply
    1. Pablo906

      They tried that on me actually, it’s a limitation of the low end controller, furthermore, the pricing on licensing and scaling on the next model up is SIGNIFICANTLY higher. I’ve told everyone I work with to simply throw those configs away and start with the model up.

      To be fair I’ve seen some NetApp partners pushing 3120′s where a step up is really the right choice for future scaling and performance needs. The difference is that going from a 3120 to a 3140 doesn’t terrorize the bottom line price.

      Reply
  4. AD

    Where has this blog been? I would be interested in learning more about Cloud Lifecycle Management software solutions. Do you know any companies that have a solid end to end management solution?

    Reply
  5. HoosierStorageGuy

    Great write-up and very applicable to any vendor. A minor jab at RecoverPoint (#12), but I can’t say that it isn’t justified. Undersized journal volumes (for capacity and performance) are definitely an issue. Anyway, disk sizing in the past year has become much more of a challenge with the release of megacaching and auto-tiering. As vendors released these new technologies, the sizing tools have STILL not been updated in many cases. To get any input as to how well megacaching will work, often complex locality of reference analysis is required. The best rule of thumb I’ve received from vendors most recently is to size it for performance using traditional methodology as if there will be no benefit from megacaching or auto-tiering. Huh? How do I explain that to the customer who was sold on the marketing slides that say “pay less $ for better performance thanks to megacaching and auto-tiering”.

    Reply
    1. Dimitris Post author

      Hi Dan,

      It’s really not easy to size properly with the advanced performance technologies on offer.

      If you are able to sell NetApp then you must have seen our sizing tools and seen just how much info you can put in there.

      Getting the info is the hard part :)

      But we provide some tools and methodologies to extract the requisite info.

      I urge all customers to really beware if vendors or VARs are not spending enough time trying to gather performance data.

      D

      Reply

Leave a comment for posterity...